Main Content

Detect and Track Object Using Deep Learning on Raspberry Pi

This example shows how to use the MATLAB® Support Package for Raspberry Pi™ Hardware to deploy a deep learning algorithm that detects and tracks an object in Connected IO and PIL modes. This algorithm uses the ResNet-18-based YOLOv2 neural network to identify the object captured by the camera mounted on a servo motor and connected to the Raspberry Pi hardware board. You can experiment with different objects in your surroundings to see how accurately the network detects images on the Raspberry Pi hardware.

Prerequisites

  • Configure the Raspberry Pi network using the Hardware Setup window. During this process, download the MathWorks® Raspbian image for deep learning. Instead of using the MathWorks Raspbian image, if you choose to customize an existing image on your hardware, ensure that you select the option to install the ARM® Compute Library.

  • For information on configuring the Raspberry Pi hardware, see the Getting Started with MATLAB Support Package for Raspberry Pi Hardware example.

Required Hardware

Hardware Setup

  1. Power the Raspberry Pi target board.

  2. Connect the servo motor to the Raspberry Pi target board using the jumper cables. Connect the GND and VCC pins. Additionally, in this example, you connect the servo motor signal pin to the GPIO pin 12 of the Raspberry Pi target board.

  3. Mount the camera on top of the servo motor using sticky tape or an adhesive. This example uses the Raspberry Pi webcam.

The example can be divided into these phases: Setup, Detection, and Tracking.

A. Setup Phase

In this phase, you create a tracker object and obtain the ground truth data. You also configure the Raspberry Pi hardware board details and the type of interface and capture time for image capture of the tracked object.

Create tracker Object

Create the tracker object.

tracker = raspiObjectTracker
tracker =
raspiObjectTracker with properties:
    BoardDetails: [1x1 struct]
           Setup: [1x1 struct]
        Detector: [1x1 struct]
  TestAndProfile: [1x1 struct]

Follow these steps to configure the tracker object properties:

1. View the Raspberry Pi target board details.

tracker.BoardDetails
ans =
struct with fields:
           Name: 'Raspberry Pi'
  DeviceAddress: '192.168.0.100'
       UserName: 'pi'
       Password: 'raspberry'

This structure shows the name, device address, user name, and password of the Raspberry Pi board. You can change the values by using dot notation. For example, change the password to MATLAB.

tracker.BoardDetails.Password = 'MATLAB';

Check your change by viewing the properties again.

tracker.BoardDetails
ans =
struct with fields:
           Name: 'Raspberry Pi'
  DeviceAddress: '192.168.0.100'
       UserName: 'pi'
       Password: 'MATLAB'

The structure shows the updated password.

2. View the image capture setup properties.

tracker.Setup
ans =
struct with fields:
  DataCaptureTime: 120
  CameraInterface: 'webcam'

This structure shows the type of interface and the capture time.

  • DataCaptureTime- Time duration in seconds for image capture of the tracked object. The default time is 120 seconds. You can change the values by using dot notation. You can increase the capture time to improve the training efficiency of the neural network for an object to detect and track. Change the capture time to 300 seconds.

tracker.Setup.DataCaptureTime = 300;
  • CameraInterface- Type of camera interface used to detect and track the object, webcam or cameraboard. The default interface is webcam.

tracker.Setup.CameraInterface = 'webcam';

Place the object to detect and track in front of the camera and run this command at the MATLAB Command Window:

objectTrackingSetup(tracker)

The camera captures images for the time specified in the tracker.Setup.DataCaptureTime field. These images are saved in the Data<Date><Time Stamp> folder in the current working directory.

Copy one sample image from the Data<Date><Time Stamp> folder and paste it in the current working directory. Note the file name so that you can use it to set the tracker.Detector.SampleImage property.

Running the objectTrackingSetup(tracker) command also opens the Video Labeler app. This app allows you to mark the region of interest (ROI), automatically label across image frames using an automation algorithm, and export the labeled ground truth.

Follow these steps in the Video Labeler app:

  1. Label the object. In this example, the object has the label ball.

  2. Mark the ROI in the image.

  3. Select the Point Tracker algorithm and click Automate. The algorithm instructions appear in the right pane, and the selected labels are available to automate.

  4. Click Run to automate labeling for the the image sequence.

  5. When you are satisfied with the algorithm results, click Accept.

  6. Select Export Labels > To File. The labeled ground truth is exported to a MAT file, objectLabel.mat.

For detailed information on how to use the Video Labeler app, see Video Labeler and Get Started with the Video Labeler.

B. Detection Phase

In this phase, the captured ground truth data collected from the Setup phase is used to train the ResNet-18 neural network and verify the detected object in Connected IO and PIL modes.

Check that the LabeledData value is the name of the exported MAT file by running the tracker.Detector command.

tracker.Detector
ans =
struct with fields:
      LabeledData: 'objectLabel.mat'
        ImageSize: [224 224 3]
       NumClasses: 1
      SampleImage: 'sampleImage.jpg'
          Network: [1x1 struct]
  TrainingOptions: [1x1 struct]
  DetectorMatFile: 'detectorSaved.mat'

Train YOLOv2 Object Detector

Train the YOLOv2 object detector with the images captured from the camera in the Setup phase. The objectLabel.mat file contains the exported ground truth data. Use this object file to train the ResNet-18 neural network.

From the current directory of the captured object images, select a valid sample image as a reference for training the neural network. Set the sample image to image_50.png.

tracker.Detector.SampleImage = 'image_50.png';

Train the ResNet-18 neural network and save it as the MAT file, detectorSaved.mat.

createObjectDetector(tracker)

A figure window opens of the selected sample image. This image shows an ROI and the probability of match within the training network.

Note: You can verify the object detection in either Connected IO or PIL mode.

Verify Object Detection in Connected IO Mode

Use Connected IO mode to verify the detected object. The image capture process takes place on the Raspberry Pi hardware itself.

Run this command at the MATLAB Command Window:

tracker.TestAndProfile
ans =
struct with fields:
  PredictFunction: 'raspi_yolov2_detect'
         TestMode: 'IO'
     TestDuration:  60
            Servo: [1x1 struct]

The default TestMode value is 'IO'.

Run this command at the MATLAB Command prompt to verify the object detection in IO mode:

startTestAndProfile(tracker)

A camera window opens showing an ROI and the probability of match within the training network for the detected object.

Verify Object Detection in PIL Mode

Use PIL mode to verify the detected object. The image capture process takes place on the Raspberry Pi hardware itself. The raspi_yolov2_detect function runs on the Raspberry Pi board in the PIL mode.

Note: The object detection and tracking in the PIL mode takes a while to execute.

Change the object detection mode to 'PIL'.

tracker.TestAndProfile.TestMode = 'PIL';

Run this command at the MATLAB Command prompt to verify the object detection in PIL mode.

startTestAndProfile(tracker)

A camera window opens showing an ROI and the probability of match within the training network for the detected object.

To view the Code Execution Profiling Report, click the Execution profiling report link.

C. Tracking Phase

In this phase, you configure the servo motor and track the moving object that has been previously detected.

Configure Servo Motor Parameters

To track an object after its successful detection, you must configure the servo motor parameters.

View the servo motor parameters.

tracker.TestAndProfile.Servo
ans =
struct with fields:
     TestWithServo: 0
         Increment: 0.5000
         PinNumber: 12
     StartPosition: 90
  MinPulseDuration: 5.0000e-04
  MaxPulseDuration: 0.0025

1. TestWithServo- Flag to enable or disable the servo motor for tracking the object. The default value is false. Enable the servo motor.

tracker.TestAndProfile.Servo.TestWithServo = true;

2. Increment- Step angle size of rotation of the servo motor. The default step angle value is 0.5 degrees.

3. PinNumber- GPIO pin number of the Raspberry Pi target board to which the servo motor is connected.

4. StartPosition- Starting angle position of the servo motor. The servo motor rotates from 0 to 180 degrees. The default starting angle is 90 degrees.

Set MinPulseDuration and MaxPulseDuration from the servo motor datasheet.

Track Object in Connected IO Mode

To track the detected object in connected IO mode, ensure that:

tracker.TestAndProfile.Servo.TestWithServo = true;
tracker.TestAndProfile.TestMode = 'IO';

Run this command at the MATLAB Command prompt:

startTestAndProfile(tracker)

Place the object in front of the camera and move the object. Observe that the servo motor rotates to follow the moving object.

A camera window opens surrounded with the ROI and the probability of match with the training network. Also, another window opens to display the angle of servo motor movement.

Track Object in PIL Mode

To track the detected object in PIL mode, ensure that:

tracker.TestAndProfile.Servo.TestWithServo = true
tracker.TestAndProfile.TestMode = 'PIL';

Run this command at the MATLAB Command prompt:

startTestAndProfile(tracker)

Place the object in front of the camera and move the object. Observe that the servo motor rotates to follow the moving object.

A camera window opens surrounded with the ROI and the probability of match with the training network. A separate window opens to display the angle of the servo motor movement.

Deploy on Raspberry Pi Target Board

Before deploying the code on the Raspberry Pi target board, open the raspi_object_tracking.m function file and configure the parameters based on:

  • Input image size for the ResNet-18 neural network

  • Observations regarding the servo motor incremental angle, starting position, and so on

  • Type of camera interface for capturing images of object

Open the raspi_yolov2_detect.m file and enter the name of the saved trained neural network MAT file detectorSaved.mat in the yolov2obj parameter.

Run these commands at the MATLAB command prompt.

t = targetHardware('Raspberry Pi')
t.CoderConfig.TargetLang = 'C++';
dlcfg = coder.DeepLearningConfig('arm-compute');
dlcfg.ArmArchitecture = 'armv7';
t.CoderConfig.DeepLearningConfig = dlcfg;
deploy(t,'raspi_object_tracking')

Observe that the camera mounted on the servo motor detects the object and also tracks its movement. On the Raspberry Pi desktop, open the camera display to observe the live tracking results.

The deployed function initiates code generation of the raspi_object_tracking function. Once code generation is complete, MATLAB generates a code generation report. Use this report to debug the raspi_object_tracking function for any build errors or warnings in the generated code.

After successfully generating the code, the support package loads and runs the object classification algorithm as a standalone executable on the hardware. The executable starts detecting the objects in the acquired video and displays the predicted labels and their associated probabilities. To view the Raspberry Pi screen, use a VNC viewer and open a remote session on the hardware to get the display. You can alternatively connect an HDMI cable from the monitor to the hardware.

Other Things to Try

  • Train the ResNet-18 neural network to detect and track more than one object.

  • Use a different ResNet-18 neural network for training the objects and observe the differences in the results.

  • Use a different algorithm in the Video Labeler app and compare the results with Point Tracker algorithm.

  • Explore the raspi_yolov2_detect function.

See Also

Identify Objects Within Live Video Using ResNet-50 on Raspberry Pi Hardware