Main Content

Automate Ground Truth Labeling For Vehicle Detection Using PointPillars

This example shows how to automate vehicle detections in a point cloud using a pretrained PointPillars object detection network in the Lidar Labeler. In this example, you use the AutomationAlgorithm interface to automate labeling in the Lidar Labeler app.

Lidar Labeler App

Good ground truth data is crucial for developing automated driving algorithms and evaluating performance. However, creating and maintaining a diverse, high-quality, and labeled data set requires significant effort. The Lidar Labeler app provides a framework to automate the labeling process using the AutomationAlgorithm interface. You can create a custom algorithm and use it in the app to label your entire data set. You can also edit the results to account for challenging scenarios missed by the algorithm.

In this example, you:

  • Use a pretrained PointPillars object detection network to detect objects of class 'vehicle'.

  • Create an automation algorithm that you can use in the Lidar Labeler app to automatically label vehicles in the point cloud using the PointPillars network.

Detect Vehicles Using PointPillars Network

Detect vehicles in a point cloud using a pretrained PointPillars object detection network. This network has been trained to detect vehicles in a point cloud. For information on how to train a PointPillars network yourself, see Lidar 3-D Object Detection Using PointPillars Deep Learning. Network performance depends on how generalizable the network is. The network might not perform well when it is applied to unseen data. Iteratively introducing custom training data to the learning process can improve the performance on similar data sets.

Download the PointPillars network, which is trained on the WPI data set.

pretrainedNetURL = 'https://ssd.mathworks.com/supportfiles/lidar/data/trainedPointPillars.zip';
preTrainedMATFile = fullfile(tempdir,'trainedPointPillarsNet.mat');
preTrainedZipFile = fullfile(tempdir,'trainedPointPillars.zip');
    
if ~exist(preTrainedMATFile,'file')
    if ~exist(preTrainedZipFile,'file')
        disp('Downloading pretrained detector (8.3 MB)...');
        websave(preTrainedZipFile, pretrainedNetURL);
    end
    unzip(preTrainedZipFile, tempdir);   
end

Depending on your internet connection, the download process can take some time. The code suspends MATLAB® execution until the download process is complete. Alternatively, you can download the data set to your local disk using your web browser and extract the file.

Use the network to detect vehicles in the point cloud by following these steps.

  • Add a folder containing the helper functions to the search path.

  • Read the point cloud from lidardata.

  • Specify anchor boxes which are predefined bounding boxes in the format {length, width, height, z-center, yaw angle}.

  • Specify grid parameters to crop the full-view point cloud to front-view in the format {{xMin, yMin, zMin}, {xMax, yMax, zMax}, {xStep, yStep, dsFactor}, {Xn, Yn}} where Xn = round(((xMax - xMin) / xStep)) and Yn = round(((yMax - yMin) / yStep)).

  • Specify the label name for the detected object.

  • Specify the confidence threshold to use only detections with confidence scores above this value.

  • Specify the overlap threshold to remove overlapping detections.

  • Select prominent pillars (P) based on the number of points per pillar (N).

  • Set executionEnvironment.

  • Use the helperCropFrontViewFromLidarData helper function, attached to this example as a supporting file, to crop the point cloud.

  • Use the generatePointPillarDetections helper function from the added search path, to get the bounding boxes.

  • Display the point cloud with bounding boxes.

% Add folder to path.
addpath(fullfile(matlabroot,'examples','deeplearning_shared','main'));
    
% Load the pretrained network.
pretrainedNet = load(preTrainedMATFile);

% Load a point cloud.
ptCloud = pcread(fullfile(toolboxdir('lidar'),'lidardata','highwayScene.pcd'));

% Anchor boxes.
anchorBoxes = {{3.9, 1.6, 1.56, -3.6, 0}, {3.9, 1.6, 1.56, -3.6, pi/2}};

% Cropping parameters.
gridParams = {{0.0,-39.68,-5.0},{69.12,39.68,5.0},{0.16,0.16,2.0},{432,496}};

% Label name for detected object.
classNames = {'vehicle'};

% Confidence threshold. 
confidenceThreshold = 0.45;

% Overlap threshold. 
overlapThreshold = 0.1;

% Number of prominent pillars.
P = 12000;

% Number of points per pillar.
N = 100;

% Set the execution environment.
executionEnvironment = "auto";

% Crop the front view of the point cloud. 
processedPointCloud = helperCropFrontViewFromLidarData(ptCloud, gridParams);

% Detect the bounding boxes.
[box, ~, ~] = generatePointPillarDetections(pretrainedNet.net, processedPointCloud, ...
                      anchorBoxes, gridParams, classNames, confidenceThreshold, ...
                      overlapThreshold, P, N, executionEnvironment);

% Display the detections on the point cloud.
figure
ax = pcshow(processedPointCloud.Location);
showShape('cuboid',box,'Parent',ax,'Opacity',0.1,'Color','red','LineWidth',0.5)
hold on
zoom(ax, 1.5)
title("Detected Vehicle on Point Cloud")

Prepare Lidar Vehicle Detector Automation Class

Construct an automation class for the lidar vehicle detector algorithm. The class inherits from the vision.labeler.AutomationAlgorithm abstract base class. The base class defines properties and signatures for methods that the app uses to configure and run the custom algorithm. The Lidar Labeler app provides an initial automation class template. For more information, see Create Automation Algorithm for Labeling. The LidarVehicleDetector class is based on this template and provides you with a ready-to-use automation class for vehicle detection in a point cloud. The comments of the class outline the basic steps needed to implement each API call.

Algorithm Properties

Step 1 contains properties that define the name and description of the algorithm and the directions for using the algorithm.

    % ----------------------------------------------------------------------
    % Step 1: Define the required properties describing the algorithm. This
    % includes Name, Description, and UserDirections.
    properties(Constant)
        
        % Name Algorithm Name
        %   Character vector specifying the name of the algorithm.
        Name = 'Lidar Vehicle Detector';
        
        % Description Algorithm Description
        %   Character vector specifying the short description of the algorithm.
        Description = 'Detect vehicles in point cloud using PointPillars network';
        
        % UserDirections Algorithm Usage Directions
        %   Cell array of character vectors specifying directions for
        %   algorithm users to follow to use the algorithm.
        UserDirections = {['ROI Label Definition Selection: select one of ' ...
            'the ROI definitions to be labeled'], ...
            ['Run: Press RUN to run the automation algorithm. '], ...
            ['Review and Modify: Review automated labels over the interval ', ...
            'using playback controls. Modify/delete/add ROIs that were not ' ...
            'satisfactorily automated at this stage. If the results are ' ...
            'satisfactory, click Accept to accept the automated labels.'], ...
            ['Change Settings and Rerun: If automated results are not ' ...
            'satisfactory, you can try to re-run the algorithm with ' ...
            'different settings. To do so, click Undo Run to undo ' ...
            'current automation run, click Settings, make changes to Settings,' ...
            'and press Run again.'], ...
            ['Accept/Cancel: If the results of automation are satisfactory, ' ...
            'click Accept to accept all automated labels and return to ' ...
            'manual labeling. If the results of automation are not ' ...
            'satisfactory, click Cancel to return to manual labeling ' ...
            'without saving the automated labels.']};
    end

Custom Properties

Step 2 contains the custom properties needed for the core algorithm.

    % ---------------------------------------------------------------------
    % Step 2: Define properties you want to use during the algorithm
    % execution.
    properties
        
        % SelectedLabelName 
        %   Name of the selected label. Vehicles detected by the algorithm 
        %   are assigned this variable name.
        SelectedLabelName
        
        % PretrainedNetwork
        %   PretrainedNetwork saves the pretrained PointPillars dlnetwork.
        PretrainedNetwork
        
        %   Range of point clouds along x,y,and z-axis used to crop
        %   full-view point clouds to front-view point clouds.
        %   These parameters guide the calculation of the size of the 
        %   input [xn,yn] passed to the network.

        %   xMin = 0.0;     % Minimum value along X-axis.
        %   yMin = -39.68;  % Minimum value along Y-axis.
        %   zMin = -5.0;    % Minimum value along Z-axis.
        %   xMax = 69.12    % Maximum value along X-axis.
        %   yMax = 39.68;   % Maximum value along Y-axis.
        %   zMax = 5.0;     % Maximum value along Z-axis.
        %   xStep = 0.16;   % Resolution along X-axis.
        %   yStep = 0.16;   % Resolution along Y-axis.
        %   dsFactor = 2.0; % Downsampling factor.
        
        %  Dimensions for the pseudo-image calculated as 
        %  round(((xMax - xMin) / xStep));.
        %  Xn = 432;
        %  Yn = 496;
        
        % GridParams 
        %  Parameter used to crop full-view point cloud to front-view,
        %  defined as {{xMin,yMin,zMin}, {xMax,yMax,zMax},
        %  {xStep,yStep,dsFactor}, {Xn,Yn}};
        GridParams = {{0.0,-39.68,-5.0}, {69.12,39.68,5.0}, {0.16,0.16,2.0}, {432,496}}
        
        % AnchorBoxes 
        %  Predefined bounding box dimensions based on the classes to
        %  detect, defined in the format {length, width, height,
        %  z-center, yaw angle}
        AnchorBoxes = {{3.9, 1.6, 1.56, -1.78, 0}, {3.9, 1.6, 1.56, -1.78, pi/2}};
        
        % P
        %  Number of prominent pillars.
        P = 12000;
        
        % N
        %  Number of points per pillar.
        N = 100;
        
        % ExecutionEnvironment
        %  Set the execution environment.
        ExecutionEnvironment = "auto";
        
        % ConfidenceThreshold
        %  Specify the confidence threshold to use only detections with 
        %  confidence scores above this value.
        ConfidenceThreshold = 0.45;
        
        % OverlapThreshold
        %  Specify the overlap threshold to remove overlapping detections.
        OverlapThreshold = 0.1;            
        
    end

Function Definitions

Step 3 deals with function definitions.

The checkSignalType function checks if the signal data is supported for automation. The lidar vehicle detector supports signals of type PointCloud.

        function isValid = checkSignalType(signalType)            
            % Only point cloud signal data is valid for the Lidar Vehicle
            % detector algorithm.
            isValid = (signalType == vision.labeler.loading.SignalType.PointCloud);           
        end

The checkLabelDefinition function checks if the label definition is the appropriate type for automation. The lidar vehicle detector requires the Cuboid label type.

        function isValid = checkLabelDefinition(~, labelDef)            
            % Only cuboid ROI label definitions are valid for the Lidar
            % vehicle detector algorithm.
            isValid = labelDef.Type == labelType.Cuboid;
        end

The checkSetup function checks if an ROI label definition is selected for automation.

        function isReady = checkSetup(algObj)            
            % Is there one selected ROI Label definition to automate.
            isReady = ~isempty(algObj.SelectedLabelDefinitions);
        end

The settingsDialog function obtains and modifies the properties defined in Step 2. This API call lets you create a dialog box that opens when you click the Settings icon in the Automate tab. To create this dialog box, use the dialog function to quickly create a simple modal window to optionally modify the confidence and overlap thresholds. The lidarVehicleDetectorSettings method contains the code for settings and input validation steps.

        function settingsDialog(algObj)
            % Invoke dialog with options for modifying the 
            % threshold and confidence threshold. 
            lidarVehicleDetectorSettings(algObj)
        end

Execution Functions

Step 4 specifies the execution functions. The initialize function populates the initial algorithm state based on the existing labels in the app. In this example, the initialize function performs the following steps:

  • Store the name of the selected label definition.

  • Add the folder containing helper functions to the search path.

  • Load the pretrained PointPillars object detection network from tempdir and save it to the PretrainedNetwork property.

        function initialize(algObj,~)           
            % Store the name of the selected label definition. Use this
            % name to label the detected vehicles.
            algObj.SelectedLabelName = algObj.SelectedLabelDefinitions.Name;
            
            % Add the folder containing helper functions to the search path.
            addpath(fullfile(matlabroot,'examples','deeplearning_shared','main'))

            % Point to tempdir, where pretrainedNet was downloaded.
            preTrainedMATFile = fullfile(tempdir,'trainedPointPillarsNet.mat');
            assert(exist(preTrainedMATFile,'file'), ...
                sprintf(['File : %s does not exists \n Download Pretrained PointPillars ' ...
                'network with the link provided in the example'],preTrainedMATFile));
            pretrainedNet = load(preTrainedMATFile);
            algObj.PretrainedNetwork = pretrainedNet.net;           
        end

The run function defines the core lidar vehicle detector algorithm of this automation class. The run function is called for each frame of the point cloud sequence, and expects the automation class to return a set of labels. You can extend the algorithm to any category the network is trained on. For the purposes of this example, restrict the network to detect objects of class 'Vehicle'.

        function autoLabels = run(algObj, pointCloud)
            bBoxes = [];
            for i = 1:2           
                if i == 2
                    % Rotate the point cloud by 180 degrees.
                    thetha = pi;
                    rot = [cos(thetha) sin(thetha) 0; ...
                        -sin(thetha) cos(thetha) 0; ...
                        0  0  1];
                    trans = [0, 0, 0];
                    tform = rigid3d(rot, trans);
                    pointCloud = pctransform(pointCloud, tform);
                end
                
                % Crop the front view of the point clouds.
                processedPointCloud = helperCropFrontViewFromLidarData(pointCloud, ...
                    algObj.GridParams);
                
                % Detect vehicles using the PointPillars network.
                [box, ~, ~] = generatePointPillarDetections(algObj.PretrainedNetwork,  ...
                    processedPointCloud, algObj.AnchorBoxes, algObj.GridParams, ...
                    {algObj.SelectedLabelName}, algObj.ConfidenceThreshold, ...
                    algObj.OverlapThreshold, algObj.P, algObj.N, algObj.ExecutionEnvironment);

                if ~isempty(box)
                    if i == 2
                        box(:,1) = -box(:,1);
                        box(:,2) = -box(:,2);
                        box(:,9) = -box(:,9);
                    end
                    bBoxes = [bBoxes;box];
                end
            end
            if ~isempty(bBoxes)
                % Add automated labels at bounding box locations detected
                % by the vehicle detector, of type Cuboid and with the name
                % of the selected label.
                autoLabels.Name     = algObj.SelectedLabelName;
                autoLabels.Type     = labelType.Cuboid;
                autoLabels.Position = bBoxes;
            else
                autoLabels = [];
            end
        end

The terminate function handles any cleanup or tear-down required after the automation is done. This algorithm does not require any cleanup, so the function is empty.

Use Lidar Vehicle Detector Automation Class in App

The properties and methods described in the previous section are implemented in the LidarVehicleDetector automation algorithm class file. Use the class in the app.

First, create the folder structure +vision/+labeler required under the current folder, and copy the automation class into it.

    mkdir('+vision/+labeler');
    copyfile(fullfile(matlabroot,'examples','lidar','main','LidarVehicleDetector.m'), ...
        '+vision/+labeler');

Download the point cloud sequence (PCD) and image sequence. For illustration purposes, this example uses WPI lidar data collected on a highway from an Ouster OS1 lidar sensor. Execute the following code block to download and save the lidar data in a temporary folder. Depending on your internet connection, the download process can take some time. The code suspends MATLAB® execution until the download process is complete. Alternatively, you can download the data set to your local disk using your web browser and extract the file.

Download the point cloud sequence to a temporary location.

    lidarURL = 'https://www.mathworks.com/supportfiles/lidar/data/WPI_LidarData.tar.gz';
    lidarDataFolder = fullfile(tempdir,'WPI_LidarData',filesep);        
    lidarDataTarFile = lidarDataFolder + "WPI_LidarData.tar.gz";

    if ~exist(lidarDataFolder)
        mkdir(lidarDataFolder)
    end

    if ~exist(lidarDataTarFile, 'file')       
        disp('Downloading WPI Lidar driving data (760 MB)...');
        websave(lidarDataTarFile, lidarURL);
        untar(lidarDataTarFile, lidarDataFolder);
    end
    
    % Check if lidar tar.gz file is downloaded, but not uncompressed.
    if ~exist(fullfile(lidarDataFolder, 'WPI_LidarData.mat'), 'file')
        untar(lidarDataTarFile, lidarDataFolder);
    end

The Lidar Labeler app supports the loading of point cloud sequences composed of PCD or PLY files. Save the downloaded point cloud data to PCD files. This example uses only a subset of the WPI point cloud data, from frames 920 to 940.

    % Load downloaded lidar data into the workspace.
    load(fullfile(lidarDataFolder, 'WPI_LidarData.mat'),'lidarData');
    lidarData = reshape(lidarData, size(lidarData,2),1);
    
    % Create new folder and write lidar data to PCD files.
    pcdDataFolder = lidarDataFolder + "lidarData";
    if ~exist(pcdDataFolder, 'dir')
        mkdir(fullfile(lidarDataFolder,'lidarData'));
    end

    disp('Saving WPI Lidar driving data to PCD files ...');
    for i = 920:940
        filename = strcat(fullfile(lidarDataFolder,'lidarData',filesep), ...
            num2str(i,'%06.0f'),'.pcd');
        pcwrite(lidarData{i}, filename);
    end

Open the Lidar Labeler app and load the point cloud sequence.

    pointCloudDir = fullfile(lidarDataFolder, 'lidarData');
    lidarLabeler(pointCloudDir);

In the ROI Labels tab in the left pane, click Label. Define an ROI label with the name Vehicle and the Cuboid. Optionally, you can select a color. Click OK.

Under Select Algorithm, select Refresh list. Then, select Algorithm > Lidar Vehicle Detector. If you do not see this option, verify that the current working folder has a folder called +vision/+labeler, with a file named LidarVehicleDetector.m in it.

Click Automate. The app opens an automation session for the selected signals and displays directions for using the algorithm.

Click Settings, and in the dialog box that opens, modify the parameters if needed and click OK.

Click Run. The created algorithm executes on each frame of the sequence and detects vehicles by using the Vehicle label type. After the app completes the automation run, use the slider or arrow keys to scroll through the sequence to locate the frame where the automation algorithm labeled incorrectly. Use the zoom, pan, and 3-D rotation options for viewing and rotating the point cloud. Manually tweak the results by adjusting the detected bounding boxes or adding new bounding boxes.

When you are satisfied with the detected vehicle bounding boxes for the entire sequence, click Accept. You can then continue to manually adjust labels or export the labeled ground truth to the MATLAB workspace.

You can use the concepts described in this example to create your own custom automation algorithms and extend the functionality of the app.

Helper Functions

helperCropFrontViewFromLidarData

function processedData = helperCropFrontViewFromLidarData(ptCloud, gridParams)
% Function to crop the front view of the point clouds
   % Set the limits for the point cloud.
   [row, column] = find(ptCloud.Location(:,:,1) < gridParams{1,2}{1} ...
       & ptCloud.Location(:,:,1) > gridParams{1,1}{1} ...
       & ptCloud.Location(:,:,2) < gridParams{1,2}{2} ...
       & ptCloud.Location(:,:,2) > gridParams{1,1}{2} ...
       & ptCloud.Location(:,:,3) < gridParams{1,2}{3} ...
       & ptCloud.Location(:,:,3) > gridParams{1,1}{3});
   ptCloud = select(ptCloud, row, column, 'OutputSize', 'full');
   finalPC = removeInvalidPoints(ptCloud);
   processedData = finalPC;           
end