Calibration and Sensor Fusion

Interactively perform lidar-camera calibration, estimate transformation matrix, and fuse data from multiple sensors

Most modern autonomous systems in applications such as manufacturing, transportation, and construction, employ multiple sensors. Sensor Fusion is the process of bringing together data from multiple sensors, such as radar sensors, lidar sensors, and cameras. The fused data enables greater accuracy because it leverages the strengths of each sensor to overcome the limitations of the others.

To understand and correlate the data from individual sensors, you must develop a geometric correspondence between them. Calibration is the process of developing this correspondence. Use Lidar Toolbox™ functions to perform lidar-camera calibration. To get started, see What Is Lidar-Camera Calibration?

You can also interactively calibrate the sensors by using the Lidar Camera Calibrator app. For more information, see Get Started with Lidar Camera Calibrator.

Lidar Toolbox also supports downstream workflows such as projecting lidar points on images, fusing color information in lidar point clouds, and transferring bounding boxes from camera data to lidar data.

Apps

Lidar Camera Calibrator

Interactively establish correspondences between lidar sensor and camera to fuse sensor data (Since R2021a)

Functions

expand all

Detect Calibration Parameters

`estimateCheckerboardCorners3d`	Estimate world frame coordinates of checkerboard corner points in image (Since R2020b)
`detectRectangularPlanePoints`	Detect rectangular plane of specified dimensions in point cloud (Since R2020b)

Calibrate Lidar and Camera

estimateLidarCameraTransform Estimate rigid transformation from lidar sensor to camera (Since R2020b)

Fuse Sensor Data

`projectLidarPointsOnImage`	Project lidar point cloud data onto image coordinate frame (Since R2020b)
`fuseCameraToLidar`	Fuse image information to lidar point cloud (Since R2020b)
`bboxCameraToLidar`	Estimate 3-D bounding boxes in point cloud from 2-D bounding boxes in image (Since R2020b)
`bboxLidarToCamera`	Estimate 2-D bounding box in camera frame using 3-D bounding box in lidar frame (Since R2021a)

Topics

What Is Lidar-Camera Calibration?
Fuse lidar and camera data.
Calibration Guidelines
Guidelines to help you achieve accurate results for lidar-camera calibration.
Coordinate Systems in Lidar Toolbox
Overview of coordinate systems in Lidar Toolbox.
Get Started with Lidar Camera Calibrator
Interactively calibrate lidar and camera sensors.
Read Lidar and Camera Data from Rosbag File
This example shows how to read and save images and point cloud data from a rosbag file.

Featured Examples

Lidar and Camera Calibration

Estimate a rigid transformation between a 3-D lidar sensor and a camera, then use the rigid transformation matrix to fuse the lidar and camera data.

Open Live Script

Multi-Lidar Calibration

Calibrate multiple 3-D lidar sensors mounted on a vehicle to estimate a relative transformation between them. Traditional methods, such as marker-based registration, are difficult when the lidar sensors have a negligible overlap between their fields of view (FOVs). The calibration also becomes more difficult as the number of lidar sensors increases. This example demonstrates the use of the trajectories of individual lidar sensors to estimate the transformation between them. This method of calibration is also known as hand-eye calibration.

Open Live Script

Detect Vehicles in Lidar Using Image Labels

Detect vehicles in lidar using label data from a co-located camera with known lidar-to-camera calibration parameters. Use this workflow in MATLAB® to estimate 3-D oriented bounding boxes in lidar based on 2-D bounding boxes in the corresponding image. You will also see how to automatically generate ground truth as a distance for 2-D bounding boxes in a camera image using lidar data. This figure provides an overview of the process.

Open Live Script

Track-Level Fusion of Radar and Lidar Data

Generate an object-level track list from measurements of a radar and a lidar sensor and further fuse them using a track-level fusion scheme. You process the radar measurements using an extended object tracker and the lidar measurements using a joint probabilistic data association (JPDA) tracker. You further fuse these tracks using a track-level fusion scheme. The schematic of the workflow is shown below.

Open Script