How to calibrate Fisheye Camera and Lidar?

Hi, I have been trying to calibrate a fisheye camera and a 3D lidar for object detection, but the fisheye parameters matrix doesn't seem to be working with functions such as estimateCheckerboardCorners3d, it expects an object type of cameraParams or cameraIntrinsics while the estimateFisheyeParameters function doesn't provide these, are there any workarounds for lidar-fisheye camere calibration? Can the estimateCameraParameters be used for fisheye camera for example, my experience is that the results are not great with it.

[Atacan Aslan]

A work around I found is that you can use the undistorted images to calibrate and undistort the image in real time for lidar - camera fusion.

[Bjorn Gustavsson]

Depending on your use-case I might have tools to calibrate and analyze fish-eye optics. What are you looking at?

[Atacan Aslan]

I am trying to match the board in the fisheye image and lidar scan, and use the transform matrix for real time object detection using ROS, MATLAB tools for camera-lidar calib. requre cameraParameter object but the calibration for fisheye cameras doesn't proive this type and the content seems different. A method to get the same object would be useful, or other tools to calibrate lidar and camera with fisheye.

I am currently undistorting the fisheye image, calibrating images like a pinhole camera and applying the lidar-camera calib. But I don't know if it is a reliable method.

[Bjorn Gustavsson]

I have developed geometric camera calibration-tools for imaging of airglow and aurora. For that I use the known celestial positions of stars identified in images. If that sounds familiar to you you might be able to use those tools.

Answer 1

[Pooja Kumari]

Hi Atacan,

It is my understanding that you are facing issues with calibrating a fisheye camera and a 3D lidar for object detection, but the fisheye parameters matrix doesn't seem to be working with "estimateCheckerboardCorners3d" and "estimateFisheyeParameters".

One possible workaround is to use the "estimateCameraParameters" function with a fisheye camera. Although this function is primarily designed for pinhole cameras, it can also be used with fisheye cameras. However, the results may not be as accurate as with the "estimateFisheyeParameters".

Another option is to manually convert the fisheye camera parameters to the required format. You can extract the distortion coefficients and the camera matrix from the output of "estimateFisheyeParameters" and create a custom "cameraParams" or "cameraIntrinsics" object. This object should contain the necessary parameters for the "estimateCheckerboardCorners3d".

After obtaining the "cameraParameters"object, you may use the "estimateCameraParameters"function or the "Stereo Camera Calibrator App" to continue with the calibration process. As a result, you will have a calibrated camera model that you may use for lidar-camera calibration and image rectification. You can refer to the below documentation for more information on Stereo Camera Calibrator:

https://www.mathworks.com/help/vision/ug/using-the-stereo-camera-calibrator-app.html

Remember that because a fisheye camera and a lidar have different properties and measurement principles, calibrating them together can be challenging. It is important to carefully consider the calibration process and ensure that the resulting calibration is suitable for your specific application.

I hope this explanation clarifies the steps for calibrating your fisheye camera and lidar for object detection!