Reconstructing 3D point cloud from two stereo images

Hi everyone!

I've two stereo images taken from Aquifi camera, and I want to compute the disparity and reconstruct the 3D point cloud.

I'd like to help me how to start with this. These are the object, the image taken from the master and the slave cameras.

 Thanks in advance.

Answer 1

[Takuji Fukumoto]

This shows the workfrow to reconstruct point cloud from stereo vision

https://www.mathworks.com/help/vision/examples/depth-estimation-from-stereo-video.html

You can get disparityMap with the folloing function.

disparityMap = disparity(frameLeftGray, frameRightGray);

[Jason Hathaway]

Hi Takuji. Thank you for your answer, I really appreciate it.

I want you to clarify something for me, can I just directly use the command you gave me to compute the disparity? (Like using master as frameLeftGray and slave as frameRightGray? I mean how disparity should look like?

[Takuji Fukumoto]

You have 2 points to confirm.

First, are Stereo camera images well calibrated? if not, you should calibrate camera with

stereoCameraCalibrator

Second, do you have raw images taken with both cameras? You can use 'disparity' command for your gray scale image. But I think amount of their feature is not so mucn. It's better to use raw images including features enough.

[Jason Hathaway]

Hi Takuji. Thank you again for your answer.

The two gray images I provided are calibrated and rectified. But unfortunately, I don't have any images other than what I've already provided in the original post. So is it possible to get stereoParams using only what I have?

Also, I did the following:

% Disparity
I1 = imread('masterRect_1.png');
I2 = imread('slaveRect_1.png');
disparityMap = disparity(I1, I2);
figure;
imshow(disparityMap, [0, 64]);
title('Disparity Map');
colormap jet
colorbar
(Where _masterRect_1.png_ and _slaveRect_1.png_ are the two gray images in the original post)

And I've got the following:

Is it correct?

[Takuji Fukumoto]

I think it's perfect. You can reconstruct the 3-d world with following way. But you need stereoPrams here.

You can get them through Stereo camera calibration process.

points3D = reconstructScene(disparityMap, stereoParams);

% Convert to meters and create a pointCloud object
    points3D = points3D ./ 1000;
    ptCloud = pointCloud(points3D, 'Color', frameLeftRect);

% Create a streaming point cloud viewer
    player3D = pcplayer([-3, 3], [-3, 3], [0, 8], 'VerticalAxis', 'y', ...
        'VerticalAxisDir', 'down');

% Visualize the point cloud
    view(player3D, ptCloud);

[Jason Hathaway]

Thank you again Takuji for your answer, and sorry for the late respond.

First of all, I wanted to ask you whether is it possible to change some parameters to compute the disparity (i.e. block size, the metrics (SAD, NCC) ).

Also, as for now I got the calibration data:

K=[ 4.3841315124560816e+002        0                                          3.2072073046333958e+002 ;
                      0.                              4.3861576800549386e+002    2.4359195157816842e+002 ;
                     0.                                    0.                                          1. ];

and distortion coefficients

d=[ 3.5700104403973414e-002  -2.5696243442005686e-002    1.2897119796380439e-003...
5.9547065657984807e-004        -5.2266254870331104e-002 ]

rotation matrix

R=[  1. 0. 0. ;
       0. 1. 0. ;
       0. 0. 1. ]

and translation vector

T=[  0. ;
       0. ;
      0.  ];

How can I assign them to stereoParams?

[Takuji Fukumoto]

First, you can change some parameters to compute disparity with input argument. block size is included. See here for the details https://jp.mathworks.com/help/vision/ref/disparity.html

You can also change the method, but I can't find NCC. You can also open the 'disparity' function and customize it.

Regarding stereoparams, you can learn its structure here.

https://jp.mathworks.com/help/vision/ref/stereoparameters-class.html

I think you need some more parameters.

[Jason Hathaway]

Hi again! And thank you again for your answers!

As for now, I might have a much more theoretical question. I've used SSD to get disparity. I've used rectangular windows and compared them with their compatible square windows.

Figure 1. 4 x 16

Figure 2. 8 x 8

It looks like the square is much better, but is that the case in theory?

Thanks in advance.

[Takuji Fukumoto]

I think so. Generally, if you use small window size, you can get details but also get noise. This means if you use rectangular window the matching result which is depending on direction.