Convert 2d image to 3d

Hi;

I'm trynig to convert 2d image to 3d with the intrinsic matrix, I expect to get in the x-axis the distance in millimeters and do not understand why it does not work. hope someone can help..

Answer 1

[Matt J]

You cannot recover a 3D point from only a single 2D projection of that point. This is because every point [X,Y,Z] along the line of sight from the camera to [u,v] also maps to [u,v] according to the equation you've shown. Because it is a many-to-one mapping from [X,Y,Z] to [u,v], you cannot invert the equation uniquely.

[Dekel Mashiach]

Do you have any idea how can I do this?

[Matt J]

As we discussed in previous posts, you need a view of [X,Y,Z] from the perspective of an additional camera. Then, you can use triangulate.

[Matt J]

Alternatively, if you know that your 3D point lies in a particular 3D plane, then you will have an additional equation which can be used to with your projection equation to solve uniquely for the 3D point.

[Dekel Mashiach]

sorry I don't understand the additional equation, how can I use it ? ;

(I have to use in only one camera- so triangulate can't help)

[Matt J]

The 2D point [u;v;1] backprojects to a 3D line, which as we have said contains many points, and makes the solution non-unique. If C is the camera center, then the line is given by,

L(s) = C+s*K^-1[u;v;1]

However, if you have additional information that the 3D point you're looking for lies in a particular 3D plane, then you just have to find the intersection of the backprojected line L(s) with that plane, and it will give you a unique point.

[Dekel Mashiach]

And if I set the coordinates of 3d (x y z) in milimeters ,will I get the system in 2d point (pixels)?

[Matt J]

You should get a dimensionless equation in s.

[Dekel Mashiach]

what is s?

[Matt J]

The parameter of the 3D line equation in my previous post.

L(s) = C+s*K^-1[u;v;1]

This is the 3D line obtained from backprojecting [u,v]

[Dekel Mashiach]

My plan was to find the factor s one time in the reverse way and then place it in order to find the 3d as I wanted at first.

C is the camera center' it's mean if my image is 720*1280, than C is [360;640]?

[Matt J]

No C is a 3x1 vector of extrinsic parameters. It is the 3D location of the camera focal spot.

https://en.wikipedia.org/wiki/Pinhole_camera_model#Geometry

[Dekel Mashiach]

what I get here is s*[u;v;1] ?

[Matt J]

Your attached image seems to just show your K matrix.

[Dekel Mashiach]

I try to get 2d coordinates by : k*[x;y;z]

my line is only in x and is 400 milimeters so [400;0;0]

[Matt J]

Trying to get 2d coordinates? In your original post, you say you are trying to get 3d coordinates (from 2d coordinates). I don't know what the input and desired output is anymore.

[Dekel Mashiach]

My project is an autonomous vehicle that tracks a route using an image (the green line in the image at the top of the post). I need for my controller the route that the vehicle should travel, so I try to convert from 2d to 3d.

I realized I could not get to 3D coordinates as I wanted at first. So I try to reverse - that is, I actually measured the length of the line, I placed it in the equation as [400; 0; 0] and I try to take out the 2 d * s.

[Matt J]

Going from 3D to 2D is a much easier thing. You just apply the equation in your posted question to map from [X,Y,Z] to [u,v]. Everything on the right hand side of the equation is known, I assume.

[Dekel Mashiach]

I dont undertand why I get this results (in v2d.jpg)...

[Matt J]

What's wrong with it?

[Dekel Mashiach]

When I place 400[mm] I expect to get the pixels in the green line in the image I posted at the beginning(y=722 x=686)

[Matt J]

Multiplying by the 3x3 intrinsic matrix K does not map 3D to 2D coordinates. Notice in the camera projection equation in your post, there is a 3x4 matrix of extrinsic parameters as well. Also, when you apply the camera matrix, it is with homogeneous coordinate vectors. You need to normalize the final vector component to 1 to retrieve the inhomogeneous u,v values, e.g.,

P=rand(3,4); %fake camera matrix

XYZ=[400;0;0] %3D coordinates (inhomogeneous)

XYZ = 3×1

400 0 0

uv_hom=P*[XYZ;1] %apply camera projection - gives homogeneous 2D point

uv_hom = 3×1

42.6613 211.8373 126.0988

uv=uv_hom(1:2)/uv_hom(3) %normalize to obtain inhomogeneous 2D point

uv = 2×1

0.3383 1.6799

[Dekel Mashiach]

still doesn't work well..

[Matt J]

I doubt your R_T is really eye(3,4). If that were true, the point XYZ=[400;0;0] would be somewhere to the far right of and level with the camera. You should probably use the extrinsics command to get the actual extrinsics of your camera. Also, you should consider using worldToImage and pointsToWorld to map between 2D and 3D. Since they take as input many of hte objects that are already available inside your cameraParameters object, it might be easier to use.

[Dekel Mashiach]

Hey; do you know why the pricipal point in my instrinsic matrix is (3,1) (3,2) and not (1,3) (2,3)?

[Matt J]

Yes, because Matlab Toolboxes like to organize coordinate vectors as row vectors and to apply transformations to them by multiplying transformatiion matrices on the right (i.e. x'*K' instead of K*x). Therefore, the intrinisc matrix as generated by Matlab is the transpose of the way you normally see it in textbooks.

Did my previous remarks resolve your question? If so, please Accept-click the answer.

[Dekel Mashiach]

You have helped me a lot and I am very thankful to you ,it is not obvious all the patience you have in giving help! But unfortunately I still can not solve it and I will be really messed up ... I found that Z=262 (u=fX/Z , v=fY/Z), But I still can't write the code in a way that it will work .. If you can help me I would be happy...

XYZ = [400;0;0];
R_T = eye(3,4);
k = cameraParams2.Intrinsics.IntrinsicMatrix
k =

648.7306         0           0
0           651.8988         0
311.8627     242.7315    1.0000

uv = k*R_T*[XYZ;1];
uv_n = uv(1:2)/uv(3);

[Matt J]

Yes, but you appear not to have implemented any of my suggestions. R_T=eye(3,4) is wrong and k needs to be transposed. Use extrinsics() to get the real extrinsic data and use worldToImage() to apply it.

[Dekel Mashiach]

I need to use with the equationI attached at the beginning of the post, can you please show me how I should write down the equation and position the data correctly? (Assuming that R_T = eye (3,4))

[Matt J]

uv = k'*R_T*[XYZ;1];

[Dekel Mashiach]

ok, but I don't need to do: u=fX/Z , v=fY/Z after?

[Matt J]

Yes, you do.

[Dekel Mashiach]

So basically the u, v I will get I have to subtract from them the principal point and then divide them by Z?

I have another problem when I do k', it causes after normalization Inf, NaN. you know how to solve it?

[Matt J]

It happens because you are assuming R_T=eye(3,4) which is wrong.

[Dekel Mashiach]

Given that I placed the correct R_T, after normalizing what is the next step I need to do for u, v?

[Matt J]

Everything you have now is correct except for R_T.