Rectify image with known coordinates

I have calibration images from four cameras in a 1x4 array. For simplicity, I'm showing the processing for just one camera. The image shows a calibration plate that contains black dots at a know separation.

The image is in pixels. I already post processes the image and obtain a rectification mapping function image coordinates to world coordinates. I input the location of known location in the image in image coordinates and the coordinates of these points in the real world. The output is the function that can take any point in image coordinates and convert it to real world coordinates.

I can plot the points in real world coordinates using the ‘scatter function’ (as in the figure below) but I’m having problems rectifying the whole image. I have no idea how to reshape the image with the new real world coordinates.

Here's the data and code (data was a bit over 5MB).

clear
clc
close all

% I = points in image coordinates
% W = same points as in I but in real world coordinates
% calImg = Raw image in image coordinates
load("calImg.mat")

rectify_quad = cell(length(calImg),1); imagePoints2 = rectify_quad;

% CALIBRATION: compute the image rectification function for this camera
% from the world coords and image coords of the calibration dots (using
% a quadratic transformation: order = 2)
trans_order = 2;
rectify_quad{1} = calibrate_camera(I,W,trans_order);    
imagePoints2{1} = rectify_quad{1}(I);

figure(1)
subplot(121)
imagesc(calImg); hold on
colormap(gray)
scatter(I(:,1), I(:,2), 'r', 'LineWidth',1)
axis equal
xlabel('pixel')
ylabel('pixel')
title('Image and detected points (Image coord)')

subplot(122)
% I want to plot the rectified image in world coordinates similar to subplot (121)
scatter(imagePoints2{1}(:,1),imagePoints2{1}(:,2), 'r', 'LineWidth',1);
axis equal
xlabel('meters')
ylabel('meters')
title('detected points (real world coord)')





function rectify = calibrate_camera(I,W,order)
% calculate the transformation function to convert image coordinates to
% world (physical) coordinates, including calibration (converting pixels to
% meters), undistortion, and rectification
%
% rectify: function handle to map image coordinates to world coordinates 
% I: set of known calibration points in image coordinates (n x 2 vector) [px]
% W: set of known calibration points in world coordinates (n x 2 vector) [m]
% order: 1 for linear transformation (corrects for camera viewing angle but not lens distortion), 
%        2 for quadratic transformation (corrects for camera viewing angle and lens distortion)
%
% references: Fujita et al 1998 (Water Res.), Creutin et al 2003 (J. Hydrol.)
%
% to use function handle: points_m = rectify(points_px)
% points_px: set of points in image coordinates (m x 2 vector) [px]
% points_m: set of points converted to world coordinates (m x 2 vector) [px]


% find transformation coefficients

if order == 2
    A = [I.^2, I, ones(size(I,1),1), -I(:,1).^2.*W(:,1), -I(:,2).^2.*W(:,1), -I(:,1).*W(:,1), -I(:,2).*W(:,1), zeros(size(I,1),5);
        zeros(size(I,1),5), -I(:,1).^2.*W(:,2), -I(:,2).^2.*W(:,2), -I(:,1).*W(:,2), -I(:,2).*W(:,2), I.^2, I, ones(size(I,1),1)];
else
    A = [I, ones(size(I,1),1), -I(:,1).*W(:,1), -I(:,2).*W(:,1), zeros(size(I,1),3);
        zeros(size(I,1),3), -I(:,1).*W(:,2), -I(:,2).*W(:,2), I, ones(size(I,1),1)];
end
Z = [W(:,1); W(:,2)];
B = (A'*A)^-1*A'*Z;
    

% function to map image coords to world coords

if order == 2
    rectify = @(I) [(B(1)*I(:,1).^2 + B(2)*I(:,2).^2 + B(3)*I(:,1) + B(4)*I(:,2) + B(5))./ ...
        (B(6)*I(:,1).^2 + B(7)*I(:,2).^2 + B(8)*I(:,1) + B(9)*I(:,2) + 1), ...
        (B(10)*I(:,1).^2 + B(11)*I(:,2).^2 + B(12)*I(:,1) + B(13)*I(:,2) + B(14))./ ...
        (B(6)*I(:,1).^2 + B(7)*I(:,2).^2 + B(8)*I(:,1) + B(9)*I(:,2) + 1)];

else
    rectify = @(I) [(B(1)*I(:,1) + B(2)*I(:,2) + B(3))./ ...
        (B(4)*I(:,1) + B(5)*I(:,2) + 1), ...
        (B(6)*I(:,1) + B(7)*I(:,2) + B(8))./ ...
        (B(4)*I(:,1) + B(5)*I(:,2) + 1)];
end

end

And here is an figure as reference. I want to rectify the raw image (convert it to real world coordinates) and plot it like in subplot(121). NOTE: the image is upside down in image coordinates. For example, the top-right corner point in the left subplot figure (121) is the bottom-left corner point in the right subplot figure (122).

Answer 1

[Umar]

Hi JCD,

Hi JCD,

Now that you have provided the full code, let me break down your requirements and provide a structured solution.You have calibration images from four cameras, and you're focusing on processing one camera for simplicity. The goal is to rectify the raw image (calImg) from image coordinates (pixels) to real-world coordinates (meters) using a quadratic transformation. You have `I` (points in image coordinates) and `W` (same points in real-world coordinates) for calibration. The function `calibrate_camera` computes the transformation function (`rectify`) based on the specified order (`trans_order = 2` for quadratic transformation). After obtaining `rectify` using calibration data, you apply it to the raw image `calImg` to convert the entire image from image coordinates to real-world coordinates. You've successfully plotted detected points in both image coordinates (`subplot(121)`) and real-world coordinates (`subplot(122)` using scatter plots). Now, you want to visualize the rectified image (`calImg` transformed to real-world coordinates) similar to how `subplot(121)` displays the raw image.Here's how you can proceed with the rectification and visualization:

% Load calibration image and data load('calImg.mat'); % Assuming calImg is loaded from your data file

% Define calibration points (I: image coordinates, W: real-world coordinates) % This is just an example, you need to provide actual values or load them I = [ ... ]; % Image coordinates of calibration points (n x 2) W = [ ... ]; % Real-world coordinates corresponding to I (n x 2)

% Perform calibration to obtain rectification function trans_order = 2; rectify_quad = calibrate_camera(I, W, trans_order);

% Apply rectification function to the entire image % Assuming calImg is the raw image in image coordinates [rows, cols] = size(calImg); [X, Y] = meshgrid(1:cols, 1:rows); % Create meshgrid of image coordinates imagePoints2 = rectify_quad([X(:), Y(:)]); % Apply rectification to all pixels

% Reshape rectified image back to original dimensions rectified_image = reshape(imagePoints2, size(calImg, 1), size(calImg, 2), []);

% Visualize results figure; subplot(121); imagesc(calImg); colormap(gray); hold on; scatter(I(:,1), I(:,2), 'r', 'LineWidth', 1); axis equal; xlabel('pixel'); ylabel('pixel'); title('Image and detected points (Image coord)');

subplot(122); imagesc(rectified_image); colormap(gray); scatter(W(:,1), W(:,2), 'r', 'LineWidth', 1); axis equal; xlabel('meters'); ylabel('meters'); title('Rectified image in real world coordinates');

% Additional adjustments for inverted image if necessary set(gca, 'YDir', 'normal'); % Ensure correct orientation if the image appears inverted

% Display or save the figure as needed

Explanation:

Calibration and Transformation: The function `calibrate_camera` calculates coefficients (`B`) based on the calibration points (`I` and `W`) and the chosen transformation order (2 for quadratic).

Rectification:The rectification function (`rectify`) is applied to each pixel of `calImg` using meshgrid to map image coordinates to real-world coordinates.

Visualization:Subplot (121) shows the original image with detected calibration points, while subplot (122) displays the rectified image in real-world coordinates, ensuring correct orientation if the image is upside down.

By visualizing both the original and rectified images, you can verify the effectiveness of your rectification process. Also, adjustments such as handling inverted images are also addressed to ensure clarity and correctness in your results. Please let me know if you have further questions.