Matlab: find the contour and straighten a nearly rectangular image

Hello,

I have got this image https://www.dropbox.com/s/5hmy6tb3ym42tic/rect.jpg?m which is rectangle-like but not exactly.

I would like to find rectangular contours and I have tried to use edge and hough but it doesn't find continuous line. I have also tried to erode and dilate but it can't erase the difference of one pixel on the horizontal upper most line for example so it still doesn't find the continuous line. The second part of the problem would be to straighten it.

Another idea I got was to find the corners to link them with straight lines but it doesn't work either.

I am sure it shouldn't be complicated but I can't make it. Any ideas will be welcome!

Cheers

Answer 1

[Teja Muppirala]

I've done this before: http://www.mathworks.com/videos/solving-a-sudoku-puzzle-using-a-webcam-68773.html

This is the strategy that I used and it seemed to work on your image as well. Basically you search for the corners, and then use IMTRANSFORM to transorm the corners into a rectangle. (I'll assume your image is called I).

%%1. Get rid of the white border
I2 = imclearborder(im2bw(I));

%%2. Find each of the four corners
[y,x] = find(I2);

[~,loc] = min(y+x);
C = [x(loc),y(loc)];

[~,loc] = min(y-x);
C(2,:) = [x(loc),y(loc)];

[~,loc] = max(y+x);
C(3,:) = [x(loc),y(loc)];

[~,loc] = max(y-x);
C(4,:) = [x(loc),y(loc)];

%%3. Plot the corners
imshow(I); hold all
plot(C([1:4 1],1),C([1:4 1],2),'r','linewidth',3);

%%4. Find the locations of the new  corners
L = mean(C([1 4],1));
R = mean(C([2 3],1));
U = mean(C([1 2],2));
D = mean(C([3 4],2));
C2 = [L U; R U; R D; L D];

%%5. Do the image transform
figure
T = cp2tform(C ,C2,'projective');
IT = imtransform(im2bw(I),T); %IM2BW is not necessary
imshow(IT); hold all

Answer 2

[Image Analyst]

Carine:

You can straighten your image with imtransform(), like in this demo I wrote for you using your uploaded image:

clc;    % Clear the command window.
close all;  % Close all figures (except those of imtool.)
imtool close all;  % Close all imtool figures if you have the Image Processing Toolbox.
clear;  % Erase all existing variables. Or clearvars if you want.
workspace;  % Make sure the workspace panel is showing.
format long g;
format compact;
fontSize = 20;

% Change the current folder to the folder of this m-file.
if(~isdeployed)
  cd(fileparts(which(mfilename)));
end
  
% Check that user has the Image Processing Toolbox installed.
hasIPT = license('test', 'image_toolbox');
if ~hasIPT
  % User does not have the toolbox installed.
  message = sprintf('Sorry, but you do not seem to have the Image Processing Toolbox.\nDo you want to try to continue anyway?');
  reply = questdlg(message, 'Toolbox missing', 'Yes', 'No', 'Yes');
  if strcmpi(reply, 'No')
    % User said No, so exit.
    return;
  end
end

% Read in a standard MATLAB gray scale demo image.
folder = 'C:\Users\Carine\Documents';
baseFileName = 'rect.jpg';
% Get the full filename, with path prepended.
fullFileName = fullfile(folder, baseFileName);
% Check if file exists.
if ~exist(fullFileName, 'file')
  % File doesn't exist -- didn't find it there.  Check the search path for it.
  fullFileName = baseFileName; % No path this time.
  if ~exist(fullFileName, 'file')
    % Still didn't find it.  Alert user.
    errorMessage = sprintf('Error: %s does not exist in the search path folders.', fullFileName);
    uiwait(warndlg(errorMessage));
    return;
  end
end
grayImage = imread(fullFileName);
% Get the dimensions of the image.  
% numberOfColorBands should be = 1.
[rows columns numberOfColorBands] = size(grayImage);
if numberOfColorBands > 1
  grayImage = grayImage(:,:,2); % Take green channel
end
% Binarize the image
binaryImage1 = grayImage > 128;
% Get rid of the white border
binaryImage1 = imclearborder(binaryImage1);

% Display the original gray scale image.
subplot(2, 2, 1);
imshow(binaryImage1, []);
title('Binary Image', 'FontSize', fontSize);
% Enlarge figure to full screen.
set(gcf, 'units','normalized','outerposition',[0 0 1 1]);
% Give a name to the title bar.
set(gcf,'name','Demo by ImageAnalyst','numbertitle','off') 

% Fill the object
binaryImage2 = imfill(binaryImage1, 'holes');
% Find the boundaries
boundaries = bwboundaries(binaryImage2);
xCoords = boundaries{1}(:, 2);
yCoords = boundaries{1}(:, 1);

% Find out which index is closest to the upper left corner
distances = sqrt((xCoords - 1).^2 + (yCoords - 1).^2);
[minDistanceUL, minIndexUL] = min(distances)
% Find out which index is closest to the upper right corner
distances = sqrt((xCoords-columns).^2 + (yCoords - 1).^2);
[minDistanceUR, minIndexUR] = min(distances)

% Find out which index is closest to the lower left corner
distances = sqrt((xCoords - 1).^2 + (yCoords - rows).^2);
[minDistanceLL, minIndexLL] = min(distances)
% Find out which index is closest to the lower right corner
distances = sqrt((xCoords - columns).^2 + (yCoords - rows).^2);
[minDistanceLR, minIndexLR] = min(distances)

% Plot circles over the corners, just for visualization purposes - for fun.
xCorners = [xCoords(minIndexUL), xCoords(minIndexUR), xCoords(minIndexLR), xCoords(minIndexLL)]
yCorners = [yCoords(minIndexUL), yCoords(minIndexUR), yCoords(minIndexLR), yCoords(minIndexLL)]
hold on;
plot(xCorners, yCorners, 'ro');
title('Bad image with corners located', 'FontSize', fontSize);

% Determine ideal corner locations - aligned with raster lines
x1 = mean([xCorners(1), xCorners(4)])
x2 = mean([xCorners(2), xCorners(3)])
y1 = mean([yCorners(1), yCorners(2)])
y2 = mean([yCorners(3), yCorners(4)])

% Show this
subplot(2, 2, 2);
imshow(binaryImage1);
hold on;
plot([x1 x2 x2 x1 x1], [y1 y1 y2 y2 y1], 'r-', 'LineWidth', 3);
title('Bad image with perfect rectangular overlaid', 'FontSize', fontSize);

% Warp the image to straighten it.
badXY = [xCorners; yCorners]'
desiredXY = [x1 x2 x2 x1; y1 y1 y2 y2]'
% Transform to a quadrilateral with vertices badXY
% into a quadrilateral with vertices desiredXY.
tform = maketform('projective', badXY, desiredXY);

% Fix/warp the image.
[binaryImage3, xdata, ydata] = imtransform(binaryImage1, ...
  tform, 'bicubic', 'size', size(binaryImage1));
% Display the fixed image.
subplot(2, 2, 3);
imshow(binaryImage3);
title('Fixed image', 'FontSize', fontSize);

Answer 3

[Carine]

Thank you to the two of you, your answers have been very useful and the tutorial on sudoku solver impressive! I have mainly used the answer of Teja Muppirala but both give nearly the same result. To get rid off the white border I have inverted the colors first:

   I2 = imclearborder(1-I); I2 = 1-I2;

Now I've still got a problem as the area size is important in my application. Here is a picture of nearly undeformed image (taken with an angle 0º more or less): <https://www.dropbox.com/s/2vpk39qi4hy9zsw/im40_0.JPG> and here is another one taken from an angle of 30º more or less: <https://www.dropbox.com/s/t0qv7yb6l9u68im/im40_30.JPG>. I would not only like to "straighten" it but also to recover more or less the same size (area). So what I do from Teja Muppirala's code is modifying the coordinates of the new corners in the following way:

      C21 = C2;

    % 5. Scale the new corners so that the size of the rectangle is the size of the largest borders of the original picture
    % Ratio between horizontal borders
    dh1 =  distance(C(1,:),C(2,:));
    dh2 =  distance(C(4,:),C(3,:));
    if dh1<dh2
      dmh = dh1; dMh = dh2;
      imax = 2;
    else
      dmh = dh2; dMh = dh1;
      imax = 1;
    end

    C21(:,1) = C21(:,1)*dMh/dmh;

    % Ratio between vertical borders
    dh1 =  distance(C(1,:),C(4,:));
    dh2 =  distance(C(2,:),C(3,:));
    if dh1<dh2
      dmv = dh1; dMv = dh2;
      if imax == 1
        imax = 2; %imaxv = 23, imaxh = 12
      else
        imax = 3; %imaxv = 23, imaxh = 34
      end
    else
      dmv = dh2; dMv = dh1;
      if imax == 1
        imax = 1; %imaxv = 14, imaxh = 12
      else
        imax = 4; %imaxv = 14, imaxh = 34
      end
    end

    C21(:,2) = C21(:,2)*dMv/dmv;
    %imax is the index of the largest and longest border. Translate the new rectangle so that it passes by C2 of that index
    for index = 1
      C22(:,index) = C21(:,index)-C21(imax,index)+C2(imax,index);
    end

where distance is just d = sqrt(sum(C1-C2).^2);.

What do you think of the idea? Any better one?

Thank you again!