data = double(YourImage);
dv = data(:);
try_it = @(ang) sum((reshape(fliplr(imrotate(data,ang,'crop')),,1) - dv).^2)
A = linspace(0,359.9,500);
fitr = arrayfun(try_it, A);
[~,idx] = min(fitr);
best_ang = A(idx);
Note: what you posted was an object in a white frame. The YourImage I indicate above should have that white frame cropped away (e.g., should be the original image.)
The code here will work for grayscale and color both.
What this does is rotates an image by an angle, with cropping, flips it left to right, and finds the euclidean distance between that and the original. The hypothesis being tested is that there is an axis of reflective symmetry running though the center of the image and that it is just necessary to find the correct angle for it.
This will probably not do exactly what you want, in that your hypothesis probably involves an axis of reflective symmetry to does not run through the center of the image. You should be able to extend the technique to two parameters and evaluating at a grid of value pairs.
fmincon() cannot really optimize this, as it is not a continuous problem: small differences in rotation angles lead to the same output.