any hints on how to reduce the size of this thing?
Hi Anton, here's a hint: You're chopping down P then taking the difference between itself and its circshift. This is very close to just a simple diff() command. Circshift is a useful function, but not quite where you've got it ;)
Think of it this way:
A diff() will get vectors from one point to the next. If any neighbouring vectors point in the exact same direction, they are fair game to remove. More specifically, whenever the direction of the line *changes*, you've found a vertex you want to keep. How can you detect that?
Test  Status  Code Input and Output 

1  Pass 
%% Edge case: no vertices
P = zeros(0,2);
P2 = zeros(0,2);
assert(isequal(simplify_polygon(P), P2));

2  Pass 
%% Edge case: one vertex
P = [1 1];
P2 = [1 1];
assert(isequal(simplify_polygon(P), P2));

3  Pass 
%% Edge case: three vertices (a single line segment)
P = [...
1 1
1 2
1 1 ];
P2 = [...
1 1
1 2
1 1];
assert(isequal(simplify_polygon(P), P2));

4  Pass 
%% Single line segment with multiple vertices
P = [ ...
1 1
2 1
3 1
4 1
5 1
4 1
3 1
2 1
1 1];
P2 = [ ...
1 1
5 1
1 1];
assert(isequal(simplify_polygon(P), P2));

5  Pass 
%% Single line segment, different spacing
P = [ ...
1 1
2 1
4 1
5 1
1 1];
P2 = [ ...
1 1
5 1
1 1];
assert(isequal(simplify_polygon(P), P2));

6  Pass 
%% Rectangle
P = [ ...
1 1
2 1
3 1
4 1
4 2
4 3
3 3
2 3
1 3
1 2
1 1];
P2 = [ ...
1 1
4 1
4 3
1 3
1 1];
assert(isequal(simplify_polygon(P), P2));

7  Pass 
%% Two rectangles separated by line segment
P = [ ...
1 2
1 1
2 1
2 2
1 2
1 3
1 4
1 5
2 5
2 4
1 4
1 3
1 2];
P2 = [ ...
1 1
2 1
2 2
1 2
1 5
2 5
2 4
1 4
1 1];
assert(isequal(simplify_polygon(P), P2));

8  Pass 
%% Nonsimple polygon (figure eight)
P = [ ...
1 1
2 2
3 3
1 3
2 2
3 1
1 1];
P2 = [ ...
1 1
3 3
1 3
3 1
1 1];
assert(isequal(simplify_polygon(P), P2));

9  Pass 
%%
P = [ ...
1 1
2 2
3 3
4 4
5 5
5 4
6 3
8 1
7 1
1 1];
P2 = [ ...
1 1
5 5
5 4
8 1
1 1];
assert(isequal(simplify_polygon(P), P2));

10  Pass 
%% Circle; no points should be removed
theta = linspace(0,2*pi,200);
theta(end) = 0;
x = 20*cos(theta);
y = 20*sin(theta);
P = [x', y'];
P2 = P;
assert(isequal(simplify_polygon(P), P2));

11  Pass 
%% Starting vertex can be removed
P = [ ...
2 1
3 1
3 2
3 3
2 3
1 3
1 2
1 1
2 1];
P2 = [ ...
3 1
3 3
1 3
1 1
3 1];
assert(isequal(simplify_polygon(P), P2));

6587 Solvers
2474 Solvers
571 Solvers
Rotate input square matrix 90 degrees CCW without rot90
462 Solvers
229 Solvers