- Reduce the number of vertices present in the polygon while maintaining the shape of the polygon. Remove the duplicate vertices by using the "simplify" function in MATLAB: https://www.mathworks.com/help/matlab/ref/polyshape.simplify.html
- You can further reduce the the number of vertices of the polygon while maintaining overall shape, however the polygon would not have 100% accurate area. Refer to the following File Exchange submission which simplifies the polygon to a specified number of vertices: https://www.mathworks.com/matlabcentral/fileexchange/45342-polygon-simplification
- If the above two steps does not help in reducing the complexity of the polygon, then compute the exact region present inside of the smaller polygon. We are only interested in the region of the large polygon being present inside the smaller one. For instance, check the following figures which demonstrates this idea:

- Compute the vertices present inside the shape by using "inpolygon" or "isinterior" functions in MATLAB: https://www.mathworks.com/help/matlab/ref/inpolygon.html
- Compute the intersection points of the polyshape by using "polyxpoly" function. Check for the individual line segments of the large polygon with the smaller one. You can optimize this by using parallel processing and being selective of the line segments associated with the points already inside the smaller polygon.
- Once you have all the desired intersection points, you can create a small polyshape out of it and calculate its area.