In order to work with nonlinear equations using fmincon you can have a look at the following documentation which explains how to work with optimization of such kind of equations:
- https://www.mathworks.com/help/optim/ug/fmincon.html
- https://www.mathworks.com/help/optim/nonlinear-programming.html?s_tid=CRUX_lftnav
