Out of memory error

I am trying to run a optimzation problem but I am keep getting out of memory error. It's a 20k variable 300 constraint problem. I dont understand why this happens. I'll give the constraint functions and the main script respectively. This is a modified version of the p-median problem but thats not very important I guess for my question at hand.

function constraints = constraintFcnEq(x,y)
for i = 1:150
    constraints(i) = sum(x(i,:)) == 1;
end    
constraints(151) = sum(y) == 3;
end

function constraints = constraintFcnIneq(x,y)
for i = 1:150
    constraints(i) = sum(x(:,i)) - 150*y(i) <= 0;
end
end

A = readmatrix("iris.csv");
A = A(:,2:5);
A = normalize(A);
dist_mat = pdist(A);
dist_mat = squareform(dist_mat);
x0x = zeros(150,150) + 1/150;
x0y = zeros(150,1) + 3/150;

x = optimvar("x",150,150,"LowerBound",0,"UpperBound",1);
y = optimvar("y",150,1,"LowerBound",0,"UpperBound",1);

initialPoint.x = x0x;
initialPoint.y = x0y;

problem = optimproblem;

problem.Objective = sum(dist_mat.*x,...
    'all')+ 100*sum(10.*(y.^2)./(10.*(y.^2) + 0.01));

problem.Constraints.constraint1 = constraintFcnIneq(x,y);
problem.Constraints.constraint2 = constraintFcnEq(x,y);

options = optimoptions("fmincon","Display","iter","PlotFcn","optimplotfval");

show(problem);

[solution,objectiveValue,reasonSolverStopped] = solve(problem,initialPoint,...
    "Solver","fmincon","Options",options);

[Matt J]

You haven't provided us the means to run your code, so we can only guess. However, you might try re-expressing your constraints in a more vectorized way:

problem.Constraints.Xeq = sum(x,2)==1;
problem.Constraints.Yeq = sum(y)==3;
problem.Constraints.XYineq = sum(x,1)-150*y.'<=0;

[Yagiz Dereboy]

Thank you for the answer. I don't know what do you mean by means to run my code. I can send you the iris.csv. But in my mind I sent the constraint functions and the main script. Please tell me what else I can do to help you help me:)

Answer 1

[John D'Errico]

You have 20000 variables. So fmincon will be computing a Hessian matrix internally of size 20000*20000 = 4e8. But that matrix is double precision. So it will alone require 8*4e8=3.2e9. So roughly 3.2 GB of RAM will be required. And if a copy is ever made, or say, a factorization of the matrix is ever computed, then you need to immediately double the memory required. Fmincon will indeed need to factor that matrix, so you are starting to get in trouble. And that is only for that one array. Worse, MATLAB stores arrays in CONTIGUOUS blocks of memory. So you need to have enough RAM around so that MATLAB will be happy.

These days, it is still not uncommon for people to try to use MATLAB to solve problems like this on a computer with 4GB of RAM, Even 8GB of RAM would not be even close to enough here.

[Yagiz Dereboy]

Thank you so much for your answer. May I ask you an another question if it is not too much. May I do a lot better job with programming language like c++

Answer 2

[Yagiz Dereboy]

Just a heads up for everyone with memory problems, use L-BFGS instead of BFGS