Computer crashed when calculating matrix (mixed with sparse and full) multiplication and summation
2 views (last 30 days)
Show older comments
In a Matlab script, I have to multiply a large sparse matrix with a full matrix, as follows:
A1 = B * C.
B is a full matrix, its size is 5000-by-4800 double. C is a sparse matrix, its size is 4800-by-3E5 sparse double, and its nnz/numel = 2.08E-4, so it’s very sparse.
My desktop computer crashed at this line of code. I thought it was because matrix C is too large, so I “broke down” matrix C along its columns, and let them multiply matrix B one by one, such as follows:
for j=1:3E5
Aj=B*C(:,j);
A1(:,j)=Aj;
end
The computer did not crash this time, it was able to calculate the multiplication and provide matrix A1. However, it was slow (elapsed time was about 10 seconds), and the “Physical Memory Usage” in the Task Manager was alarmingly approaching the maximum.
Immediately following this “for” loop, I have one more line of code, which is a simple summation of two matrices:
A = A1 + A2.
Matrix A1 (a full matrix) is obtained from above. Matrix A2 is a sparse matrix of the same size (5000-by-3E5), it is very sparse.
As I mentioned, the computer did not crash calculating the “for” loop. But it crashed at this summation line of code afterwards. Could it be because too much “Physical Memory Usage” was used up already prior to the summation?
In conclusion, the whole thing is a mess, and I appreciate any help to be able to calculate these large matrix multiplication and then summation. Or could it be that the matrix C is simply too large and there’s no way to solve it?
Thank you!
-------------------------
Update: I just tried something else: Without running any code, I entered in the Command window: A1 = zeros(5000, 3E5). This simply creates a full matrix of zeros, and I see that the “Physical Memory Usage” in the Task Manager immediately goes up to about 14GB, approaching the maximum (16GB).
Does this mean a full matrix that has 3E5 number of columns is just too big for the computer to handle, and there's nothing I can do to solve it?
6 Comments
James Tursa
on 15 Jun 2021
Edited: James Tursa
on 15 Jun 2021
You have pretty much come to the conclusion yourself ... you will need to chunk out your calculations to avoid the memory issue and simply live with the calculation runtimes, or figure out another way to solve your problem. We don't have any insight into what that problem actually is, but maybe you could solve a smaller problem that approximates your actual problem within acceptable error, etc.
Accepted Answer
James Tursa
on 16 Jun 2021
Remember that each element of the B*C result is simply the dot product of a row of B with a column of C. If that column of C has any non-zeros in it (even if it is only one non-zero), then in general you would expect the dot product to be non-zero, the only exception being if you had very fortunate cancellation. The fact that sum(any( C )) = 3E5 for your particular case means that all of the columns of C have at least one non-zero, so the result of B*C would be expected to be essentially a full matrix. Hence your runtime and memory problems ...
You will need to chunk out your calculations to avoid the memory issue and simply live with the calculation runtimes, or figure out another way to solve your problem. We don't have any insight into what that problem actually is, but maybe you could solve a smaller problem that approximates your actual problem within acceptable error, etc.
0 Comments
More Answers (1)
Steven Lord
on 15 Jun 2021
What release of MATLAB are you using?
What operating system? I'm suspect you're going to say Linux. If I'm right, the OOM Killer may have struck again. The page linked as the first footnote says how to check if this was the case.
3 Comments
Steven Lord
on 15 Jun 2021
Can you clarify did your entire computer crash (Windows shut down, you received a Blue Screen of Death, etc.) or did MATLAB crash? In the latter case, there should be a crash log file in the temporary directory:
ls(fullfile(tempdir, '*crash*'))
If one exists that was created at the time MATLAB crashed please post it or send it to Technical Support.
See Also
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!