Out of memory issue although there is more RAM space than matlab requires

Hi everyone,

I'm trying to apply Singular Value Decomposition (SVD) on a large matrix (80447x80287) which I get from xyz coordinates by applying some grid function. The SVD simply looks like this [U E V]= svd(Z). I've tried couple of tricks:

1) Java Heap Memory increased to the maximum.

2) MATLAB array size limit unchecked.

3) Paging file size in PC virtual memory increased.

None of them seemed to work out the issue. I'm not a Matlab expert but looking at the memory used and the available memory, memory should not be an issue. I also checked the RAM usage which seems to be around 60%. Can anyone think of a way to tackle this? (PC properties: windows 64bit 96 GB RAM)

whos

Name          Size                     Bytes  Class     Attributes

A         81023x3                    1944552  double              
Z         80447x80287            51670786312  double              
x         81023x1                     648184  double              
y         81023x1                     648184  double              
z         81023x1                     648184  double              

 memory
Maximum possible array:      106862 MB (1.121e+11 bytes) *
Memory available for all arrays:      106862 MB (1.121e+11 bytes) *
Memory used by MATLAB:       52273 MB (5.481e+10 bytes)
Physical Memory (RAM):       98226 MB (1.030e+11 bytes)

• Limited by System Memory (physical + swap file) available.

Answer 1

[Christine Tobler]

Hi Oguzhan,

As I remember from your previous post, this large matrix is the result of interpolating scattered data on a grid. One way of course would be to choose a coarser grid size for your matrix, but I assume there is a reason this is not possible?

I think you wanted to compute the SVD as a way to compress your data, so that only the largest singular values and vectors are of interest. In that case, you could consider using the svds function instead, and computing the largest 10, 100, or 500 singular values of this matrix.

Answer 2

[Jan]

Your matrix Z needs a contiguous block of 51 GB RAM. The SVD of a 80447x80287 matrix will create 3 matrices 80447x80447, a 80447x80287 and a 80287x80287. They require about 3 * 51 GB RAM in addition. This is a total of over 200 GB, such that your 96 GB machine will be exhausted, even if you increase the virtual memory.

Increasing the Java Heap memory is counter-productive, because SVD is not calculated in Java, but the memory reserved for Java is not available for the other calculations.

[EDITED] Think of using the "economy size" output of SVD. Either you have to reduce the size of the problem, or install more RAM. Maybe your problem fits into 256 GB RAM, but a secure amount of RAM is the double size of the used arrays.

[Christine Tobler]

I agree, and would add that svd also needs some internal memory. I don't remember how much just now, but I'd say at least another 51 GB.