Solving a large (263169x263169) linear system in cluster using 'directed' function

I want to solve a linear system in MATLAB whose size is (263169x263169). The major obstacle I am facing is creating the matrix of this size itself. I am working in a cluster with around 10TB soft quota. When I try to create this matrix, I get the following error:

n = 128;
C = zeros((2*n+1)^2); % I get the following error in this line
% C matrix (full matrix) is completely filled after some calculations
x = C\b;

"Error using zeros

Requested 263169x263169 (516.0GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take a long time and cause MATLAB to become unresponsive. See <a href="matlab: helpview([docroot '/matlab/helptargets.map'], 'matlab_env_workspace_prefs')">array size limit</a> or preference panel for more information."

The problem is that no single node I am allocated can handle 516.0 GB memory alone. So I need to distribute the data somehow between different nodes available to me. The "distribute" function in MATLAB takes already created matrix or a data containing csv file and distributes it to different nodes. However, I do not have the data in a file as I can only fill the elements in the matrix after running the MATLAB code. The only idea I have for now is to save the values of 'C' matrix directly into a csv file as I run the required code and later distribute the csv file using 'distribute' command for solving the linear system in MATLAB. But this process seems to take a lot more time than I expected.

Are there any ways in MATLAB to create a matrix which can be distributed between different nodes during the matrix creation itself? Further, can the entire problem be solved more efficiently in MATLAB with other functions?

Thank you.

Answer 1

[Raymond Norris]

Hi Yaswanth,

Take a look at codistributed arrays

https://www.mathworks.com/help/parallel-computing/working-with-codistributed-arrays.html

This will allow you to build up a distributed array from local variants. Depending on the shape of your CSV files and if you don't need to read the entire data set, you might also look at Tall Arrays and datastores.

https://www.mathworks.com/help/matlab/tall-arrays.html

https://www.mathworks.com/help/matlab/ref/datastore.html

Thanks,

Raymond

[Yaswanth Sai Jetti]

Hi Raymond,

Thank you very much for the references.

Thanks,

Yaswanth Sai