Parallel computing memory allocation

I am running code on an HPCC. I have a very large loop so I am trying to split it up. The problem is that I seem to have to give each parallel instance the entire memory reserve for the job leading to too much memory usage.

ClusterInfo.setMemUsage('10gb');  %command from our hpcc
parpool(100)
inside_code_file

Where inside_code_file generates a very large matrix [just under 10 gb] and runs 'parfor' through each cell independently.

When doing it this way, I have to give Matlab 10gb for each of the 100 parallel cores. I'm pretty sure I set up the parfor correctly as there are no `split' warnings.

Is this something I can fix through Matlab or is it on the HPCC side?

Or is this just a fundamental aspect of parallel computing and unavoidable?

Answer 1

[Tushar Upadhyay]

From the information you provided, It seems that you want to find a better memory efficient way to avoid allocating 10gb of memory to each worker.

Does each worker require a separate copy of the matrix to do their tasks or they can share the same matrix?

[CJ]

Each worker's input can be a shared matrix, but each worker outputs 1 cell into a new matrix that becomes very large. Does that answer your question?

[Nagarjuna Manchineni]

If the input matrix can be shared, then MATLAB will distribute the array to the respective workers during the runtime and once the run for that particular input is completed, then MATLAB worker returns the data back to client (MATLAB).

So, once verify which kind of variable (sliced, broadcast, temporary, loop, etc..) you are trying to use in 'parfor' and let us know.

Also, see the following documentation link for more troubleshooting steps on variables that are shared across workers from MATLAB:

https://www.mathworks.com/help/distcomp/troubleshoot-variables-in-parfor-loops.html