parallel.pool.Constant: An error occurred building a parallel.pool.Constant.

Hi

I am trying to run a script which is of the type below:

            ni=670; 
            N=100;
            z = randn(512,512,100,ni*N);
            C = parallel.pool.Constant(z);
            parfor nn=1:N
                a{nn} = function1( C.Value(:,:,:,(nn-1)*ni + 1:nn*ni) + d{nn}, lambda1 );
                d{nn} = function2( d{nn} + C.Value(:,:,:,(nn-1)*ni + 1:nn*ni) );
                temp{nn} = function3( a{nn} - d{nn}, lambda2 );
            end

However I am getting the following error:

        *Error using parallel.pool.Constant (line 177)
        An error occurred building a parallel.pool.Constant.

        Caused by:
        No workers are available for FevalQueue execution.

        A write error occurred while sending to worker 6.*

Could anybody help with why this is happening and what can I do to resolve it?

System Config:

     CPU: Intel(R) Xeon(R)CPU E5-2687Wv4 @3.00GHz
     RAM: 512GB 
     OS: Windows 10 Pro 64-bit
     Checked on MATLAB R2016a and R2016b

thanks, Bis

Answer 1

[Edric Ellis]

That error means that a worker crashed for some reason, I strongly suspect out-of-memory. Note that your z matrix is huge - 1400GB; also note that parallel.pool.Constant still requires a copy of z on each worker (the benefit of parallel.pool.Constant is not that it reduces overall memory usage, rather that it stops you having to transfer data multiple times).

I think it should be possible to slice z, rather than use a parallel.pool.Constant. If you reshape z, you should be able to pick whole slices using an expression something like

    z(:,:,:,:,nn)

That way, each worker only receives the portion of z it requires. This will drastically reduce the memory usage required.

[Biswarup Choudhury]

Thanks much Edric!

I tried with a smaller version of z and it worked! That confirmed that it was an out-of-memory issue. There is no way the algorithm could have run if entire z was copied to each worker, like you pointed out.

Also you are right in the observation that, in our case, we should rather slice z, and send each slice to a worker. Already did that!

thanks again,

Bis