Number of workers reducing automatically in parallel computing
2 views (last 30 days)
Show older comments
I am trying to run multiple instances of a function using 10 workers in a parallel pool. One instance of a function takes around 20 hours to run that is why I am using parallel computing so that I could get 10 results in 20 hours. But, after a few hours of starting the parallel computing, the number of workers automatically get reduced to 2. I am not sure what is happening. Here is the code for parallel computing that I am using:
clear all;
clc;
close('all')
numOfIterations = 60;
delete(gcp('nocreate'));
p = parpool(10);
% To request multiple evaluations, use a loop.
for idx = 1:numOfIterations
f(idx) = parfeval(p,@parallel_computing_distributed_fun,1,(idx+1043)); % Square size determined by idx
end
% Collect the results as they become available.
counter = numOfIterations;
for idx1 = 1:numOfIterations
%fetchNext blocks until next results are available.
flag = fetchNext(f);
disp('Completed the fetch of')
disp(idx1 + 1043)
disp(clock)
end
The pool does start with 10 workers. Here is output on the command window:
Parallel pool using the 'local' profile is shutting down.
Starting parallel pool (parpool) using the 'local' profile ...
Connected to the parallel pool (number of workers: 10).
But after a while the number of workers gets reduced to 2. Can someone please help me in finding out why the number of workers are getting reduced and how to fix it?
2 Comments
Edric Ellis
on 9 Nov 2022
This almost certainly means that workers are crashing or otherwise terminating abruptly. Do you see any warnings in the command window relating to this? You might also look in the JobStorageLocation of the local cluster to see if there are any crash dump files.
c = parcluster('local')
c.JobStorageLocation % look inside here
Answers (0)
See Also
Categories
Find more on Parallel Computing Fundamentals in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!