use parfor if larger than certian size

2 views (last 30 days)
Is there an easy method to specify in the parfor loop to use parfor only if a value is larger than a certian size? I.E. I have a user selecteable option, and if they only chose to select to perform the operation on a few files, the time it takes to initiate the parllel pool is longer than it would take to perform a normal for loop. But if the user selects a large number of files, the parfor loop is significantly faster even if one has to wait for the parallel pool to start.

Accepted Answer

Raymond Norris
Raymond Norris on 16 Sep 2020
Hi Shane,
There might be several options here, in no particular order
1. For starters, you might consider starting the pool (would it be local?) as part of the startup process of your application. By default, the parallel pool will shutdown after a timeout period. You might want to suggest they disable this, so that it doesn't shutdown in case they don't use it for a while. Of course this may have some performance degregation if you don't need the pool running at all. When your application is done, and if you've had them override auto-shutdown of the pool, you might want to close the pool as well.
2. In R2020a we introduced "threads" pool, which might be quicker to start. You can read more here
p = parpool("threads");
parfor fidx = 1:numel(files)
process(files{fidx})
end
3. parfor takes several arguments. Two to look at are:
  • Specifiying the maximum number of workers to use. For example
if nfiles>=threashold
psize = 4;
else
% Not enough files to process in parallel
psize = 0;
end
parfor (fidx = 1:numel(files),psize)
process(files{fidx})
end
If psize is 0, then parfor operates as a for (though maybe a bitter slower? worth checking).
  • Request a cluster pool (introduced in R2019a), rather than a parallel pool. By specifying a parfor options, jobs are submitted via batch submission. There's more to understand about cluster pools, so read more here
The advantage to this is that it doesn't take as long to start the pool. This is more applicable to jobs running on an cluster.
cluster = parcluster;
opts = parforOptions(cluster);
parfor (fidx = 1:numel(files),opts)
process(files{fidx})
end
Raymond

More Answers (0)

Categories

Find more on Startup and Shutdown in Help Center and File Exchange

Tags

Products


Release

R2020a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!