Maximizing the number of workers used in a parfor loop using the "batch" command
11 views (last 30 days)
I have four cores, and would like to use all four of them in parallel. To accomplish this, I have attempted to use the "batch" command, combined with a parfor loop as follows. (The example is completely silly, I've tried to strip it down to the most basic components for clarity.)
I have a program called myTest.m as follows:
where sillyProgram is defined as follows (it's silly):
ctr = 0;
fid = fopen(['Silly' num2str(ii)],'a');
while 1 > 0;
ctr = ctr + 1;
I then try to run myTest with the following command:
(I'm running R2013a, hence matlabpool instead of pool.)
The idea was: use three workers in the parfor loop, then exit out of the loop and use the fourth worker to run sillyProgram one more time. But for some baffling reason, the fourth job does not get run. And consequently I'm unable to get all 4 workers running simultaneously.
Am I making a stupid mistake? If not, and this is intended behavior. is there a more intelligent way to use batch, but not waste a a precious worker on the wrapper program, that really does nothing but spawn the real jobs?
Thomas Ibbotson on 14 Aug 2014
Edited: Thomas Ibbotson on 14 Aug 2014
I think it might help to explain how batch with the 'matlabpool' argument works. This causes 1 worker to run 'myTest' as the client, and the other 3 workers are used to form the pool with that client worker. The client worker then runs the code in 'myTest' just as it would if you had run the script on your local machine.
This means that on the client worker the code containing the parfor runs and calls 'sillyProgram' within the parfor on each of the 3 other workers. Because 'sillyProgram' has an infinite loop it will never complete, and your client worker will wait forever for the parfor to finish, never reaching the second call to 'sillyProgram' outside the loop.
To answer your second question, you might want to look at the jobs and tasks API.