PARFOR behavior sensitive to comments???

2 views (last 30 days)
I am finding that the speed performance of PARFOR is greatly impacted depending on whether certain inconsequential lines in the code are commented out or not. In particular, when running the code below with a pool of 12 workers, I obtain a time of 26 sec (4 times that of a normal for loop).
However, if I comment out either the first line (thus converting the mfile to a script) or if I comment out the inconsequential line
A=zeros(M,N); B=A;
the time drops sharply to less than 1 sec!!
This is under Windows 7 64-bit. Processor is Intel Xeon X5680 @3.33 Ghz, dual hexacore. I've tested with R2011b,R2012b,R2013b, all with similar results.
Can anyone reproduce this? Is there something obvious that I'm not seeing?
function test1 %Comment this out
P=100;
fun=@(X)imrotate(X,30,'crop');
Xtypical=zeros(P);
M=P^2;
N=P^2;
A=zeros(M,N); B=A; %or Comment this out
A=cell(1,N);
tic;
parfor j=1:N
input=Xtypical;
input(j)=1;
T=fun(input);
A{j}=sparse(T(:));
end
A=cell2mat(A);
toc;
  2 Comments
amir
amir on 9 Feb 2014
Edited: amir on 9 Feb 2014
with my system, if I comment first line, the result is generated after about 3.5 seconds but if I run it with first line (as a function) my system goes to a bad state, CPU usage is about zero but whole of my system has no response and I must reboot my system.
by smaller values for P (for example 70), my results was like yours, running as script was better than running as function (opposite to my results in here )
Matt J
Matt J on 9 Feb 2014
Thanks! Guess it's time to make a Bug Report.

Sign in to comment.

Accepted Answer

Edric Ellis
Edric Ellis on 10 Feb 2014
Unfortunately what I think you're seeing there is a limitation of the way function handles are created. If you use the FUNCTIONS function on your function handle 'fun' just before entering the PARFOR loop, you can see that in the 'workspace' entry, it contains everything in the function workspace. In particular, the large value of 'B' is there - hence why commenting out the creation of that variable 'fixes' things.
Function handles are created differently at the command-line or in scripts, and don't end up grabbing everything in this way - hence why using a script also 'fixes' things.
Perhaps the best workaround is either to use internal functions, or generate the function handles in a scope that cannot see any large variables.
  1 Comment
Matt J
Matt J on 10 Feb 2014
Edited: Matt J on 10 Feb 2014
Pretty subtle pitfall, Edric! But it looks like you're right.
Could you also take a look at this one,
and see if you have more insights into what is happening than we did?
There, we also saw parfor performance change greatly depending on whether running as a script or mfile function. Initially, I thought it might have been caused by reasons related to my post here, but now I guess not.

Sign in to comment.

More Answers (1)

Eric Sampson
Eric Sampson on 10 Feb 2014
Matt, what happens if you create the anonymous function inside the parfor loop? If that helps, but if in actual code you need to define it before the parfor loop, then could you first create a string like fun_str = '@(X)imrotate(X,30,''crop'')' and then inside the parfor loop do fun = str2func(fun_str); ?
  9 Comments
amir
amir on 17 Feb 2014
sorry but I can not find where Edric do that (creating data directly on the workers and making it persistent across multiple calling parfor).

Sign in to comment.

Categories

Find more on Parallel for-Loops (parfor) in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!