Parfor loop just hangs, CPU usage goes to zero

Hi all. Here is a sample code of what I am attempting to run.

parfor i = 1:num
answer(:,i) = someFunction(someData(:,i));
end

Key information: "someFunction" is a C++ mex file. "someData" is a memmapfile (memmapfilename.data) because it is too large to be loaded onto each worker

Oddly, the parfor loop just hangs, the CPU usage goes to zero, and when I CTRL+C, here is what I get:

Operation terminated by user during distcomp.remoteparfor/getCompleteIntervals (line
127)
In parallel_function>distributed_execution (line 820)
        [tags, out] = P.getCompleteIntervals(chunkSize);
In parallel_function (line 587)
        R = distributed_execution(...

This isn't an issue if I replace the "parfor" with a simple "for" - everything works fine. What seems to happen is that some of the workers become unresponsive. After the above issue is encountered, even running a simple command such as

pctRunOnAll 1+1

will return "2" on only some, but not all, workers.

Any help would be great. A fresh re-installation did not help. Validation for "parpool" passed.

[Benoit Pouyatos]

Hi there, I have the same issue here using Matlab 2017a. One peculiar fact is that the exact same code never hangs with Matlab 2014a. Did somebody find a fix for this problem?

[arvid Martens]

Recently, I have been experiencing the same problem with parfor in Matlab 2017a. Has a solution to this problem been found?

[Cole]

I tried MATLAB 2014a, but the same error occurred.

[John Tencer]

I'm observing this behavior with 2017b.

[David Saidman]

Did anybody get any success with this? I'm having exact same 2017b, definitely no keyboard statement.

If I wait a bit, it ends up running but on a single CPU (event tho I have 18 in my pool on a cpu with 20 physical and 40 logical cores, about 10gb spare memory in performance monitor).

Answer 1

[Dave Behera]

It seems that there is a deadlock when the workers are trying to the access the file using the same object (that you got from memmapfile). Due to that, the progress is getting stalled with zero CPU usage and no abort message.

Can you try creating a separate memmapfile object within each parfor iteration and passing it to the someFunction function? This may make the file access thread-safe.

Also, could you try the same workflow with spmd?

http://www.mathworks.com/help/distcomp/spmd.html

[JohnDapper]

Thanks for the answer. Making a memmapfile object inside each parfor iteration (say, mm{i}), and then passing mm{i}.Data to someFunction, did not help. Continues to hang.

Same problem with spmd.

Once again, no problem at all when I change parfor to for. You`d think MATLAB would have sorted this out before releasing memory mapping...

[M L]

I have the same problem-- did you ever find a fix?

[Ryan]

Same problem for me too... any updates? I found my issue only occurs on my workstation that has 4 CPUs (6 cores per CPU for a total of 24 workers), but not on my computer that has only 1 CPU (6 cores, 6 workers total). Anyone having this issue using a dual or more CPU socket system?

[Zeb Kurth-Nelson]

I'm having exactly the same problem. Running the loop as a for loop works fine. But running it as a parfor loop hangs with zero CPU usage. Ctrl-C results in the same error message as in the original post.

I'm not loading any data from files.

[Sanjay Manohar]

Happened suddenly after I set a breakpoint within a subfunction.

[Cole]

I have just tried SPMD, but it doesn't work. The similar problem still exists.

[Hakon Haugnes]

I have the same problem on R2018a

[Arabarra]

Same problem here. In my case it is not reproductible, sometimes it will work, sometimes not. No deadlocks or anything suspicious in the code.

Answer 2

[arvid Martens]

I noticed that the problem started to occur after I updated the drivers of the GPUs that are being used during the calculations. Rolling back the drivers resolved the problem. However, new GPU hardware is on its way, as the current ones are pretty old. So I hope the problem is resolved by then.

Is there a way to throw an error when this stalling occurs? I could write an error handling to reduce the time lost by this stalling.

Answer 3

[Andrea Stevanato]

I have the same error with matlab 2018a.

Answer 4

[Sanjay Manohar]

I was having the same parfor problem, until I noticed I had a "keyboard" instruction in my code.