MATLAB (Linux) crashes during parallel read in of files

6 views (last 30 days)
Hello,
I've had problems with MATLAB crashing (it simply crashes and exits) when I try to read in a large number of files within a parfor loop. I have noticed that it seems to happen when my RAM (monitored using htop) gets full and then MATLAB suddenly crashes.
I have upgraded my RAM to 16GB and it seemed to help in some cases, however the problem still occurs.
Is the only solution to upgrade my RAM, or can something else be done? Actually one other thing I tried was to use less CPU cores which seems to help.
I used to use MATLAB on Windows 11 and it didn't crash due to this.
Thanks!
  2 Comments
Edric Ellis
Edric Ellis on 10 Oct 2022
One thing you could try is to use parpool("threads") - not everything works there, but if it does, it often uses less memory.
Without some more details, it's hard to know exactly what to suggest - it's not clear if the problem is during the data processing inside the parfor loop, or during the transfer to and from the workers. Both are possible. You can use ticBytes and tocBytes to see how much data transfer there is.
Otherwise, you might need to see if you can reduce the amount of memory needed to process each file. Maybe you can read each file in pieces rather than all at once. Maybe something like datastore could help.
Andrew Sandeman
Andrew Sandeman on 8 Nov 2022
Hi, sorry I didn't see this sooner I didn't recieve some kind of notification / email.
When I read in the same number of files using less cores, or even just a single core (without using the parfor loop) it seems to be less likely to crash. In particular I haven't had any crashes when I just use single core. Does this suggest that the data processing inside the parfor loop is ok? (just that it may use significant memory)
Just to rephrase / rethink: it's possible that either the reading in of a single file takes significant memory, or its the communication of data between cores?
I am in fact already using a datastore when using the parfor read in. That is the difference in the code between the parfor loop and normal loop versions.
Thank you for your suggestions, I will try to use them at some point.

Sign in to comment.

Answers (0)

Categories

Find more on Parallel for-Loops (parfor) in Help Center and File Exchange

Tags

Products


Release

R2022a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!