2016b and later Disk Activity

Hi,

I have a parallel simulation set up using this guide: https://uk.mathworks.com/help/simulink/ug/running-parallel-simulations.html

It's a pretty intense workload, taking around 18 hours on a 40 core, 128GB RAM workstation and it worked fine on R2016a and earlier, but I have recently updated to 2017b and the same script is suddenly using the HDD a lot. This is resulting in the following problems;

1) It's filling my temporary directory with files which eventually causes the script to crash. The script works using a temporary directory for each thread so I expect some disk activity, but R2016b and later seem to be generating a huge amount more data in these directories.

2) I now have to manually delete the temporary files, previously I could using Windows built-in Disk Clean-Up utility to clear the temporary files if the script crashed during a parfor, but this no longer seems to work.

3) Most importantly, I'm running 100,000+ simulations which previously took around 18 hours. With 2016b and later, I'm lucky to get 5% of the simulations done in this time. I think I'm running into the SSD's maximum write speed with 40 threads and the sudden increase in drive activity.

So far, I've found out: - Speed issues are pretty similar to running out of RAM, but I have checked this and I am well under my 128GB. - I have tried all MATLAB versions between R2015a and R2017b. R2016a and earlier run fine, but R2016b and later all have the same issue. - The simulation speed issue behaves like high disk I/O (like running out of RAM). I've confirmed this by watching task manager and seeing high disk activity when running on R2016b and later. - I have identical problems running the script on other PC's, including a quad-core with 16GB RAM. - I have tried parsim on R2017b (https://uk.mathworks.com/help/simulink/slref/parsim.html), but still having similar problems.

I was wondering if anyone was aware of any changes which came into effect with R2016b that might be causing this, and how I could go about fixing the issue?

Regards, Tom

[Walter Roberson]

Odd...

Where is the disk activity taking place?

https://www.raymond.cc/blog/3-portable-tools-monitor-files-folders-changes/

[Tom]

It's in "C:/Users/{username}/Appdata/Local/Temp" and takes the form of *.dmr files.

I think they are related to the Simulation Data Inspector, which should be turned off according to this setting in the simulation parameters?

https://uk.mathworks.com/help/simulink/gui/record-logged-workspace-data-in-simulation-data-inspector.html.

I have found a workaround for some of the issues using;

 spmd
     Simulink.sdi.clear
 end

This clears the huge files (30GB+) after the parfor loop and allows me to then start another. This isn't as efficient as doing everything in one loop because some cores finish before others. Also it doesn't combat the disk write speed issue on the 40 core machine.

[Striker121]

Having a similar issue using parsim when doing a parameter sweep. The initial simulations run in quick succession but after about 50 simulations, they each take increasingly longer. Is there any way to use a function similar to "Simulink.sdi.clear" with parsim? Unlike parfor, I cannot find a way to run this command during a parameter sweep.

Answer 1

[Rahul Kumar]

Adding link to a related question

https://www.mathworks.com/matlabcentral/answers/385898-parsim-function-consumes-lot-of-memory-how-to-clear-temporary-matlab-files