Parfor is running much slower on a 18-core PC than a 10-core PC

1 view (last 30 days)
I am currently running a compute-intensive financial trading test; the test has 503 independent loops. I have two PCs in my office, one has 10-core Intel i9-10900k @ 3.7GHz with 128GB DDR4; the other one has 18-core Intel i9-7980XE @ 3.4GHz with 128GB DDR4 as well.
In order to speed up the whole processing, the first 10-core PC runs loop 1-to-189 in parfor; the second 18-core PC runs loop 190-to-503 in parfor.
After one-day running, I found the 10-core PC finished 25 loops, the 18-core PC only finished 4 loops. I found very curious, can anyone know the reason of this problem?
P.S. 10-core PC running MATLAB 2020b with latest updates with MOSEK 9.2.35
18-core PC running MATLAB 2021a with latest updates with MOSEK 9.2.36
I also checked the taskmgr.exe, unlike the 10-core PC, I found in 18-core PC, 10 matlab tasks are in one group, the other 10 tasks are seperated....
  6 Comments
Walter Roberson
Walter Roberson on 28 Aug 2021
parfor has communications overheads, and if you give each node too little to do, then you can end up spending more time communicating than you get back from splitting the work. So it is possible for progress to get slower as you add more workers. It would be interesting to restrict both machines to (say) four cores and see what their performance is.
The studies I looked at a number of years ago suggested that 4 to 6 cores was the "sweet spot" for a variety of tasks -- at least with the hardware of those days. There are tasks for which the communications might be comparatively small and so net performance can improve as you add cores, even into the tens of thousands, but most code has to be specially written for that kind of situation.
Mingyang Sun
Mingyang Sun on 2 Sep 2021
When using the MOSEK (interior-point or integer programming) inside parfor loop, it is better to turn off the multi-threads in MOSEK, otherwise, by default, MOSEK will using all cores.
param.MSK_IPAR_NUM_THREADS = 1;
param.MSK_IPAR_INTPNT_MULTI_THREAD = 'MSK_OFF';
After I turn off the multi-threads in MOSEK, parfor-loop is now approx. 17 times faster than for-loop!

Sign in to comment.

Answers (0)

Categories

Find more on Parallel Computing Fundamentals in Help Center and File Exchange

Products


Release

R2021a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!