Parallel workers automatically shutting down in the middle of RL parallel training.

18 views (last 30 days)
Hello,
I am currently training a reinforcement learning PPO agent on a Simulink model with UseParallel=true. The total episodes that it should be running for are 5000 (about 10/11 hours of training), but i'm noticing that as the training goes on, more and more workers of the parallel pool are automatically shutting down, making training slower and slower as it progresses. I start with 8 workers, and they consistently decrease one at a time, until errors are generated.
I've been noticing this consistently in each training that i do, and would like to know if there are any workarounds.
For the parpool, i am letting Matlab start it automatically with all options set to default. I have also tried playing around with the number of workers, but the same thing happens.
  2 Comments
Emmanouil Tzorakoleftherakis
What errors are you seeing? Maybe try training on a single worker initially to make sure you don't see any errors before moving to parallel.
Matteo D'Ambrosio
Matteo D'Ambrosio on 10 May 2023
Edited: Matteo D'Ambrosio on 10 May 2023
On a single worker everything works fine, and the errors i get happen after 3000+ training episodes, after the workers have slowly started shutting down one at a time. I've also used the environment validation function.
I've noticed that after training for this number of episodes, i also get the following error (tied to the PPO algorithm):
Dot indexing is not supported for variables of this type.
Error in rl.agent.rlPPOAgent/learnFromAdvantages (line 40)
advantageData.Advantages,this.AdvantageBuffer_,...
Error in rl.train.parallel.AsyncPPOParallelTrainer/processSimOutput_ (line 62)
learnData = learnFromAdvantages(agent,data.AdvantageData);
Error in rl.train.parallel.AsyncParallelTrainer/processFutures_ (line 22)
processSimOutput_(this,out);
Error in rl.train.parallel.AbstractParallelTrainer/run (line 64)
[F,outs,taskIDs] = processFutures_(this,F);
Error in rl.train.TrainingManager/train (line 479)
run(trainer);
Error in rl.train.TrainingManager/run (line 233)
train(this);
Error in rl.agent.AbstractAgent/train (line 136)
trainingResult = run(trainMgr,checkpoint);

Sign in to comment.

Accepted Answer

Edric Ellis
Edric Ellis on 11 May 2023
If workers are leaving the pool one at a time while the pool is busy, this almost certainly means that they are crashing. I recommend contacting MathWorks support for help diagnosing and resolving this problem. You could also check by running the following location to see if any "matlab_crash_dump.*" files have been left behind:
jsl = parcluster().JobStorageLocation
dir(fullfile(jsl, '**', 'matlab_crash_dump.*'))

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!