Once you have created an environment and reinforcement learning agent, you can train the
agent in the environment using the
train function. To
configure your training, use the
function. For example, create a training option set
opt, and train agent
agent in environment
opt = rlTrainingOptions(... 'MaxEpisodes',1000,... 'MaxStepsPerEpisode',1000,... 'StopTrainingCriteria',"AverageReward",... 'StopTrainingValue',480); trainStats = train(agent,env,opt);
For more information on creating agents, see Reinforcement Learning Agents. For more information on creating environments, see Create MATLAB Environments for Reinforcement Learning and Create Simulink Environments for Reinforcement Learning.
train updates the agent as training progresses. To preserve the
original agent parameters for later use, save the agent to a MAT-file.
Training terminates automatically when the conditions you specify in the
StopTrainingValue options of
rlTrainingOptions object are satisfied. To manually terminate
training in progress, type Ctrl+C or, in the Reinforcement Learning
Episode Manager, click Stop Training. Because
updates the agent at each episode, you can resume training by calling
train(agent,env,trainOpts) again, without losing the trained parameters
learned during the first call to
In general, training performs the following steps.
Initialize the agent.
For each episode:
Reset the environment.
Get the initial observation s0 from the environment.
Compute the initial action a0 = μ(s0), where μ(s) is the current policy.
Set the current action to the initial action (a←a0), and set the current observation to the initial observation (s←s0).
While the episode is not finished or terminated, perform the following steps.
Apply action a to the environment and obtain the next observation s''and the reward r.
Learn from the experience set (s,a,r,s').
Compute the next action a' = μ(s').
Update the current action with the next action (a←a') and update the current observation with the next observation (s←s').
Terminate the episode if the termination conditions defined in the environment are met.
If the training termination condition is met, terminate training. Otherwise, begin the next episode.
The specifics of how the software performs these steps depend on the configuration of the agent and environment. For instance, resetting the environment at the start of each episode can include randomizing initial state values, if you configure your environment to do so. For more information on agents and their training algorithms, see Reinforcement Learning Agents.
By default, calling the
train function opens the Reinforcement
Learning Episode Manager, which lets you visualize the training progress. The Episode
Manager plot shows the reward for each episode (EpisodeReward) and a
running average reward value (AverageReward). Also, for agents that
have critics, the plot shows the critic's estimate of the discounted long-term reward at the
start of each episode (EpisodeQ0). The Episode Manager also displays
various episode and training statistics. You can also use the
function to return episode and training information.
For agents with a critic, Episode Q0 is the estimate of the discounted long-term reward at the start of each episode, given the initial observation of the environment. As training progresses, if the critic is well designed. Episode Q0 approaches the true discounted long-term reward, as shown in the preceding figure.
To turn off the Reinforcement Learning Episode Manager, set the
During training, you can save candidate agents that meet conditions you specify in the
SaveAgentValue options of your
rlTrainingOptions object. For instance, you can save any agent whose
episode reward exceeds a certain value, even if the overall condition for terminating
training is not yet satisfied. For example, save agents when the episode reward is greater
opt = rlTrainingOptions('SaveAgentCriteria',"EpisodeReward",'SaveAgentValue',100');
train stores saved agents in a MAT-file in the folder you specify
SaveAgentDirectory option of
rlTrainingOptions. Saved agents can be useful, for instance, to test
candidate agents generated during a long-running training process. For details about saving
criteria and saving location, see
After training is complete, you can save the final trained agent from the MATLAB® workspace using the
save function. For example, save the
myAgent to the file
finalAgent.mat in the
current working directory.
save(opt.SaveAgentDirectory + "/finalAgent.mat",'agent')
By default, when DDPG and DQN agents are saved, the experience buffer data is not saved.
If you plan to further train your saved agent, you can start training with the previous
experience buffer as a starting point. In this case, set the
SaveExperienceBufferWithAgent option to
some agents, such as those with large experience buffers and image-based observations, the
memory required for saving the experience buffer is large. In these cases, you must ensure
that enough memory is available for the saved agents.
You can accelerate agent training by running parallel training simulations. If you have Parallel Computing Toolbox™ software, you can run parallel simulations on multicore computers. If you have MATLAB Parallel Server™ software, you can run parallel simulations on computer clusters or cloud resources.
When you train agents using parallel computing, the host client sends copies of the agent and environment to each parallel worker. Each worker simulates the agent within the environment and sends their simulation data back to the host. The host agent learns from the data sent by the workers and sends the updated policy parameters back to the workers.
To create a parallel pool of
N workers, use the following
pool = parpool(N);
If you do not create a parallel pool using
parpool (Parallel Computing Toolbox), the
train function automatically creates one
using your default parallel pool preferences. For more information on specifying these
preferences, see Specify Your Parallel Preferences (Parallel Computing Toolbox).
For off-policy agents, such as DDPG and DQN agents, do not use all of your cores for parallel training. For example, if your CPU has six cores, train with four workers. Doing so provides more resources for the host client to compute gradients based on the experiences sent back from the workers. Limiting the number of workers is not necessary for on-policy agents, such as PG and AC agents, when the gradients are computed on the workers.
For more information on configuring your training to use parallel computing, see the
ParallelizationOptions options in
To benefit from parallel computing, the computational cost for simulating the environment must be relatively expensive compared to the optimization of parameters when sending experiences back to the host. If the simulation of the environment is not expensive enough, the workers idle while waiting for the host to learn and send back updated parameters.
When sending experiences back from the workers, you can improve sample efficiency when the ratio (R) of the environment step complexity to the learning complexity is large. If the environment is fast to simulate (R is small), you are unlikely to get any benefit from experience-based parallelization. If the environment is expensive to simulate but it is also expensive to learn (for example, if the mini-batch size is large), then you are also unlikely to improve sample efficiency. However, in this case, for off-policy agents, you can reduce the mini-batch size to make R larger, which improves sample efficiency.
For an example that trains an agent using parallel computing in MATLAB, see Train AC Agent to Balance Cart-Pole System Using Parallel Computing. For an example that trains an agent using parallel computing in Simulink®, see Train DQN Agent for Lane Keeping Assist Using Parallel Computing.
When using deep neural network function approximators for your actor or critic
representations, you can speed up training by performing representation operations on a GPU
rather than a CPU. To do so, set the
UseDevice option to
opt = rlRepresentationOptions('UseDevice',"gpu");
The size of any performance improvement depends on your specific application and network configuration.
When validating your agent, consider checking how your agent handles the following:
Changes to simulation initial conditions — To change the model initial conditions, modify the reset function for the environment. For example reset functions, see Create MATLAB Environment Using Custom Functions, Create Custom MATLAB Environment from Template, and Create Simulink Environments for Reinforcement Learning.
Mismatches between the training and simulation environment dynamics — To check such mismatches, create test environments in the same way that you created the training environment, modifying the environment behavior.
As with parallel training, if you have Parallel Computing Toolbox software, you can run multiple parallel simulations on multicore computers. If
you have MATLAB
Parallel Server software, you can run multiple parallel simulations on computer clusters or
cloud resources. For more information on configuring your simulation to use parallel
If your training environment implements the
plot method, you can
visualize the environment behavior during training and simulation. If you call
plot(env) before training or simulation, where
is your environment object, then the visualization updates during training to allow you to
visualize the progress of each episode or simulation.
Environment visualization is not supported when training or simulating your agent using parallel computing.
For custom environments, you must implement your own
For more information on creating a custom environments with a
function, see Create Custom MATLAB Environment from Template.