rlSimulinkEnv

Create reinforcement learning environment using dynamic model implemented in Simulink

Since R2019a

Syntax

env = rlSimulinkEnv(mdl,agentBlocks)

env = rlSimulinkEnv(mdl,agentBlocks,obsInfo,actInfo)

env = rlSimulinkEnv(___,'UseFastRestart',fastRestartToggle)

Description

The rlSimulinkEnv function creates a reinforcement learning environment object from a Simulink^® model. The environment object acts an interface so that when you call sim or train, these functions in turn call the Simulink model to generate experiences for the agents.

env = rlSimulinkEnv(mdl,agentBlocks) creates the reinforcement learning environment object env for the Simulink model mdl. agentBlocks contains the paths to one or more reinforcement learning agent blocks in mdl. If you use this syntax, each agent block must reference an agent object already in the MATLAB^® workspace.

env = rlSimulinkEnv(mdl,agentBlocks,obsInfo,actInfo) creates the reinforcement learning environment object env for the model mdl. The two cell arrays obsInfo and actInfo must contain the observation and action specifications for each agent block in mdl, in the same order as they appear in agentBlocks.

env = rlSimulinkEnv(___,'UseFastRestart',fastRestartToggle) creates a reinforcement learning environment object env and additionally enables fast restart. Use this syntax after any of the input arguments in the previous syntaxes.

Examples

Create Simulink Environment Using Agent in Workspace

Create a Simulink environment using the trained agent and corresponding Simulink model from the Create Simulink Environment and Train Agent example.

Load the agent in the MATLAB® workspace.

load rlWaterTankDDPGAgent

Create an environment for the rlwatertank model, which contains an RL Agent block. Since the agent used by the block is already in the workspace, you do not need to pass the observation and action specifications to create the environment.

env = rlSimulinkEnv("rlwatertank","rlwatertank/RL Agent")

env = 
SimulinkEnvWithAgent with properties:

           Model : rlwatertank
      AgentBlock : rlwatertank/RL Agent
        ResetFcn : []
  UseFastRestart : on

Validate the environment by performing a short simulation for two sample times.

validateEnvironment(env)

You can now train and simulate the agent within the environment by using train and sim, respectively.

Create Reinforcement Learning Environment for Simulink Model

For this example, consider the rlSimplePendulumModel Simulink model. The model is a simple frictionless pendulum that initially hangs in a downward position.

Open the model.

mdl = "rlSimplePendulumModel";
open_system(mdl)

Create rlNumericSpec and rlFiniteSetSpec objects for the observation and action information, respectively.

The observation is a vector containing three signals: the sine, cosine, and time derivative of the angle.

obsInfo = rlNumericSpec([3 1]) 

obsInfo = 
  rlNumericSpec with properties:

     LowerLimit: -Inf
     UpperLimit: Inf
           Name: [0x0 string]
    Description: [0x0 string]
      Dimension: [3 1]
       DataType: "double"

The action is a scalar expressing the torque and can be one of three possible values, -2 Nm, 0 Nm and 2 Nm.

actInfo = rlFiniteSetSpec([-2 0 2])

actInfo = 
  rlFiniteSetSpec with properties:

       Elements: [3x1 double]
           Name: [0x0 string]
    Description: [0x0 string]
      Dimension: [1 1]
       DataType: "double"

You can use dot notation to assign property values for the rlNumericSpec and rlFiniteSetSpec objects.

obsInfo.Name = "observations";
actInfo.Name = "torque";

Assign the agent block path information, and create the reinforcement learning environment for the Simulink model using the information extracted in the previous steps.

agentBlk = mdl + "/RL Agent";
env = rlSimulinkEnv(mdl,agentBlk,obsInfo,actInfo)

env = 
SimulinkEnvWithAgent with properties:

           Model : rlSimplePendulumModel
      AgentBlock : rlSimplePendulumModel/RL Agent
        ResetFcn : []
  UseFastRestart : on

You can also include a reset function using dot notation. For this example, randomly initialize theta0 in the model workspace.

env.ResetFcn = @(in) setVariable(in,"theta0",randn,"Workspace",mdl)

env = 
SimulinkEnvWithAgent with properties:

           Model : rlSimplePendulumModel
      AgentBlock : rlSimplePendulumModel/RL Agent
        ResetFcn : @(in)setVariable(in,"theta0",randn,"Workspace",mdl)
  UseFastRestart : on

Create Simulink Environment for Multiple Agents

Create an environment for the Simulink model from the example Train Multiple Agents to Perform Collaborative Task.

Load the agents in the MATLAB workspace.

load rlCollaborativeTaskAgents

Create an environment for the rlCollaborativeTask model, which has two agent blocks. Since the agents used by the two blocks (agentA and agentB) are already in the workspace, you do not need to pass their observation and action specifications to create the environment.

env = rlSimulinkEnv( ...
    "rlCollaborativeTask", ...
    ["rlCollaborativeTask/Agent A","rlCollaborativeTask/Agent B"])

env = 
SimulinkEnvWithAgent with properties:

           Model : rlCollaborativeTask
      AgentBlock : [
                     rlCollaborativeTask/Agent A
                     rlCollaborativeTask/Agent B
                   ]
        ResetFcn : []
  UseFastRestart : on

You can now simulate or train the agents within the environment using sim or train, respectively.

Input Arguments

mdl — Simulink model name

string | character vector

Simulink model name, specified as a string or character vector. The model must contain at least one RL Agent block.

agentBlocks — Agent block paths

string | character vector | string array

Agent block paths, specified as a string, character vector, or string array.

If mdl contains a single RL Agent block, specify agentBlocks as a string or character vector containing the block path.

If mdl contains multiple RL Agent blocks, specify agentBlocks as a string array, where each element contains the path of one agent block.

mdl can contain RL Agent blocks whose path is not included in agentBlocks. Such agent blocks behave as part of the environment, selecting actions based on their current policies. When you call sim or train, the experiences of these agents are not returned and their policies are not updated.

Multi-agent simulation is not supported for MATLAB environments.

The agent blocks can be inside of a model reference. For more information on configuring an agent block for reinforcement learning, see RL Agent.

obsInfo — Observation information

rlNumericSpec object | rlFiniteSetSpec object | array of rlNumericSpec objects | array of rlFiniteSetSpec objects | cell array

Observation information, specified as a specification object, an array of specification objects, or a cell array.

If mdl contains a single agent block, specify obsInfo as an rlNumericSpec object, an rlFiniteSetSpec object, or an array containing a mix of such objects.

If mdl contains multiple agent blocks, specify obsInfo as a cell array, where each cell contains a specification object or array of specification objects for the corresponding block in agentBlocks.

For more information, see getObservationInfo.

actInfo — Action information

rlNumericSpec object | rlFiniteSetSpec object | cell array

Action information, specified as a specification object or a cell array.

If mdl contains a single agent block, specify actInfo as an rlNumericSpec or rlFiniteSetSpec object.

If mdl contains multiple agent blocks, specify actInfo as a cell array, where each cell contains a specification object for the corresponding block in agentBlocks.

For more information, see getActionInfo.

fastRestartToggle — Option to toggle fast restart

'on' (default) | 'off'

Option to toggle fast restart, specified as either 'on' or 'off'. Fast restart allows you to perform iterative simulations without compiling a model or terminating the simulation each time.

For more information on fast restart, see How Fast Restart Improves Iterative Simulations (Simulink).

Output Arguments

env — Reinforcement learning environment

SimulinkEnvWithAgent object

Reinforcement learning environment, returned as a SimulinkEnvWithAgent object.

For more information on reinforcement learning environments, see Create Simulink Reinforcement Learning Environments.

Version History

Introduced in R2019a