How can I provide constraints to the actions provided by the Reinforcement Learning Agent?

Hi,

I have a reinforcement learning environment on simulink with 2 observations and 5 continuous Actions. I want to add constraints to the actions.

Example:

Action(1)+Action(2)+Action(3)+Action(4)+Action(5) should not exceed 1.

I have provided upper and lower limits for the actions. But, is there a generic method to relate all the 5 action values with the above constraint and send out the value to simulink?

Any help on the same would be really grateful !

Thanks in Advance !

Answer 1

[Emmanouil Tzorakoleftherakis]

Hard constraints are not typically supported during training in RL. You can specify limits/constraints as you mention above, but this is only taking the ouput of the policy and saturating it as needed, meaning that only the original action is considered when the agent is collecting data/training, not the saturated one (that will affect convergence, etc).

One way to add soft constraints is by adding penalties in the reward signal. It's still not guaranteed that it will be respected, but at least these are taken into account during training as they directly change the reward. See this example for instance.