Clear Filters
Clear Filters

使用PPO和TRPO​算法在reinfor​cement learning design app输出连续动作时,动作值不在设定好的区间内

15 views (last 30 days)
宝
on 17 Feb 2023
Answered: Aiswarya on 27 Feb 2024
%Open model
mdl='FCEV';
blk='FCEV/RL Agent';
%open_system(mdl);
%(s,a)
obsInfo = rlNumericSpec([3 1]);
obsInfo.Name = 'observations';
obsInfo.Description = 'A, B, and C';
numObservations = obsInfo.Dimension(1);
actInfo = rlNumericSpec([1 1],'LowerLimit',0,'UpperLimit',35);
actInfo.Name = 'Pbat_factor';
numActions = actInfo.Dimension(1);
%env
env = rlSimulinkEnv(mdl,blk,...
obsInfo,actInfo);
需要的动作区间是在0到35连续输出,但是在reinforcement learning design app中使用PPO或者TRPO算法输出的动作不在区间内,甚至一直输出负值。请问是网络结构设置有问题还是什么原因?

Answers (1)

Aiswarya
Aiswarya on 27 Feb 2024
请注意,我将用英语回答这个问题。
(Please note that I will be answering the question in English.)
I understand that you are using the PPO/TRPO agent and trying to set the "LowerLimit" and "UpperLimit" of data space as "actInfo" using "rlNumericSpec" function. But the action output values are not in the range specified by you.
The action bounds depend on the type of agent. Both PPO and TRPO are on-policy agents and they don’t enforce constraints set in the action specification (specified using "rlNumericSpec") for continuous action spaces. If you want to enforce these limits you have to do it explicitly on the environment side.
You may refer to the "rlTRPOAgent" documentation which also mentions the fact that action bounds need to be set by the user within the environment:
Also note that this is not the case with agents like SAC, for which the action bounds can be enforced with "rlNumericSpec".

Products


Release

R2022b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!