Understanding Action Dimension Formatting in MATLAB's DDPG with LSTM-Based Networks

In MATLAB's Reinforcement Learning Toolbox, when using DDPG with LSTM-based actor and critic networks, the conversion of actions to dlarray is handled automatically. Since users do not have direct control over this process: Are actions formatted with a 'T' (time) dimension or a 'C' (channel) dimension when passed between the actor and critic networks? How does MATLAB structure these actions to ensure compatibility with recurrent layers, such as aligning sequences for LSTM time steps?

Answer 1

[Snehal]

Hello @Abdolrazzagh,

I understand that you are trying to know how actions are formatted by the underlying mechanism when using MATLAB’s DDPG with LSTM-Based Networks.

Actions are formatted with a 'T' (time) dimension to ensure compatibility with LSTM layers.

The data is structured in the ‘CBT’ format to ensure that both actor and critic networks can process sequences effectively.

Therefore, MATLAB automatically handles the reshaping of actions to align them with the expected input format for LSTM layers.

For more insights, you may refer to the following documentation link:

https://www.mathworks.com/help/reinforcement-learning/ref/rl.agent.rlddpgagent.html#mw_692a723c-8e84-4d86-b43d-248fb989e57c:~:text=Create%20DDPG%20Agent%20Using%20Custom%20Recurrent%20Neural%20Networks

Below is the link to a similar MATLAB question addressed previously:

https://www.mathworks.com/matlabcentral/answers/2174967-how-does-matlab-internally-format-actions-as-dlarray-in-ddpg-with-recurrent-networks-lstm?s_tid=answers_rc1-2_p2_MLT

Hope this helps.