DDPG algorithm/Experience Buffer/ rl.util.ExperienceBuffer

I want to code my own DDPG algorithm. In the intial steps, the Batch size is bigger than the number of experiences in experience buffer, how can I still get enough sampled data for my miniBatch ?

I use rl.util.ExperienceBuffer to create my experience buffer and use createSampledExperienceMiniBatch(buffer,BatchSize) function to get datas for minBatch. However, when the data in experience buffer is smaller than the BatchSize, the function return 0x0 cell.

Answer 1

[Aravind]

Hi @hieu nguyen,

To manage the initial phase when your experience buffer has fewer experiences than the desired mini-batch size in a DDPG algorithm, you consider one of these two options:

1. Adjust the Mini-Batch Size Dynamically: Use the current buffer size as the mini-batch size if it is smaller than the desired size when calling the “createSampledExperienceMiniBatch” function. This allows the agent to learn at each time step. However, the downside is that the agent might not explore sufficiently, potentially leading to a sub-optimal policy by the end of training.
2. Warm-Up Phase: Implement an initial phase where the agent collects experiences without updating the policy to ensure the buffer is adequately filled. This is a common approach in training a DDPG agent. The agent takes random actions until the buffer is filled with enough entries to match the batch size. Once the buffer length meets the batch size, you can begin training thus avoiding errors with the “createSampledExperienceMiniBatch” function.

I hope this addresses your query.