How to check the change of weight in reinforcement learning training？

I have read the example "tune PI controller using reinforcement learning". And I wonder how to check the change of weights of the actor network every step? Could anyone help me? Thanks very much!

Answer 1

[Nithin]

Hi @Xianjin Zhu,

To monitor the change in weights of the actor network at every step during training in the example "Tune PI Controller Using Reinforcement Learning", you can use a custom callback function to record the model weights at each step.

You can create a custom callback function to extract and log the weights of the actor network which will be called at each training step.

global weightsHistory; 
weightsHistory = []; 

% Custom callback function to log actor weights 
function stop = logActorWeights(agent, ~, ~) 
    global weightsHistory; 
    actor = getActor(agent); 
    parameters = getLearnableParameters(actor); 
    % Append the current weights to the history 
    weightsHistory = [weightsHistory; parameters{1}(:)']; 
    stop = false; 
end 

"logActorWeights" function is defined to extract the learnable parameters (weights) of the actor network and append them to "weightsHistory".

Then, add the callback function to the training options

trainOpts = rlTrainingOptions(... 
    MaxEpisodes=maxepisodes, ... 
    MaxStepsPerEpisode=maxsteps, ... 
    ScoreAveragingWindowLength=100, ... 
    Verbose=false, ... 
    Plots="training-progress", ... 
    StopTrainingCriteria="AverageReward", ... 
    StopTrainingValue=-355, ... 
    Callbacks=@logActorWeights); % Add the callback function 

Hope this answers your query.