How to restart TrainingProgressMonitor after loading from checkpoint?

Hello,

I'm running a custom training loop in Deep Learning Toolbox and am using TrainingProgressMonitor to record progress.

Occasionally, I take checpoints bt saving all variables into a file.

When I restart from a checkpoint by loading from the previously saved file, the monitor is stopped (monitor.Stop property is true).

How can I restart the monitor or alternatively merge the previously recorded data into a new one?

Thx

Answer 1

[Ayush Aniket]

This is a known limitation of the TrainingProgressMonitor in MATLAB's Deep Learning Toolbox: once the monitor is stopped (i.e., monitor.Stop == true), it cannot be restarted or resumed directly. This typically happens when you save and reload your workspace from a checkpoint.

Here are two practical workarounds:

1. You can instantiate a new TrainingProgressMonitor after loading your checkpoint and continue logging from where you left off as shown below:

% After loading checkpoint
monitor = trainingProgressMonitor;

% Reinitialize monitor settings
monitor.Info = ["Epoch", "Iteration"];
monitor.Metrics = ["Loss", "Accuracy"];
monitor.XLabel = "Iteration";

% Optionally, set the starting point
monitor.Progress = previousProgress;  % if you saved this

You will need to manually track and re-log any previous metrics if you want continuity in the visual plot.

2. If you saved the monitor’s data (e.g., metrics, progress values) before checkpointing, you can replot or merge them into a new monitor:

% Load previous metrics
load('checkpoint.mat', 'previousLoss', 'previousAccuracy');

% Create new monitor
monitor = trainingProgressMonitor;
monitor.Metrics = ["Loss", "Accuracy"];

% Re-log previous metrics
for i = 1:length(previousLoss)
    recordMetrics(monitor, i, ...
        Loss=previousLoss(i), ...
        Accuracy=previousAccuracy(i));
end

This won’t “resume” the monitor per se, but it gives you a visual continuation.

To make merging easier, consider saving your metrics separately during training:

lossHistory(end+1) = currentLoss;
accuracyHistory(end+1) = currentAccuracy;

Then, when restarting, you can reconstruct the monitor’s state.