Issue with batch normalization layer of saved CNN

17 views (last 30 days)
When loading a previously trained CNN, I got an issue with the batch normalization layers. When looking into the loaded CNN model the trainable mean and variance are empty.
Name: 'batchnorm_1'
TrainedMean: []
TrainedVariance: []
So the checkpoint doesn't seem to save these parameters. Are there any workarounds for this issue? I am using Matlab R2018b.
  1 Comment
Wes Baldwin
Wes Baldwin on 29 Jul 2020
Doesn't this mean using checkpoints on networks with a batchnorm layer is useless??? Kinda a big deal for long training!!! You could potentially lose days or weeks of training with no option but to start from the beginning.

Sign in to comment.

Accepted Answer

Vivek Singh
Vivek Singh on 22 Mar 2019
We were able to reproduce the issue. We will inform you once the issue is fixed.
Since TrainedMean and TrainedVariance are calculated after the training is finished, therefore as a workaround you can use the below mentioned codes to explicitly save and load the Model.
%To save model with name "demoModel", assuming your network is in "net"
save('demoModel','net')
%To load model to variable net1
net1=load('demoModel.mat','net');
net1.net.Layers(n).TrainedMean %where n is the batch normalization layer
  4 Comments
ramin nateghi
ramin nateghi on 5 Nov 2020
Edited: ramin nateghi on 5 Nov 2020
Hello,
I also faced this problem. When a model is saved by "save" function, it is ok and the model contains all information (TrainedMean and TrainedVariance) of the trained batch normalization layers. But, when the model is saved by the"checkpoint" during training, both of the TrainedMean and TrainedVariance params became empty. This is a bug for "checkpoint".
Yi Wei
Yi Wei on 30 Nov 2020
Does it mean the saving process will cost too much time if a network contains normalization layers and the training data volume is large(e.g.8T)?

Sign in to comment.

More Answers (1)

Sam Leeney
Sam Leeney on 15 Dec 2022
For anyone else stuck, there is a fix here; https://uk.mathworks.com/matlabcentral/answers/423588-how-to-classify-with-dag-network-from-checkpoint

Products


Release

R2018b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!