Neural Network Checkpoint Frequency

I am training a U-net for image segmentation and I noticed that it seems MATLAB does not allow you to specify the number of iterations per epoch in the training options, and instead defines an epoch as a single run through all the training images. I am using a very large number of images, so that the network runs through around 25,000 iterations per epoch and I would like to be able to test checkpoint networks more frequently than this.

Is there any way to save checkpoint networks more frequently than once per epoch, or to specify iterations per epoch?

Answer 1

[Akshay Kumar]

To reduce the number of images(iterations) the network sees in every epoch, you can specify a mini batch.

For example: If you run mini-batch update with batch size = b, every parameter update requires your algorithm see b of n training instances, i.e., every epoch your parameters are updated about n/b times.

You can specify the mini batch size using the 'MiniBatchSize' name-value argument of the 'trainingOptions' function.You can refer the below documentation for more information regarding this:https://www.mathworks.com/help/deeplearning/ref/trainingoptions.html

It is not possible to save checkpoints more than once per epoch, however by this you can save checkpoints based on your mini batch size.

[Ash Ash]

Is it possible to save checkpoints every x number of epochs insteadof every epoch ?

[Jordan Houri]

Thank you Akshay. Unfortunately I am already using a minibatch size of 4, which is just about the maximum that memory allows with 512 by 512 images. It seems there is nothing I can do about it then.