Where and with how much data is the initial normalization done when training a network

When I start a network training (say VGG16) there is an initial step of data normalization. I was expecting all the dataset to load on the RAM, perform the corresponding preprocessing (done by the CPU) and later on, whenever it is needed, compose each batch, do the required additional preprocessing over the batch (done again by the CPU) and then transfer it to the GPU memory. However, seeing the behaviour while running my procedure, I can see that the RAM load does not go so high when doing the aforementioned data normalization (zeromean in this case). Why is this? Am I missing something? Is this initial preprocessing done per batch and stored again in the main memory or something like that?

Thank you so much in advance,

(By the way, sorry if this is a so basic question).

[yanqi liu]

yes，sir，i aslo has this confused， may be the imageDatastore and augmentedImageDatastore use on cpu to prepare data，and then make them to gpu and train

Answer 1

[Sahil Jain]

Hi Iker, the mean and standard deviation statistics are calculated iteratively by computing them for each mini-batch and then merging them with the running statistics. Since it's done batch-wise, not all the data is loaded onto the RAM so RAM usage isn't high. These calculated statistics are then used by the imageInputLayer to perform normalization.