GPU error + Working with total XTrain data in custom training loop

Hi. In my custom training loop, (at the moment) I do not want to use mini-batches. Instead I need to work with all 60,000 images XTrain (full batch). I have to check my solver performance without using minibatch at first. My Matlab runs with high performance processor as default.

By the way, for this (a part of my) code I received following error:

I will be thankful for your help to find a solution for this problem.

Answer 1

[Joss Knight]

The error is telling you you can't process all your data at once because you don't have enough GPU memory. You either have to use mini-batches, or you could try using the CPU instead.

You could try getting a GPU with more memory, but the raw arithmetic doesn't work in your favour. Even if you had 12GB of memory, the largest resolution image you could fit on it is around 230x230 if you want to fit all 60,000; for training you need approximately as much memory as your input data multiplied by the number of layers in your network.

Implementing custom training with batching is relatively simple with minibatchqueue. Take a look. You could even use an arrayDatastore as input.

[MAHSA YOUSEFI]

Thanks for answer. Yes, I see using minibatches is simple and my code works with it. But as I told, I have to check without using them as well to compar my own optimizer's performance in terms of it. I already wrote my code for a fully connected network from 0 to 100, (my own forward and bacward process I mean) and used all the data (28*28*1*600000 which stored in a matrix X with 784*600000) and get the result. But for RGB with 3 channels through CNN I had to use custom training loop for using its dlfeval(@modelgradient,...). Now, in this way I have gpu limitation to call a all rgb images. You mentioned I can fit on 230 by 230 image size; my data set has samples in 28*28*3*600000. To use more gpu capacity, can I call matlab in google colab, to call my m-files and run the codes? what should I do if it is possible? (with cpu my training loop takes long to do process on all images (full batch data)).

[Joss Knight]

It's pretty hard to tell because I don't know what network you're using, presumably not one with too many layers. Clearly when you use grayscale data, which takes 179MB, it works and with colour data it doesn't, so the answers are basically: use a GPU with 3 times as much memory, or use 3 GPUs and train in parallel. If you want to do that via Google colab or some other cloud system then go for it.

Of course, training on 3 GPUs isn't exactly the same as training on 1 GPU with 3 times as much memory but the maths is the same.

Take a look at this documentation page to learn how to do multi-GPU training with a custom training loop.