ImageDataStore and tall array, How to use to save Labels and 4D Matrices in for loop?

4 views (last 30 days)
M on 7 Nov 2023
Commented: Matt J on 13 Nov 2023
I generate 4D matrices of images (Double) and their respective labels (Double) with huge sizes in each iteration,
for example in one iteration I get images with size 120*120*1*6000 where 6000 the number of images, 1 is the number of channels, and their respective lables 1*6000
How to use ImageDataStore and tall array to store Labels and 4D Matrices in for loop in each iteration to use it then in further analysis in machine learning?
Also, Is there any other efficient solution to deal with huge data?
Note that the number of images and their labels vary in each iteration
M on 13 Nov 2023
Edited: M on 13 Nov 2023
@Walter Robersonany hint how to pass in each iteration the 4D double matrices and double lables to imagedatastore and tall arrays please?
Because in the documentation there is no double format supported!
SupportedOutputFormats: ["png" "jpg" "jpeg" "tif" "tiff"]
M on 13 Nov 2023

Jorrit Montijn on 7 Nov 2023
I'm not sure what your workflow exactly looks like, but if you transform your images from double-precision floating numbers (64 bits per pixel) to uint8 (8 bits per pixel) you'll only need 1/8th of the memory capacity.
In most cases this won't even cause a loss of data, as images are often encoded in 8-bit depth anyway. In each iteration, you can then transform the (set of) image(s) you're working on back to double-precision floats so you don't have to change your workflow.
Transform the whole stack:
Ims_uint8 = cast(Ims_double*255,'uint8');
Then in each iteration:
Ims_double = cast(Ims_uint8,'double')/255;
Walter Roberson on 7 Nov 2023
im2uint8 and im2double potentially
M on 12 Nov 2023
@Walter Roberson @Jorrit Montijn extremely slow, not efficient solution

Matt J on 7 Nov 2023
imageDatastores are provided for situations like that.
M on 13 Nov 2023
Edited: M on 13 Nov 2023
@Matt J still I am not getting the process, what do you mean by Each image should be kept in its own file, which the data store would read in one-by-one as you loop
do you mean that I have to save all the files first then put them in the imds? What is the benefits of doing that??
If your images must be doubles (though I wonder why),
to not change the work flow
you can put them in individual .mat files.
how to do that? by them in the desk??
I cant find any simple example, the process is complex!
Matt J on 13 Nov 2023
do you mean that I have to save all the files first then put them in the imds? What is the benefits of doing that??
Yes. That way, the images are stored on disk, rather than consuming RAM.

Image Analyst on 7 Nov 2023
Cell arrays are very inefficient compared to regular numerical arrays. They use a lot more memory. You could even use single instead of double to preallocate using half the memory.
Not sure what you're doing but you may be doing what a lot of beginners do and that is to read ALL the input images into one huge array in a loop, then after the loop process each slice of the huge array as an individual image. This is usually NOT the way to process a sequence of images. You usually (if it's a series of 2-D images you want to analyze and not a volumetric image that you're reading in slice-by-slice) want to have a loop where you read in one image at a time and then analyze it immediately in that same iteration. See the FAQ:
Another option is an imageDatastore as @Matt J already mentioned.
If you must read all the images into memory at once time, then try to use a single array rather than a cell array.
Walter Roberson on 7 Nov 2023
A cell uses about 104 extra bytes per initialized cell entry.