Query regarding matlab maximum allowed size of an array

I need to assign more then 16GB array for which it is showing following error. (I am assigning using x = zeros(size)):

Requested 50000x50000 (18.6GB) array exceeds maximum array size preference (6.0GB). This

might cause MATLAB to become unresponsive.

Now I found created three different array with x = zeros(28000); which is approximately 6 GB for each. Even then also on assigning third array it is giving "Out of Memory" error.

I also tried using tall array as follows. But still it is giving of out of memory error

a = tall(zeros(28000));

I checked my system memory using memory command as follows:

>> memory

Maximum possible array: 8600 MB (9.018e+09 bytes) *

Memory available for all arrays: 8600 MB (9.018e+09 bytes) *

Memory used by MATLAB: 3256 MB (3.414e+09 bytes)

Physical Memory (RAM): 6100 MB (6.396e+09 bytes)

* Limited by System Memory (physical + swap file) available.

So, Is there any way to assign and perform calculation on more than 16 GB array?

[Matt J]

1. Have you considered sparse?
2. How much disk space do you have?
3. How much memory beyond this one array will you need to simultaneously use?

[Rik]

Since your actual memory doesn't allow for arrays this size, it will have to be stored in a page file, which will be painfully slow.

Generally, needing a large array is a sign that you're doing something wrong in your implementation. Perhaps if you explain what you want to do, we might be able to find a solution that doesn't require such large arrays.

[John D'Errico]

You want to do something that effectively (compared to what you have) requires a supercomputer. Well, again, compared to what you have, it would be one. Do you have a supercomputer? That is, one with probably at least 32 GB of memory? And preferably 3x as much, to be sure. Because you will never just have one such array. Copies will be made on the fly to do stuff. If you do not have that supercomputer, then what do you expect? The little toy thing you do have cannot leap tall buildings in a single bound. It trips over train tracks. You just want it to do more.

So, you can either get more memory, or you can change your requirements, making your problem smaller, or you can find a bigger computer. And that does not mean finding another way to let MATLAB allow you to try to allocate 16GB of memory on your existing machine. You still cannot stuff 10 pounds of crap in a 5 pound bag, no matter how diligently you try. I'm sorry if I have said this too plainly for you, but that is the essential problem.

If you want help, then you need to explain enough about what you are doing, NOT just what you think you want to do at this step, since what you think you need to do now is create a 16GB array on a 6GB computer. You need to explain what you are doing. Go back to the beginning. That may help someone to show you how to do what you need to do, perhaps using sparse matrices. Or possibly you might be able to get away with allocating a uint8 or int8 array instead. That might just barely fit into your memory. (Until next month, when you will decide to make the problem larger. John's rule applies here, that computer problems always expand to be just a little larger than the largest computer resource you have available. The easy problems have already been solved by someone else.)

[yash]

I have 8GB of installed RAM. I will need two arrays of the same size

Actually, I want to read binary file, which contains around 1.9 GB of data. Now I want to extract this data and multiply with one constant to make it array of double of approximate size 15.2 GB. This data will be considered as my Y axis data and I want to calculate x axis data which will be of same size. And I want to plot this x and y data.

So, what is the maximum size of an array which can be stored in this machine with Matlab.

[Rik]

You're in luck: you can't see millions of data points at the same time anyway, so you can read parts of your file and do the processing you need to do that way. For each chunk you can collate the data you want to plot, and once you have gathered all chunks, your plot is done.

The processing will probably be faster if you choose an array size that is a lot closer to your CPUs cache size than your RAM. I would suggest a few thousand elements at most. You can experiment to find the optimal chunk size (that balances file IO, processing time, and time spent on collation) once you have the code ready to process one chunk.

[yash]

Actually I want whole data in an array at a time. Can I use tall arrays or parallel processing etc. to achieve the goal? Or increasing RAM size is the only option.

[Rik]

Previously you stated you wanted to create some sort of plot. That seems a much more sensible end goal than just having an array in RAM. Why to you want to store the whole data in RAM? What does that achieve that isn't already achieved by having it on disk?

You want to DO something. You should explain what that is. If you indeed actually want to have the entire array in memory, you could try a tall array, but that will not actually change your situation, since it which will page to disk. You can't magically fit more in memory than you have.

Can you give a small example of what plot you're looking for? Attach a mat file with 200 elements and format your code so it can run in the forum interface. That way we can help you find a way process all you data without having to resort to magic.

[DGM]

The fact that those are your array sizes tells me that your data is stored in uint8 format. If the only reason you're converting it to double is to scale it by some non-integer scaling factor for display purposes, then you can possibly avoid the whole problem and just keep the data in uint8.

I'm assuming the following:

• The data is stored in uint8
• The scaling is just a simple global linear transformation
• The data are just vectors to be plotted with a line/scatter plot or something similar

On those assumptions, consider the example where no rescaling is used:

% these are our uint8-scaled arrays (1x100 vectors)
load uint8_x.mat
load uint8_y.mat

% this is where our xy data are in uint8-scale
xurange = imrange(xu);
yurange = imrange(yu);

% let's say we know these are the extrema of our x,y data after rescaling
% you could also just calculate these by transforming xurange,yurange
% using whatever scaling operation you would have used on the data
xrange = [0 2];
yrange = [0 4];

% plot the data without rescaling anything
hp = plot(xu,yu);

% set plot extents in uint8 scale
xlim(xurange) 
ylim(yurange)

% arrange the ticks and ticklabels
numticks = [6 6]; % number of ticks [x y]
% place the tick positions in uint8 scale
xticks(linspace(xurange(1),xurange(2),numticks(1)));
yticks(linspace(yurange(1),yurange(2),numticks(2)));
% label the ticks as if rescaled
xticklabels(linspace(xrange(1),xrange(2),numticks(1)));
yticklabels(linspace(yrange(1),yrange(2),numticks(2)));

That could probably be simplified, but it should demonstrate the concept.

While that keeps the data arrays from ballooning by a factor of 8, you still have 1.9 billion data points plotted in a raster image which is no larger than your screen. For a simple single-valued function like this, you have several orders of magnitude more samples than you can physically display. Graphs and charts are visualizations of data, not stores of data. You might not want to downsample your data, but your display is going to eliminate information whether you want to or not. What's appropriate depends on the data, how it's being plotted, and what information is important.

[yash]

Okay, I understood I cannot have full array at a time in my memory due to memory limitation. My idea is I will read data in an in-memory array according to my available memory and rest of data I will dump in a binary file. Now, I need to extract a part of this file as and when needed from particular index. I see there are two options for doing this.

1) data store and tall arrays

2) memmapfile

So, for this application which is the best way of doing this and can you provide me one small example of doing this?

[Rik]

You are ignoring what we're telling you. You will need to write code capable of analyzing a small part of your data. Then you can process your full file in chunks. If you need help, you need to be explicit in what data you have and what output you want.