What's the upper limit for how much data can be stored in a structure?

26 views (last 30 days)
I am working with about 20 datasets, each with a size of about 2500x300. Is it possible to open these datasets, store them, and save off the resulting structure as a mat file without running into memory issues? This is my current code, and I run into an "Out of memory" error when I run it.
for i=1:length(fieldnames(dat))
a=fieldnames(dat); %files to work with
a=a{i}; %selects upper structure
a2=fieldnames(foldername); %foldername where .dat files are held
a2=a2{i}; %select which folder
for j=1:length(fieldnames(dat.(strcat(a))))
b=fieldnames(dat.(strcat(a))); %list of filenames in upper structure
b=b{j}; %select file to open
for k=1:length(dat.(strcat(a)).(strcat(b)))
filepath = strcat(foldername.(strcat(a2)),'\', dat.(strcat(a)).(strcat(b))); %sets file path
fid = fopen(filepath{k}); %opens file
fmt = repmat('%s', 1, 360); %reads file, text string delimter
C= textscan(fid,fmt,'Delimiter','\t','CollectOutput',true); %records csv data to cell array C
fclose(fid); %closes file
DLG.(strcat(a)).(strcat(b)){k}=C{1}; %writes data to structure
end
end
end
  4 Comments
Guillaume
Guillaume on 15 May 2017
Edited: Guillaume on 15 May 2017
"I can't call Dat.a"
Indeed, the proper syntax is simply:
dat.(a)
No need for the strcat that does absolutely nothing.

Sign in to comment.

Accepted Answer

Guillaume
Guillaume on 15 May 2017
Your code is a bit confusing with all these apparently useless strcat (see comment to the question).
Anyway, to answer your question, you at least need 2500*300*20*8 bytes ~= 120 MB to store your datasets. Add to that the overhead of the structure fields (around 20*(numberofcharactersinfieldname + constant) and of the structure itself (not much).
Unless you've got very little memory on your computer or the data size is vastly different from what you've stated, it should all fit in memory.

More Answers (1)

Jan
Jan on 15 May 2017
A cleaner version of the code:
fmt = repmat('%s', 1, 360); %reads file, text string delimter
datFields = fieldnames(dat);
folderFields = fieldnames(foldername); % "foldername" is a bad name for a struct
for i = 1:length(datFields)
a = datFields{i}; %selects upper structure
a2 = folderFields{i}; %select which folder
aFields = fieldnames(dat.(a));
for j = 1:length(aFields)
b = aFields{j}; %select file to open
filepath = fullfile(foldername.(a2), dat.(a).(b)); %sets file path
DLG.(a).(b) = cell(1, length(dat.(a).(b))); % Pre-allocate
for k = 1:length(dat.(a).(b))
fid = fopen(filepath{k}); %opens file
C = textscan(fid,fmt,'Delimiter','\t','CollectOutput',true); %records csv data to cell array C
fclose(fid); %closes file
DLG.(a).(b){k} = C{1}; %writes data to structure
end
end
end
I tried to move all repeated work before the loops, removed the useless strcat and there is still much potential to improve the readability. Fieldnames like "a" and "b" are not clear.

Categories

Find more on Large Files and Big Data in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!