importing large 4GB+ tab-delimited ascii file makes MATLAB crash
1 view (last 30 days)
Show older comments
Sean Phillips
on 17 May 2019
Answered: Sean Phillips
on 18 May 2019
i didnt have a problem until now the data files have been scaled up in complexity
i usually use
IMPORT = importdata('components.dat')
in this case components.dat is 4 GB+
it is a table of numerical values consisting of 12 columns and millions of rows
for smaller models this has worked but now MATLAB crashes when i try to import at this size.
I have 96GB of RAM and dual Xeons.
Do i need to change setting in MATLAB or is there better ways to import this data
this is an exmaple of the file
1 0.01 -50.00 0.00 -3950.00 -2.0000E+04 0.000 -4000. -1.409E-15 -2.138E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9900E+04 0.000 -4000. -1.410E-15 -2.193E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9800E+04 0.000 -4000. -1.411E-15 -2.249E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9700E+04 0.000 -4000. -1.412E-15 -2.306E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9600E+04 0.000 -4000. -1.411E-15 -2.364E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9500E+04 0.000 -4000. -1.411E-15 -2.424E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9400E+04 0.000 -4000. -1.409E-15 -2.485E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9300E+04 0.000 -4000. -1.407E-15 -2.547E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9200E+04 0.000 -4000. -1.405E-15 -2.611E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9100E+04 0.000 -4000. -1.401E-15 -2.676E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.9000E+04 0.000 -4000. -1.397E-15 -2.742E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.8900E+04 0.000 -4000. -1.392E-15 -2.810E-15 0.00E+00 0.00E+00
1 0.01 -50.00 0.00 -3950.00 -1.8800E+04 0.000 -4000. -1.386E-15 -2.879E-15 0.00E+00 0.00E+00
it is tab delimited
0 Comments
Accepted Answer
More Answers (2)
Steven Lord
on 17 May 2019
Consider making a tall array or perhaps using memmapfile. These tools are linked from this documentation page.
per isakson
on 18 May 2019
Edited: per isakson
on 18 May 2019
That's strange. I use Win10 and R2018b (64bit). btw: What do you mean by "[...] up in complexity" ?
Anyhow try
>> num = cssm( 'components.dat' );
>> whos num
Name Size Bytes Class Attributes
num 13x12 1248 double
where
function num = cssm( ffs )
fid = fopen( ffs );
cac = textscan( fid, '%f%f%f%f%f%f%f%f%f%f%f%f' ...
, 'Delimiter','\t', 'CollectOutput',true );
fclose( fid );
num = cac{1};
end
If that doesn't work, I guess there is some strange stuff in the middle of your text file.
I added
, 'ReturnOnError',false
to the textscan call and the string, "an error", to the text file an run
>> tic, num = cssm( 'cssm.txt' );toc
Error using textscan
Mismatch between file and format character vector.
Trouble reading 'Numeric' field from file (row number 1005147, field number 1) ==> an error\n
Error in cssm (line 4)
cac = textscan( fid, '%f%f%f%f%f%f%f%f%f%f%f%f' ...
>>
textscan() found my string "an error" on the row 1005147.
0 Comments
See Also
Categories
Find more on Text Data Preparation in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!