MATLAB Answers

How to increase reading speed from a Gigabyte large file ?

13 views (last 30 days)
farzad
farzad on 17 Jun 2019
Commented: farzad on 20 Jun 2019
Hi all
how do I increase reading speed from an Excel file that contains rows and columns with a volume of some GigaBytes?

  18 Comments

Walter Roberson
Walter Roberson on 18 Jun 2019
I created a format with repmat of '%f' 50 times. I fopen and then
datacell = textscan(fid, fmt, 'collectoutput', 1);
Because this puts everything into a single cell the overhead to extract the array is trivial.
The timing with collectoutput 0 without joining the columns after, was a hair higher but not statistically significant.
dpb
dpb on 18 Jun 2019
Yeah, that's kinda' what I suspected, thanks for confirming, Walter.
I still find it more than strange that there's 30% reduction over fscanf -- what are they doing wrong with it then is the question that there's that much room for improvement?
These timings couldn't possibly be related to caching issues, I presume; you're too careful for that! :)
farzad
farzad on 20 Jun 2019
Thank you all guys ! It was a great answer ! thank you

Sign in to comment.

Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!