Fastest way to save and retrieve floating point vectors
8 views (last 30 days)
Show older comments
I am writing a code that generates a row vector of size 1x600 (containing floating point numbers) in each iteration. There are about 100,000 such iterations performed in the entire code. Also, I have to read these saved vectors (row wise) in a different code at a later point of time. I have been using 'dlmwrite' so far for appending the vectors row wise to a .csv file, and the function 'load' for retrieving this data. I found that saving and retrieving data in this way is consuming a lot of time. I wanted to know if there is a better way of saving and retrieving these vectors.
I have read some posts in the forum suggesting to save the data in a binary file/.mat file as some alternatives. But I am not sure which would be the best for my case. Could someone please suggest me a faster way to do the above exercise?
0 Comments
Accepted Answer
Walter Roberson
on 11 Mar 2021
Simplest way, that will take very low overhead for your program (such as might be needed for real-time work):
fid = fopen(OutputFileName, 'w'); %not 'wt'
for ...
fwrite(fid, YourBufferOfDoubles);
end
fclose(fid)
However, you have about 480 megabytes worth of raw data, and that is enough that unless you are writing to SSD, that the file overhead is going to add up.
Because of that, you might want to consider methods that compress data as you go. You would buffer some data and call compression routines sometimes, and write out the compressed data. One way to do that would be https://docs.oracle.com/javase/7/docs/api/java/util/zip/GZIPOutputStream.html
A third way... just store all the values in memory as you produce them (having pre-allocated the array), and afterwards save() them to a .mat file. .mat does compression.
More Answers (0)
See Also
Categories
Find more on Data Import and Analysis in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!