Reading a large binary file

15 views (last 30 days)
Dominik Rhiem
Dominik Rhiem on 18 Oct 2023
Commented: Dominik Rhiem on 19 Oct 2023
I want to load a 8 GB binary data file into Matlab. My computer has 24 GB of RAM. It runs out of memory, despite me closing literally everything besides Matlab, and I don't understand why. The file should not take more than its size, right?
Code for loading is simply:
fd = fopen(['data.bin'], 'r');
data = fread(fd, 'int16');
fclose(fd);

Accepted Answer

James Tursa
James Tursa on 18 Oct 2023
Edited: James Tursa on 18 Oct 2023
Try
data = fread(fd, '*int16');
When you use 'int16' for the type to read, MATLAB reads as int16 and then converts to double (four times the memory). When you use '*int16', MATLAB keeps the type as int16. E.g.,
% Create a small sample file
fid = fopen('junk.bin','wb');
fwrite(fid,int16(4),'int16');
fclose(fid);
% read it with 'int16'
fid = fopen('junk.bin','rb');
x = fread(fid,'int16')
x = 4
class(x) % double, not what you wanted
ans = 'double'
% read it with '*int16'
frewind(fid);
x = fread(fid,'*int16')
x = int16 4
class(x) % int16, this is what you wanted
ans = 'int16'
fclose(fid);
  1 Comment
Dominik Rhiem
Dominik Rhiem on 19 Oct 2023
Positive side effect: This is extremely fast.

Sign in to comment.

More Answers (0)

Categories

Find more on Large Files and Big Data in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!