MATLAB Answers

0

Saving many variables as NetCDF makes file too large. How to reduce?

Asked by Macarena on 28 Mar 2018
Latest activity Edited by Macarena on 2 Apr 2018
Hello all,
I would REALLY appreciate any help on this, as I looked online but didn't find straight answers. I am trying to save many variables of different sizes as NetCDF format in this way:
numrow = 576;
numcol = 48;
numlayers = 125;
numtime_save = numtime + 1; %This is usually a value of '8'
%%Define the dimensions
netcdf.setDefaultFormat('NC_FORMAT_CLASSIC') ;
ncid = netcdf.create(files_name,'NC_WRITE');
dimidrow = netcdf.defDim(ncid,'rows',numrow);
dimidcol = netcdf.defDim(ncid,'length',numcol);
dimidlay = netcdf.defDim(ncid,'layers',numlayers);
dimidtime = netcdf.defDim(ncid,'time',numtime_save);
%%Define the name of variables, with dimensions
%%Question: do these all really have to be double precision???
varmask = netcdf.defVar(ncid,'cloudmask','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);
varmask_sigma = netcdf.defVar(ncid,'cloudmask_sigma','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);
varmask_clear = netcdf.defVar(ncid,'cloudmask_clear','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);
varmask_clear_sigma = netcdf.defVar(ncid,'cloudmask_clear_sigma','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);
varmask_radar = netcdf.defVar(ncid,'cloudmask_radar','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);
varmask_radar_sigma = netcdf.defVar(ncid,'cloudmask_radar_sigma','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);
varmask_both = netcdf.defVar(ncid,'cloudmask_both','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);
varmask_both_sigma = netcdf.defVar(ncid,'cloudmask_both_sigma','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);
varmask_lidar = netcdf.defVar(ncid,'cloudmask_lidar','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);
varmask_lidar_sigma = netcdf.defVar(ncid,'cloudmask_lidar_sigma','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);
varheight = netcdf.defVar(ncid,'height','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);
varheight_sigma = netcdf.defVar(ncid,'height_sigma','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);
varN = netcdf.defVar(ncid,'N','NC_BYTE',[dimidrow dimidcol dimidtime]);
%%end the variable naming and dimensions, then actually input
%%variables into netcdf file, then close the netcdf file
netcdf.endDef(ncid);
netcdf.putVar(ncid,varmask,cloudmask);
netcdf.putVar(ncid,varmask_sigma,cloudmask_sigma);
netcdf.putVar(ncid,varmask_clear,cloudmask_clear);
netcdf.putVar(ncid,varmask_clear_sigma,cloudmask_clear_sigma);
netcdf.putVar(ncid,varmask_radar,cloudmask_radar);
netcdf.putVar(ncid,varmask_radar_sigma,cloudmask_radar_sigma);
netcdf.putVar(ncid,varmask_both,cloudmask_both);
netcdf.putVar(ncid,varmask_both_sigma,cloudmask_both_sigma);
netcdf.putVar(ncid,varmask_lidar,cloudmask_lidar);
netcdf.putVar(ncid,varmask_lidar_sigma,cloudmask_lidar_sigma);
netcdf.putVar(ncid,varheight,height);
netcdf.putVar(ncid,varheight_sigma,height_sigma);
netcdf.putVar(ncid,varN,N);
netcdf.close(ncid);
clear mex
However, when I save it this way, the file size is 1.33 GB! That is huge...If I save the files as:
save(filename, 'cloudmask','cloudmask_sigma','cloudmask_clear','cloudmask_clear_sigma',...
'cloudmask_radar','cloudmask_radar_sigma','cloudmask_both','cloudmask_both_sigma',...
'cloudmask_lidar','cloudmask_lidar_sigma','height','height_sigma','N');
this file has a size of 16 MB. I would REALLY like to save in NetCDF format, because my boss needs it this way. I have many files to save, this is just an example but it is inside of a loop. PLEASE HELP!

  5 Comments

This worked! Thank you! I did this:
example:
comp = 5; %compression size
varmask_radar = netcdf.defVar(ncid,'cloudmask_radar','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);
netcdf.defVarDeflate(ncid,varmask_radar,true,true,comp);
on all the variables, and now my file has been reduced to 6.3MB!!
You should post that as an Answer so we can vote for it. (You did the work of putting together the answer that would be of use to other people.)

Sign in to comment.

1 Answer

Answer by Macarena on 2 Apr 2018
Edited by Macarena on 2 Apr 2018
 Accepted Answer

%Example for one variable. Have to do on all
numrow = 576;
numcol = 48;
numlayers = 125;
numtime_save = numtime + 1; %This is usually a value of '8'
comp = 5; %compression size, for example
%%Define the dimensions
netcdf.setDefaultFormat('NC_FORMAT_CLASSIC') ;
ncid = netcdf.create(files_name,'NC_WRITE');
dimidrow = netcdf.defDim(ncid,'rows',numrow);
dimidcol = netcdf.defDim(ncid,'length',numcol);
dimidlay = netcdf.defDim(ncid,'layers',numlayers);
dimidtime = netcdf.defDim(ncid,'time',numtime_save);
varmask_radar = netcdf.defVar(ncid,'cloudmask_radar','NC_FLOAT',[dimidrow dimidcol dimidlay dimidtime]);
netcdf.defVarDeflate(ncid,varmask_radar,true,true,comp);
netcdf.endDef(ncid);
netcdf.putVar(ncid,varmask_radar_sigma,cloudmask_radar_sigma);
netcdf.close(ncid);
clear mex

  0 Comments

Sign in to comment.