MATLAB Answers

ncwrite creates files that are too large

6 views (last 30 days)
Jonathan King
Jonathan King on 2 Jun 2019
I'm creating some NetCDF4 files involving climate data. The datasets can expand along any dimension, so all dimensions need to be unlimited. To create the NetCDF files, I'm using ncwriteschema. Here is an example schema.
%% Initialize a NetCDF schema
schema = struct();
schema.Name = '/';
schema.Format = 'netcdf4';
%% Dimensions
% Latitude
schema.Dimensions(1).Name = 'lat';
schema.Dimensions(1).Length = 96;
schema.Dimensions(1).Unlimited = true;
% Longitude
schema.Dimensions(2).Name = 'lon';
schema.Dimensions(2).Length = 192;
schema.Dimensions(2).Unlimited = true;
% Time
schema.Dimensions(3).Name = 'time';
schema.Dimensions(3).Length = 12000;
schema.Dimensions(3).Unlimited = true;
% Run (ensemble member)
schema.Dimensions(4).Name = 'run';
schema.Dimensions(4).Length = 1;
schema.Dimensions(4).Unlimited = true;
%% The variable
schema.Variables(1).Name = 'gridData';
schema.Variables(1).Dimensions = schema.Dimensions(1:4);
schema.Variables(1).Datatype = 'double';
schema.Variables(1).Attributes(1).Name = 'Name';
schema.Variables(1).Attributes(1).Value = 'tas';
schema.Variables(1).Attributes(2).Name = 'Units';
schema.Variables(1).Attributes(2).Value = 'Kelvin';
This schema works fine for initializing the .nc file:
%% Initialize the NetCDF file by writing the schema
file = 'myfile.nc';
ncwriteschema( file, schema );
But I run into issues when I actually try to write data into the .nc file. For example, the lines:
%% Write data in the .nc file
data = rand(96, 192, 12000, 1);
ncwrite( file, 'gridData', data);
create ~1.75 GB of example data. However, when I use the ncwrite command, it seems to get stuck and writes ~50GB of data to the file. When I use the ncdisp command, it reports that only a (96 x 192 x 12000 x 1) matrix of ~1.75GB is stored in the file, but this is clearly not the case. This issue seems to be related to the number of dimensions in the schema. For example, if I add another dimension (for height) to the schema, ncwrite will continue well past 50GB until it fills my local (C:) drive and crashes.
What is causing this to happen?
(I'm using MATLAB R2018b with 64 bit Windows 10)

Answers (0)

Products


Release

R2018b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!