Matlab slowing down while reading netCDF

I'm experiencing a problem when I try to read data from a succession of netCDF files. After a certain point (~10 loops of variable y in the example below), the reading operation suddenly slows to a crawl (a few seconds per read -> hundreds or thousands of seconds per read) and never recovers (though it "happily" runs overnight if I let it). The way I resolve it is to save my read-in data frequently, and just kill then restart Matlab to pick up where it got itself into a tizz.

I am currently using customised libraries to read netCDF-4, but I've experienced the same problem in the past with netCDF-3. In the deeper past, and with a much older version of Matlab, I had a faintly similar problem that stemmed from me not closing the files I was reading from, but that doesn't seem to apply here (I'm not even sure if there's a way to close open files now).

About the only thing I've been able to establish is that it appears to be a function of the number of read operations (19 per file below) rather than the number of files being read (336 in total below).

Anyway, is there some basic rule of file management in Matlab that I'm ignorantly flouting? It wouldn't be the first time.

Andrew.

-----

fld2d = zeros([292 362 28 19]);
for y = 1:1:28
  for m = 1:1:12
    iname = sprintf('EXP500/%d/ORCA1_%dm%dD.nc', 1965+y, 1965+y, m);
    for f = 1:1:19
      t1 = nc_varget(iname, fld(f,1:flen(f)));
      fld2d(:,:,y,f) = fld2d(:,:,y,f) + (t1 / 12);
    end
  end
end

[Ashish Uthama]

Andrew, just curious to know if you tried the native support: http://www.mathworks.com/help/techdoc/ref/netcdf.html

[Andrew Yool]

Sorry, no. For historical reasons I'm back at v2008a, and have been using non-native libraries compiled for netCDF-4. I wasn't aware that netCDF-4 had any native support in Matlab, so hadn't looked this up yet. Anyway, does it? It looks like it does from the link above, but it also looks like I might need to know what I'm doing to set it up.

[Ashish Uthama]

It you are in a position to upgrade, version R2010b onwards support netcdf 4 out of the box, no additional set up required.

[Andrew Yool]

I'm afraid that's not an option just yet. We're stuck here at v2009a, and I'm a little worried that the precarious financial situation may persuade our IT services to skip upgrading for a bit. Anyway, thanks very much for letting me know about v2010b - I'll be prodding my IT group to see if I can convince them to adopt it.

Answer 1

[Andrew Yool]

For those who come across this archaic question while looking for an answer to a similar problem, a solution (or workaround) that I've found is to use the command ...

clear functions

I'm not sure what exactly this is doing, but it appears to clear out whatever buffer it is of Matlab's that is causing the problem, and it does so while leaving variables, etc., all in place.

Andrew.

[shunping chen]

I also tried clear mex, not getting better. It looks like clear makes no defferent for me.

I tried profile, but I'm not sure if I can get any improvements from the reports.

Answer 2

[Kelly Kearney]

What version of Matlab are you running? I don't have a clear answer for you, but I experienced some rather dramatic slowdown of the snctools functions when I upgraded to a version of Matlab with the native netcdf support. Though that was more of a across-the-board slowdown, rather than the gradual slowing you describe.

[Andrew Yool]

It's v2008a. As you'll see above, it's not the most recent available to me, though even that's below native support for netCDF-4. Anyway, this problem only occurs for me when I read in netCDF data, and it really does come inexplicably out of the blue. It's not even a gradual appearance - one read call will be fast then the next will be annoyingly slow (before increasing through "painfully slow" and "tortuously slow" to reach "intolerably slow" in short order). And it's not a memory thing either - the machine I'm currently running on has 200 Gb RAM to play with (and about double that in swap space).

Anyway, thanks for taking the time to reply. I think I'm just going to have to muddle along.

[Jan]

200 GB RAM. Wow. Is it free?

[Andrew Yool]

We (that's the institutional "we") bought it to do very specific tasks that involved large memory overheads. Basically high resolution ocean modelling (where "high" = 1/4 or 1/12 degree global resolution). The downside is that, while fast, its speed is not as impressive as its memory. So if too many people use it at once (not unusual), it can be slow even while it's more happily managing a vast memory overhead. Oh, and no, it's not free. ;-)

[Jerry Gregoire]

If you are not using the 64 bit matlab, on a 64 bit machine your 200GByte Ram will be worthless. ML will only see about 2GB for workspace use.

Answer 3

[Ashish Uthama]

I really have no clue, wild guesses:

• Instead of having to restart MATLAB, try issuing a clear mex after a read, how does this impact the next read?
• Does the order of the files being read make a difference? maybe the second file is 'different' in some way?
• Could you elaborate on what you mean by 'customized libraries'? might help some of us try to replicate this.

[Andrew Yool]

I'll try your first suggestion when I'm in Matlab next (should be later this PM). It certainly "feels" like Matlab has hit up against some limit, but I just can't see what this might be.

I don't think the file order matters at all. If I kill Matlab then start it up again, the "second" file is read just as fast at the "first" was on the previous instance. Usually, I'm quite a few files down before I hit up against any problems. And a restart of Matlab always solves them (... at first).

By "customised libraries" all I meant was that our third party toolbox for reading netCDF (which we started using before native netCDF was an option) was recompiled with netCDF-4 libraries instead of the usual netCDF-3 ones. I'm afraid my grasp on the specifics doesn't extend much further than that.

Thanks again for your help. I'll report back once I've tried your first suggestion.

Answer 4

[Jan]

Please use the PROFILEr to find out, which lines needs to most time. Perhaps there is a missing pre-allocation in the function nc_varget, you did not show?

[Andrew Yool]

That sounds quite sensible. How does one use profile (say on the code above)? It's a new tool to me.

[Jan]

See "help profile"... E.g. you can type this in the command line:

profile on

<start your code>

profile report

profile off

[Ashish Uthama]

http://www.mathworks.com/help/techdoc/matlab_env/f9-17018.html

Answer 5

[Kevin]

I have similar problems. This worked for me. At the end of your loop (whatever it does that accesses files and plots figures...) add the following line just before the return to the top of the loop statement (end):

close all force

This helps!