Is mat.h thread-safe?

So here's the problem--I have a directory containing about 10,000 matlab files, and I need to read them all into matlab. The thing is, this directory is mounted over a network. These are small files, and the overall network nandwidth is pretty good. However, the latency is terrible--on the order of 300ms per file. I've tested it, and asynchronous file reading gets me a speed-up of 10x or more in reading files like these. The thing is, matlab dosn't support asynchronous file reads out of the box, for reasons we can only imagine.

So, I'm going to try writing it myself in mex. The matlab docs say that you can get threading to work in a mex file, but that you can't interact with matlab, since the mex interface isn't thread-safe:

http://www.mathworks.com/support/solutions/en/data/1-V3B5T/

However, the interface for file reading in c doesn't actually use mex.h, it uses mat.h:

http://www.mathworks.com/help/matlab/apiref/matgetvariable.html

Furthermore, the examples they provide don't actually show any interaction with matlab--instead they load a .mat file inside an ordinary C main() function. This suggests that I can spawn multiple threads and perform multiple asynchronous loads via the matGetVariable interface, and then collect them back into the main mex workspace at the end. However, I'm not sure whether these calls to matGetVariable are going to step on eachother. (I'm also not terribly familiar with pthreads; I know they CAN do stuff like this, but suggestions on how would be appreciated)

To save everyone time, I've already thought about copying the files to the local filesystem (e.g. via the unix() command) as well as using the matlab parallel toolbox. Neither of these are viable options for various reasons.

[Walter Roberson]

Would it perhaps be viable to do something like

rsh SERVER -c '(produce a batch of file names) | cpio -oac | gzip -9 | ttcp -s -t CLIENT'

and on client,

ttcp -r -t | gunzip | cpio -idm

That is, bundle up a group of files into an archive, compress the archive, send it fast over the net, have the receiver unzip and unbundle

cpio (or even tar) are amazingly faster at creating bunches of files than ftp or rcp. I do not recall that I ever did a comparison test to scp -r though.

[leptogenesis]

Copying the files to the local filesystem won't work. Basically the issue is that there are actually 100 separate processes, and each of these files has about 1000 variables in them. Each process needs to read only 10 variables in each file. Hence, copying the files themselves would involve much more data transfer than is really necessary.

It's possible that I could open up multiple matlab sessions just for the sake of reading the variables and saving them locally. but then there's the overhead of extra matlab processes...plus the issue of interprocess communication in matlab. Ironically, though, that interprocess communication could be done asynchronously :-)

[Jan]

I do not dare to post this as an answer, but it is at least an obvious and successful strategy: Do not store small data in 10'000 file, when you access them over a network with a high latency. Better use one or a few large MAT files instead.

[leptogenesis]

I wish I could do that, but these files are being written in parallel by hundreds of machines. My setup is actually a lot like hadoop/mapreduce: I have thousands of objects of type X, thousands of objects of type Y, all of which are distributed among hundreds of machines. Each object of type X needs to send a small message to every object of type Y. In theory this could all be done over the network directly, but it turns out that's even more of a nightmare in matlab :-/

Answer 1

[James Tursa]

The restriction about not "interacting with MATLAB", as you put it, has to do mainly with the MATLAB memory manager, which is not thread-safe. Basically, anything you do that would cause the MATLAB memory manager to allocate/deallocate memory, or to alter memory that another thread depends on, will likely cause MATLAB to not work properly or crash. The function matGetVariable causes the MATLAB memory manager to allocate memory for a new variable, so it would likely not be thread-safe. Stuff you can get away with inside a thread in a mex routine would be functions like mxGetPr, mxGetPi, mxIsDouble, etc. I.e., functions that simply interrogate MATLAB memory and return a value without allocating/deallocating memory will work. Bottom line is I would be very surprised if your scheme works.

[leptogenesis]

Ok, fine. Actually trying to compile their example convinces me that yes, using matGetVariable actually requires an entire matlab running in the background. Thanks for the help, James.

Most likely I will just use the parallel computing toolbox--that'll get me at least 12 parallel loads. But if anyone else has a better idea, I'd still like to hear it.