An efficient (quick) way of entering complex data into Matlab workspace.

A HW device stores binary data on disk as an array of 32.5e6 unsigned short values which are then read into Matlab workspace, using either fread or memmapfile. The data is actually complex in nature, arranged as I/Q/I/Q/,,, Our goal is to convert the short values into a complex array so that we may perform FFT and other Matlab operations. The problem is that the conversions to single and then to complex are each very time consuming; please see below.

    datasize = 32.5e6;
    fileID = fopen('dataFile_uint16.bin', 'r');
    temp = fread(fileID,  dataSize, 'int16=>int16');   % read file as I & Q;  memmapfile is faster...
    fclose(fileID);
    
    temp1 = single(temp);                % Takes between 100-200 ms.  
                                         % NOTE: Matlab does not allow short 
                                         % data to be converted to complex
    complexArray = temp1(1:2:dataSize) + 1i*temp1(2:2:dataSize);  % takes ~550 ms

I have tried alternatives to the last line of code above such as complex(a,b) and reshape instead of indexing, but no real-time savings is noticed. Any suggestions?

A rigourous alternative is to write a mex-file which reads, converts, and returns the data to an mxComplexArray type. However, before enbarking on such a journey, I would like to know if the savings in real-time will justify the effort.

Thank you,

Yisrael Loecher

Answer 1

[James Tursa]

Assuming you are using R2018a or later, you might try this FEX submission which can reinterpret real variables as interleaved complex variables without a deep copy:

https://www.mathworks.com/matlabcentral/fileexchange/65842-sharedchild-creates-a-shared-data-copy-of-contiguous-subset

This might save you some time in your second step, but of course won't save any time associated with that first step of converting the uint16 to single. But if the original data is stored as uint16 you are probably just going to have to live with the time it takes to convert it all to single.

[Yisrael Loecher]

James,

It seems reasonable to think that if my source data is "I,Q" interleaved in memory, this may create a natural situation for a mex routine to make it accessible to Matlab in this new release.

Assuming the source data was stored as single and using my wild imagination, could it possible to memory map the source data directly into Matlab in such a way that it would be compatible with an interleaved numerical complex array? If this is a factor, we must remember that Matlab uses column orientation as opposed to Fortran/C row orientation for storing contiguous arrays of data.

[James Tursa]

"Assuming the source data was stored as single and using my wild imagination, could it possible to memory map the source data directly into Matlab in such a way that it would be compatible with an interleaved numerical complex array?"

Easy to read it directly in a mex routine. Assuming you know the size of the data in advance, inside the mex routine just mxCreateEtc the complex variable up front, then pass the pointer returned by mxGetComplexSingles to fread, and pass the resulting mxArray back to MATLAB via plhs[0]. Only one data copy involved throughout the whole process.

"Matlab uses column orientation as opposed to Fortran/C row orientation"

Actually, Fortran uses column orientation same as MATLAB. In fact, MATLAB was originally built around the linear algebra libraries written in the Fortran column order standard.

If your data is written in row order, then you will have to transpose it after reading it in.

Side Note: One would have thought that once MATLAB went to interleaved complex data, they would have updated their fread function at the same time with 'complexdouble', 'complexsingle', etc options, but I don't see any such thing in the doc. Maybe this would be an appropriate enhancement request.

[Yisrael Loecher]

...inside the mex routine just mxCreateEtc the complex variable up front, then pass the pointer returned by mxGetComplexSingles to fread, and pass the resulting mxArray back to MATLAB via plhs[0].

I am assuming that since this is NOT a direct shared memory map but rather one copy need be performed, I need not worry about column/row array orientation differences. Also, while we at it, a cast from short to single can be performed on the fly, thereby saving disk space by keeping the source data at 16 bits.

...they would have updated their fread function at the same time with 'complexdouble', 'complexsingle', etc options, but I don't see any such thing in the doc. Maybe this would be an appropriate enhancement request.

May I add that the memmapfile.m function also should encorporate this enhancement request.

[James Tursa]

"... I need not worry about column/row array orientation differences ..."

Well, whatever order was written to the file is the order you will get when you read it back in, unless you transpose on the fly as part of the read.

"... a cast from short to single can be performed on the fly ..."

Yes, you could also read 16-bit ints from the file and convert to single on the fly as well, thus having only one copy of the data as single in memory at one time.

[James Tursa]

See also this FEX submission, which contains freadcomplex and fwritecomplex mex routines for reading and writing interleaved complex data files directly in R2018a or later without the need for extra data copies:

https://www.mathworks.com/matlabcentral/fileexchange/77530-freadcomplex-and-fwritecomplex

Answer 2

[Jan]

What about avoiding the conversion from int16 to single by doing this on the fly:

temp = fread(fileID,  dataSize, 'int16=>single');

Then maybe it is cheaper to import the real and imaginary data separately:

datasize = 32.5e6;
fileID = fopen('dataFile_uint16.bin', 'r');
realX = fread(fileID,  dataSize, 'int16=>single', 2);
fseek(fid, 'bof', 2);
imagX = fread(fileID,  dataSize, 'int16=>single', 2);
fclose(fileID);

X = realX + 1i * imagX;

Is this faster?

What about:

datasize = 32.5e6;
fileID = fopen('dataFile_uint16.bin', 'r');
tmp = fread(fileID, [2, datasize/2], 'int16=>single');
fclose(fid);
X   = tmp * [1; 1i];

[Yisrael Loecher]

Jan,

Thank you for your reply.

I remember seeing that Mathworks suggests NOT to perform casting during the call to fread for time efficiency, but I don't remember where. I myself tried this and found that they are correct.

Leaving that aside, I nonetheless tried your code and came up with a very slight increase in time. However, when I cast tmp from short to single NOT in fread but separately as I did originally, and then performed the following conversion to complex as you suggest:

X   = tmp * [1; 1i];

a reduction of over 120 ms is noticed. Thats a savings of ~17%. Bravo! Keep 'im coming.

[Yisrael Loecher]

Jan,

Sorry-I didn't relate to your first suggestion above. The time killer is not the fread, but rather the conversion to single, and worst off, the conversion to complex. Therefore, splitting up the fread shouldn't really help. Your last comment is best.

P.S.

I altered the code according to your last suggestion and checked further with a few other permutations of where to place the cast from short to single. This is what I came up with, which is close to a 50% savings in real-time:

memMapPtr2 = memmapfile('dataFile_uint16.bin', 'Format', 'int16', 'Offset', 0);
temp = memMapPtr2.data;
temp = reshape(single(temp),2,[]);
temp4 = temp.' * [1; 1i];        % need to transpose here, since fread and 
                                 % memmap both output column-based arrays.

Thanks again.

[Jan]

"Interleaved numerical arrays" sounds like the optimal solution, if you import the data in a Mex function.

[Yisrael Loecher]

Jan,

I am only now aware of the change that went on under the Matlab hood in R2018a, concerning how complex data (I,Q) is stored in memory. James (below) seems to be an expert on what was before the change, and what is now after the change. I'll be looking into it...