Performance problems: using Msgpack library with MATLAB

msgpack is a high-performance serialisation library used in many 3rd-party products. There are native interfaces available for many languages, except for MATLAB. For MATLAB there is a interpreted solution: https://github.com/bastibe/matlab-msgpack — but for my use case (deserialising messages for an eyetyracker used for neuroscience experiments), it takes ~4ms which is not viable when our experiment display loop is 120Hz (8ms). In Python the same deserialisation takes around 0.1ms. There is a C library: https://github.com/msgpack/msgpack-c — but when I try to load it in MATLAB I get the following errors:

>> [n,w]=loadlibrary('/Users/ian/Downloads/msgpack-x86/lib/libmsgpack-c.dylib','/Users/ian/Downloads/msgpack-x86/include/msgpack.h','alias','msgpack')

Warning: No functions found in library.

> In loadlibrary

n =

1×0 empty cell array

w =

'

Failed to parse type 'union { char *__mbstate8; long long _mbstateL ; } __mbstate_t' original input 'union { char __mbstate8 [ 128 ]; long long _mbstateL ; } __mbstate_t'

Found on line 67 of input from line 81 of file /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/include/i386/_types.h

Type '__mbstate_t' was not found. Defaulting to type error.

Found on line 69 of input from line 83 of file /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/include/i386/_types.h

Type 'longdouble' was not found. Defaulting to type error.

Found on line 1870 of input from line 16 of file /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/14.0.3/include/__stddef_max_align_t.h

Type 'sockaddrPtr' was not found. Defaulting to type voidPtr.

Found on line 2209 of input from line 317 of file /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/include/sys/socket.h

Type 'sockaddrPtr' was not found. Defaulting to type voidPtr.

Found on line 2209 of input from line 317 of file /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/include/sys/socket.h

No match found for enum value expression sizeof in (( sizeof ( __int64_t )) - sizeof ( __uint8_t ) - sizeof ( sa_family_t )) expression ignored.

Found on line 2257 of input from line 470 of file /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/include/sys/socket.h

No match found for enum value expression sizeof in ( 128 - sizeof ( __uint8_t ) - sizeof ( sa_family_t ) - (( sizeof ( __int64_t )) - sizeof ( __uint8_t ) - sizeof ( sa_family_t )) - ( sizeof ( __int64_t ))) expression ignored.

Found on line 2257 of input from line 470 of file /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/include/sys/socket.h

Type 'in6_addr' was not found. Defaulting to type error.

Found on line 2438 of input from line 177 of file /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/include/netinet6/in6.h

Type 'in6_addr' was not found. Defaulting to type error.

Found on line 2451 of input from line 540 of file /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/include/netinet6/in6.h

Type 'in6_addr' was not found. Defaulting to type error.

Found on line 2459 of input from line 548 of file /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/include/netinet6/in6.h

Function pointer types are unsupported in structures fpos_t (* _Nullable _seek ) ( void *, .

Found on line 2730 of input from line 157 of file /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/include/_stdio.h

Failed to parse type 'union { _Bool boolean ; uint64_t u64 ; int64_t i64 ; double f64 ; msgpack_object_array array ; msgpack_object_map map ; msgpack_object_str str ; msgpack_object_bin bin ; msgpack_object_ext ext ; } msgpack_object_union' original input 'union { _Bool boolean ; uint64_t u64 ; int64_t i64 ; double f64 ; msgpack_object_array array ; msgpack_object_map map ; msgpack_object_str str ; msgpack_object_bin bin ; msgpack_object_ext ext ; } msgpack_object_union'

Found on line 2990 of input from line 88 of file /Users/ian/Downloads/msgpack-x86/include/msgpack/object.h

Type 'msgpack_object_union' was not found. Defaulting to type error.

Found on line 2995 of input from line 93 of file /Users/ian/Downloads/msgpack-x86/include/msgpack/object.h

'

Does anyone with C experience know if this is easy to workaround or not?

[Christopher Nadler]

I have the same performance problems. Also tried to convert the deserialized data from python to matlab over Core_py2matlab, saving directly from python with scipy.io.savemat and loading th file in matlab, trying to use msgpack-python directly in matlab without succesfully deserializing my data, but every aproach takes to long or is not possible to realize without a substancial effort.

I also tried to compile https://github.com/bastibe/matlab-msgpack into a mex-function, but the code is not ready for this (needs to be fully readapted).

So, If someone has a performat msgparser for matlab, I would be more than happy (and Ian maybe also) =)

Answer 1

[Christopher Nadler]

I found the bottleneck in https://github.com/bastibe/matlab-msgpack. I'm attemping to push a pull-request in the repo.

In my special case I could reduce the mean parse time from ~40ms to ~6ms by making additional adaptations to the code I posted. I am receiving data every ~4ms, so I am still not in real-time, but at least near enought.