How to receive an full binary data using mqtt callback function

I have sent a binary blob (for example 4k) from C++ app by mqtt mosquitto. In matlab only a chunk of sending data is received. This chunk is limited by the first zero byte which is in the blob. How to receive a full message?

Short code snippest below:

mqttClient = mqttclient("tcp://127.0.0.1");
mySub = subscribe(mqttClient, "topic", Callback=@MsgCallvBack)
%
function MsgCallvBack(topic, data)
 fprintf("topic=%s, data size=%u\n", topic, length(data));
end

Result for 4 sending attempts of random 4k blob:

>topic=testtopic, data size=40

>topic=testtopic, data size=22

>topic=testtopic, data size=3

>topic=testtopic, data size=241

[dpb]

OK, I didn't fully grasp how the callback functioned before -- I'll agree the implementation appears to not support anything but the text string. I presume at this point you are in control of the publishing side while testing at least? If so, I'd just do one last "prove it to myself" exercise in formatting the array as a string and then sending it and ensuring it comes over. Just a simple num2str(0:N) should be enough with a random N.

Presuming that does work as expected, it appears to be an implementation limitation as you've thought and the only recourse would be to request an enhancement from Mathworks or report it as a bug (although I'm pretty sure in this case they would use the "WAD" (Working As Designed) fallback.

I did find a <link to a Simulink MQTT block> that does support numeric arrays, but that requires Simulink, not just MATLAB.

I don't know of another workaround other than by using another protocol instead that does support binary data in MATLAB or try to link to Python or find C opensource and create a mex file of your own.

[Gavrila Pi]

> I don't know of another workaround other than by using another protocol instead that does support binary data in MATLAB or try to link to Python or find C opensource and create a mex file of your own.

MATLAB cannot be proud of a large number of interaction protocols. Writing your own MEX will require significant time investment.

Thank you for your help.

[dpb]

They (new protocols) seem to spring up like weeds; trying to support them all is a Herculean task it would seem and requires resources that must divert away from core development...

Answer 1

[dpb]

mqttclient and subsequent subscribe shows the callback function is to read the subscribed-to channel and the input function is an overloaded read method. Never heard of it before so no experience, but it appears it is a text-based messaging system so null bytes will be interpreted as character terminators unless treated specifically as text (like embedding as char(0) and the like for construction), but are highly likely to be considered terminators on reading internally.

From a quick look, it wouldn't seem to be a suitable transmission choice for arbitrary binary data...

[Walter Roberson]

https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901006

A UTF-8 Encoded String MUST NOT include an encoding of the null character U+0000. [MQTT-1.5.4-2]. If a receiver (Server or Client) receives an MQTT Control Packet containing U+0000 it is a Malformed Packet. Refer to section 4.13 for information about handling errors.

[Gavrila Pi]

Popular eclipse mosquitto, python paho mqtt, mqttnet allows to transmit and receive blob data without distortion. Only Matlab makes exchange impossible.

[dpb]

Well, I don't know that it can't return data; your callback function doesn't do anything that would return anything anyway; as noted above the overloaded read and peek functions would do whatever is able to be done; it doesn't appear there's anything that defines the data packet from the calling side to tell it what to expect...

I made a presumption from the examples it was text-based; it looks like that isn't necessarily the case from the spec Walter posted, but the sender would have to encode the packet correctly. Can't expect MATLAB to handle something that is outside the standard even if others might.

If there is something it doesn't do that is within the specifications, then that would be reason for a bug report but I don't know that that has yet been conclusively demonstrated to be the case.

[Gavrila Pi]

The problem is that MATLAB forces you to receive only text strings. Why? The sender transmits an array of a fixed length. Consequently, MATLAB should get the same message of exactly the same length, not truncate it at the first zero value found in it.

[dpb]

I don't see that the documentation says that, but as pointed out, if that is what the above read statement actually does, then it is worthy of a bug/support request.

Now, I do admit, returning a timetable object does seem like a funny choice and may well be what does, but the code you've shown us doesn't demonstrate that is all I'm saying...

[Gavrila Pi]

Do you have a plan to open a bug request?

[dpb]

Personally, no, since I have no background in the protocol. See <product support link> for how to get in touch...