coreMQTT: Handling extremely large payloads

Libraries
1

Hi,
I want to receive files over mqtt. These files are larger than what I can store in RAM. Could this be possible with coreMQTT? It seems to me that the maximum payload size is now limited to the size of the networkbuffer provided with MQTT_init, which would make this impossible.

I have done this before with another MQTT library. This library invoked the callback with each incoming payload chunk, so I could write the data to SD while it was coming in.

I would be nice if coreMQTT had something similar.

Regards,
Peter

2

Are you talking about this buffer? If yes, it is supplied by the application in MQTT_Init. You should be able to supply a larger buffer suitable for your use case.

3

Yes, but I cannot make that buffer so large that I can receive the entire payload. I can at best make this buffer 100kb or so, but I want to receive 3Mb.

4

Can you receive the payload in chunks? Our OTA works that way.

5

You mean publish the file in sizable chunks, and let the subscriber reassemble everything? Yes I can do that. But I was hoping I didn’t have to.

6

well, if you do not have enough RAM to receive your data in a single chunk, you have to chop it up, there is no magic that can create memory out of nothing?..

7

It is possible if coreMQTT would feed the data to the application as it comes in.
I have used another library (wolfMQTT) that did this.

8

The size of the payload that can be received by coreMQTT depends on the network buffer size. Currently, the function you mentioned is not supported. We will bring back this requirement for discussion to see if it’s possible for us to support this function in the future.