OTA not sequential

Libraries
1

Hi,
Can anybody explain why the delivery of file blocks is not sequential?

[INFO] [OtaConfig] [validateDataBlock:2535] Received valid file block: Block index=283, Size=2048
[INFO] [OtaConfig] [ingestDataBlockCleanup:2771] Number of blocks remaining: 7
[INFO] [OtaConfig] [executeHandler:2939] Current State=[WaitingForFileBlock], Event=[ReceivedFileBlock], New state=[WaitingForFileBlock]
[INFO] [OtaTask] [MqttDataCallback:738] Received data message callback, size 2074.

[INFO] [OtaConfig] [validateDataBlock:2535] Received valid file block: Block index=282, Size=2048
[INFO] [OtaConfig] [ingestDataBlockCleanup:2771] Number of blocks remaining: 6
[INFO] [OtaConfig] [executeHandler:2939] Current State=[WaitingForFileBlock], Event=[ReceivedFileBlock], New state=[WaitingForFileBlock]
[INFO] [MqttAgent] [MqttAgentTask_Publish:965] Sent PUBLISH packet to broker $aws/things/XXXXXX/streams/XXXXXXXXXXXXXXXXXXXXXXXXXX/get/cbor.

[INFO] [OtaConfig] [requestFileBlock_Mqtt:1166] Published to MQTT topic to request the next block: topic=$aws/things/XXXXXX/streams/XXXXXXXXXXXXXXXXXXXXXXXXXX/get/cbor.
[INFO] [OtaConfig] [executeHandler:2939] Current State=[WaitingForFileBlock], Event=[RequestFileBlock], New state=[WaitingForFileBlock]
[INFO] [OtaTask] [MqttDataCallback:738] Received data message callback, size 2074.

[INFO] [OtaConfig] [validateDataBlock:2535] Received valid file block: Block index=284, Size=2048
[INFO] [OtaConfig] [ingestDataBlockCleanup:2771] Number of blocks remaining: 5
[INFO] [OtaConfig] [executeHandler:2939] Current State=[WaitingForFileBlock], Event=[ReceivedFileBlock], New state=[WaitingForFileBlock]
[INFO] [OtaTask] [MqttDataCallback:738] Received data message callback, size 2074.

[INFO] [OtaConfig] [validateDataBlock:2535] Received valid file block: Block index=285, Size=2048
[INFO] [OtaConfig] [ingestDataBlockCleanup:2771] Number of blocks remaining: 4
[INFO] [OtaConfig] [executeHandler:2939] Current State=[WaitingForFileBlock], Event=[ReceivedFileBlock], New state=[WaitingForFileBlock]
[INFO] [MqttAgent] [MqttAgentTask_Publish:965] Sent PUBLISH packet to broker $aws/things/XXXXXX/streams/XXXXXXXXXXXXXXXXXXXXXXXXXX/get/cbor.

[INFO] [OtaConfig] [requestFileBlock_Mqtt:1166] Published to MQTT topic to request the next block: topic=$aws/things/XXXXXX/streams/XXXXXXXXXXXXXXXXXXXXXXXXXX/get/cbor.
[INFO] [OtaConfig] [executeHandler:2939] Current State=[WaitingForFileBlock], Event=[RequestFileBlock], New state=[WaitingForFileBlock]
[INFO] [OtaTask] [MqttDataCallback:738] Received data message callback, size 2074.

[INFO] [OtaConfig] [validateDataBlock:2535] Received valid file block: Block index=287, Size=2048
[INFO] [OtaConfig] [ingestDataBlockCleanup:2771] Number of blocks remaining: 3
[INFO] [OtaConfig] [executeHandler:2939] Current State=[WaitingForFileBlock], Event=[ReceivedFileBlock], New state=[WaitingForFileBlock]
[INFO] [OtaTask] [MqttDataCallback:738] Received data message callback, size 2074.

[INFO] [OtaConfig] [validateDataBlock:2535] Received valid file block: Block index=286, Size=2048
[INFO] [OtaConfig] [ingestDataBlockCleanup:2771] Number of blocks remaining: 2
[INFO] [OtaConfig] [executeHandler:2939] Current State=[WaitingForFileBlock], Event=[ReceivedFileBlock], New state=[WaitingForFileBlock]
[INFO] [OtaTask] [OtaThread:342] Received: 288 Queued: 288 Processed: 288 Dropped: 0
[INFO] [MqttAgent] [MqttAgentTask_Publish:965] Sent PUBLISH packet to broker $aws/things/XXXXXX/streams/XXXXXXXXXXXXXXXXXXXXXXXXXX/get/cbor.

[INFO] [OtaConfig] [requestFileBlock_Mqtt:1166] Published to MQTT topic to request the next block: topic=$aws/things/XXXXXX/streams/XXXXXXXXXXXXXXXXXXXXXXXXXX/get/cbor
[INFO] [OtaConfig] [executeHandler:2939] Current State=[WaitingForFileBlock], Event=[RequestFileBlock], New state=[WaitingForFileBlock]
[INFO] [OtaTask] [MqttDataCallback:738] Received data message callback, size 1750.

[INFO] [OtaConfig] [validateDataBlock:2535] Received valid file block: Block index=289, Size=1724
[INFO] [OtaConfig] [ingestDataBlockCleanup:2771] Number of blocks remaining: 1
[INFO] [OtaConfig] [executeHandler:2939] Current State=[WaitingForFileBlock], Event=[ReceivedFileBlock], New state=[WaitingForFileBlock]
[INFO] [OtaTask] [MqttDataCallback:738] Received data message callback, size 2074.

2

While I don’t know if this is the cause specifically here, it is common in network communication, particularly for wide area networks, to stream multiple packets “at once” rather that doing it sequentially with handshaking. This reduces the effect of the latency of the network.

It does mean that the receiver needs be prepared for some number of out of order blocks, and have space to buffer them, and negotiate with the sender that number.

Even if the sender sends the packets in order, due to variations in network routing, or packet loss and retransmission, it is quite possible for packets that started later to pass packets sent earlier.

3

Files are transferred in blocks through the network, and there’s no guarantee that they will arrive in sequence. Can you check if the OTA library is capable of creating the file without any issues once all the blocks have been received?

4

If you are using the OTA agent then I believe it tracks received blocks in a bitmap, then requests any that are missing.