How to upload data that does not fit in memoy to AWS S3 using HTTP request

I need to upload large data files (around 200MB) from my STM32L485 board to AWS S3 storage. I was planning to use direct S3 upload via an HTTP request as there is an example available . However, my data cannot be loaded in RAM (only 128KB) so I was thinking of sending it in chunks. But, I have seen that the HTTP library used in the examples, coreHTTP from aws c sdk, does not support streaming. Because of this, I’m not sure that uploading my data via HTTP is the best way to go. Is there a way to this using HTTP direct upload ?

I am new to these kinds of challenges so any advice on how to solve this problem will be much appreciated !

Thank in advance !

Hi! You can send data in chunks to S3 using multipart uploads. Documentation for multipart uploads is available at: Uploading and copying objects using multipart upload - Amazon Simple Storage Service. You are able to use S3 APIs over either HTTP or MQTT.

Thanks for you answer @archigup the problem is that the minimum chunk size is 5MB. However, I have come accross this post Building microcontroller-based IoT applications using HTTPS client in Amazon FreeRTOS | The Internet of Things on AWS – Official Blog. It this mention “You can transfer files in pieces and specify the size of each payload piece”. However the examples that I could find that uses this mechanism are always for download and never upload. I’m not sure to construct my PUT requests to be accepted by aws and be seen as one single file.

Hmm, you might be able to use HTTP chunked requests. There is documentation here for using HTTP chunking with SigV4 authentication: Signature Calculations for the Authorization Header: Transferring Payload in Multiple Chunks (Chunked Upload) (AWS Signature Version 4) - Amazon Simple Storage Service.

We have examples for coreHTTP here, but they unfortunately only have upload in one piece.

Thanks @archigup. Would that process mean that for each chunk of data I need to do one request?
For example, if I have 1000 chunks to send will I need to do 1000 HTTP requests ?

That is correct.
If the overhead of HTTP requests is a concern, you could also consider MQTT. If the constraints of the existing solutions don’t work for you, you could upload chunks to separate files and have a Lambda combine them.