I am using esp32 with amazon freertos for sending data to aws iot
I have a 500kb jpeg file in the flash which i want to upload it to aws S3 using http.
I cannot allocate a larger buffer of size 500kb in ram to send the jpeg file in single http request.
Is there any methodology to send this jpeg image in blocks using https directly to S3?
If above is possible, then i will allocate a smaller buffer say 10kb and read only 10kb block from flash in to ram buffer and send using IotHttpsClient_SendSync()/async().
What is the approach which i should take to upload these kind of larger files in to S3?
Transfer encoding chunked uploads are not supported in the IoT HTTPS Client library because IotHttpsClient_WriteRequestBody() tries to write a Content-Length header before sending the body. I apologize for the inconvenience this is causing. It is also difficult to do this with the current library given the hiding of the transport write implementation in the fully managed library.
Thanks for the response,
How to integrate this library to my current AWS freertos project?
Is there any sample AWS freertos project available with integrated coreHttp library?
If possible share any documentation or example code available to speed up my integration with the newlibrary.
Thanks for the valuable suggestion @SarenaAWS,
I have gone through the Multipart upload documentation, and i found that minimum part size should be 5 Mb, refer below
Currently i am using presigned url for sending http request which is a manual process, i understand we can automate with the help of authorization header but it requires a signature version 4 signature as one parameter.
2.How to generate this signature on the esp32 device itself? is there any sample code/documentation available?
If you are OK with writing a Lambda you could do this:
Send smaller chunks from ESP32 to a temp location, using a part naming scheme, e.g. instead of writing to “images/myimg.jpg” you write to “tmp_images/myimg.jpg.1_5.part” (1 is the part, 5 is the total parts expected)
Set up an S3 trigger to Lambda when writing anything matching “*.part” (suffix), which would check if all parts are uploaded yet or not, and if they are combine them and write “images/myimg.jpg” (and remove the old parts)
The reason to add “.part” is so your S3 trigger can filter to only react to those specific writes, and not all writes of the S3 bucket.
Thanks for the new approach @joel.byren
I will try this but we need a presigned url for each part we are sending, currently i am manually generating this url using presigned_urls_gen.py script.
If I have to send the images as multiple .parts then i would need to generate this presigned url for each .part separately on my esp32 device itself.
I don’t have any clue on this yet which was my second question in my previous reply
Currently i am using presigned url for sending http request which is a manual process, i understand we can automate with the help of authorization header but it requires a signature version 4 signature as one parameter.
2.How to generate this signature on the esp32 device itself? is there any sample code/documentation available?
Hi @SarenaAWS
In this library, there is a function HTTPClient_Send() for sending both header and body using sendHttpHeaders() and sendHttpBody()
Both these functions ultimately use pTransport->send() for sending data over the network. so the send function doesn’t know whether it is a http header or http body. Since only for http body i have to send the data as multiple chunks not for the http header.
Is my understanding correct?
Is there any other way the transport.send function knows about the http header or body information?
We have a similar approach, but we already have established MQTT for this, to get a secure connection. In your case, sending a generic HTTP post, it will not really work. But how are you getting the presigned URL to your target now? The presigns are short lived, so you must have a way of sending it to your esp32 device already? If you do, why not change it from a single presigned to an array of presigned, which would point to the (expected) parts. If you know already roughly how large your file will be, and you also know how much you can send in one request, you could add the appropriate number of presigns, and maybe send empty bytes for the last few just for writing them?
Can you tell me what is the approach you have followed since i don’t have any requirement that i should use only http, combination of Mqtt and http is also fine for me.
I am manually generating the presigned url using presigned_urls_gen .py and adding it to my esp32 project.
Yes i know it is short lived and generating presigned using a python script is not practical approach, but i don’t have any clue on the recommended approach as of now
Your are saying, if i have 60 parts , generate 60 presigned urls using python script and store it as a array in my esp32 project. Is my understanding correct?
Is there any other way where i can generate the signed url on my esp32 device itself
Well, if you are generating the URL from a script and then updating the code, and downloading it to your target, you really do need a new way of doing it anyway; that will only work in testing, not ever in any production environment.
We use the off-the-shelf demo for AWS MQTT for our target, which includes a step-by-step guide on setting up the AWS part (under AWS IoT Core), generating the needed certificates, and establishing connection from your target to the AWS cloud. With this, the code to publish an MQTT message is very basic.
And as for the presigned URLs, I assumed you were generating them when needed and sending them in some way to your target, not using a script on your machine and changing the build project. So no, that is not what I recommend.
Well, you need to generate them from within your cloud from some compute service (Lambda, EC2…) and send it to your target (somehow). You really can’t rebuild the whole thing every time you send something, right?
There is not currently an example of this in our codebase. One approach that may fit your application is to use a combination of the AWS IoT Rules and the AWS Lambda service.
Thanks for the reply, i will try to experiment with this approach
I have came across an example in demo folder for uploading file to S3 using https, and i ran that demo it is working fine.
With this demo i could able to upload a new file in S3, Is it possible to append the data to a existing file in S3?
If the above question is possible, then i will simply append the jpeg image blocks to the existing file in S3 and my problem will be solved
Thanks for the questions and suggestions in this post. From what it seems, there were suggestions provided for all the questions asked. Please let us know if there are anymore questions that needs help.
