Upload a file from ESP32 SPIFFS to Amazon S3

Hi!

I’m programming an Espressif microcontroller, the esp32-wroom-32, and I’ve followed the getting started guide (https://docs.aws.amazon.com/it_it/freertos/latest/userguide/getting_started_espressif.html) and downloaded the amazon-freertos directory with all the demos and specific vendors folders.

I’m now trying to use the “iot_demo_https_s3_upload_sync.c” to upload a file from the SPIFFS File System on ESP32 flash, to Amazon S3.
I’m using the wrapper “esp_vfs.h” to use SPIFFS. So I’m able to operate on the files and folders contained in the SPIFFS partition on flash using the C standard functions fopen, fread, fwrite…

At first, I succesfully tried to fopen the file, fread (from the stream represented by the FILE *) on a “uint8_t buffer[file size]” and after fclosed the file, to pass the uint8_t pointer of the buffer to “reqSyncInfo.pBody” at this line of the demo:
iot_demo_https_s3_upload_sync.c#L297
and to pass the file size to “reqSyncInfo.bodyLen” at this line of the demo:
iot_demo_https_s3_upload_sync.c#L298

But the problem comes when the file size is larger than the stack size: the demo crashes due to the excess of the stack size by the buffer allocated. I can try to increase the stack size, but the RAM size is limited, I would like to take advantage of all 2 MB of SPIFFS and to upload on Amazon S3 log files of my application which can reach hundreds of KB, or almost 1 MB.

I read from the HTTPS Client API Reference (https://www.freertos.org/Documentation/api-ref/https/structIotHttpsSyncInfo__t.html#add7230f049c1f80e1a87bee9e5750853) that the data field " pBody" “for a request this is the file or data we want to send. The data is separated from the headers for the flexibility to point to an already established file elsewhere in memory”.

And my question is: how can I retrieve the uint8_t pointer to an already established file elsewhere in memory? What exactly does it means?
I saw that the FILE struct for my system is the _sFILE Struct implementation for Linux, so here “https://lattice.github.io/quda-ref/v0.9.0/doc/struct____s_f_i_l_e.html” I found which are the fields of the structure and I tried, without success, to retrieve here the uint8_t pointer to assign to the “pBody” data field of “reqSyncInfo”.
My hope was that in the FILE structure there was a pointer to a buffer that it contained the file contents.

Thanks in advance for your attention,
Regards

Hello Luca,

At first, I succesfully tried to fopen the file, fread (from the stream represented by the FILE *) on a “uint8_t buffer[file size]” and after fclosed the file, to pass the uint8_t pointer of the buffer to “reqSyncInfo.pBody” at this line of the demo:
iot_demo_https_s3_upload_sync.c#L297
and to pass the file size to “reqSyncInfo.bodyLen” at this line of the demo:
iot_demo_https_s3_upload_sync.c#L298

But the problem comes when the file size is larger than the stack size: the demo crashes due to the excess of the stack size by the buffer allocated. I can try to increase the stack size, but the RAM size is limited, I would like to take advantage of all 2 MB of SPIFFS and to upload on Amazon S3 log files of my application which can reach hundreds of KB, or almost 1 MB.

For the above comments, it sounds like you are saving the file in a buffer allocated on the stack in the local function space. Could you declare the buffer in the static global space and try again?

And my question is: how can I retrieve the uint8_t pointer to an already established file elsewhere in memory? What exactly does it means?

This is referring to a file that may be stored for example at an address in the memory mapped flash space. This file must be stored in a contiguous memory location in order for the library to send it.

Hello @SarenaAWS,

Thanks for the reply!

For the above comments, it sounds like you are saving the file in a buffer allocated on the stack in the local function space. Could you declare the buffer in the static global space and try again?

Yes, I was saving the file in a buffer allocated on the stack in the local function space.
Yes, I can declare the buffer in the static global space and thank you for the advice. But the RAM size is still limited, I would like to be able to send larger files.

This is referring to a file that may be stored for example at an address in the memory mapped flash space. This file must be stored in a contiguous memory location in order for the library to send it.

Ok, so if I memory map the flash spiffs partition where my file is stored in a contiguous memory location, then I just need to pass the file name to pBody?
It might be a good solution dedicate a spiffs partition only for the file to be sure that it is saved contiguously?

Thanks for your attention,
Regards

From HTTP library’s perspective, it can send any data stored in a contiguous memory location. Then you need to pass the location of that memory (i.e. pointer) to pBody.

I am not very familiar with the SPIFFS library and related partitions, so it will probably be better to ask on an Espressif forums.

Thanks.

1 Like