1 MB file upload to s3 bucket is taking almost 3 sec

Partners & Sponsors AWS
1

1 MB file upload to s3 bucket is taking almost 3 sec. Does it have any reference on this. Also seek help whether I am missing something. Also I would like to know the achievable speed for this.

2

Which SDK are you using?

You need to elaborate this. What do you mean by “missing something”?

3

I am pushing the data from the esp32 to aws s3 by using “HTTP_METHOD_PUT” method.
part of a code I have used
esp_http_client_config_t config = {
.url = url,
.method = HTTP_METHOD_PUT,
.buffer_size_tx =20000,
.buffer_size = 20000,
.keep_alive_enable = true,
.disable_auto_redirect = true,
.event_handler = _http_event_handler,
};

while ((read_bytes = fread(buffer, 1, sizeof(buffer), file)) > 0) {
int write_result = esp_http_client_write(client, buffer, read_bytes);
if (write_result <= 0) {
ESP_LOGE(TAG, “Failed to write data to HTTP client. HTTP status code: %d”, esp_http_client_get_status_code(client));
fclose(file);
esp_http_client_cleanup(client);
vTaskDelete(NULL);
return;
}

4

Here are couple of things you can try -

  1. Ensure that your bucket is in the AWS region closest to you.
  2. Use S3 multipart upload to upload your files in multiple parts parallelly.
  3. Try enabling S3 Transfer Acceleration for your bucket.
5

Thanks for the reply. Let me test it …

6

Another way to possibly save some time here is to send more bytes per HTTP post. In fact you should be able to upload the entire file in one part if you wanted to (AWS S3 quotas and limits). This may not be possible on your memory constrained MCU.

7

The problem lies in the upload process where it takes 33 seconds to write 0.2 MB using a buffer size of 20k in the following code.

while ((read_bytes = fread(buffer, 1, sizeof(buffer), file)) > 0) {

    int write_result = esp_http_client_write(client, buffer, read_bytes);
 
 
if (write_result <= 0) {
        ESP_LOGE(TAG, "Failed to write data to HTTP client. HTTP status code: %d", esp_http_client_get_status_code(client));
        fclose(file);
        esp_http_client_cleanup(client);
        vTaskDelete(NULL);
        return;
    }

   
}