Uploading to AWS S3 bucket using a HTTPS PUT

isnore · March 9, 2020, 4:00pm

I was able to upload to the S3 bucket using a simple HTTPS PUT request and a pre-signed url. However, the pre-signed URLs expire. How should I include the AWS keys to my request, so that I don’t have to use a pre-signed url?

AWS FreeRTOS HTTPS Demos
I saw the following code: AWS FreeRTOS HTTPS Demos but the upload to S3 uses a pre-signed.

On the sync upload demo code I saw the following lines of code:

    connConfig.pClientCert = ( ( IotNetworkCredentials_t * ) pNetworkCredentialInfo )->pClientCert;
    connConfig.clientCertLen = ( ( IotNetworkCredentials_t * ) pNetworkCredentialInfo )->clientCertSize;
    connConfig.pPrivateKey = ( ( IotNetworkCredentials_t * ) pNetworkCredentialInfo )->pPrivateKey;
    connConfig.privateKeyLen = ( ( IotNetworkCredentials_t * ) pNetworkCredentialInfo )->privateKeySize;

and I was two things.

Could I simply use this demo without a pre-signed url since I would be passing these pieces of information in?
How do I create the pClientCert?

HTTPS PUT Request
I also saw the following page, S3 PUT but I was confused how to implement this in C.

I am looking for a longer term solution and I don’t want to create a pre-signed url for each item I upload. Thanks!

I am using an ESP32 and I am on the latest AmazonFreeRTOS version.

MuneebAWS · March 10, 2020, 7:54pm

Hi @isnore,

You will be able to sign your own requests to AWS using temporary credentials, so that you do not have to use a pre-signed URL. The process for obtaining temporary credentials is described here: https://docs.aws.amazon.com/iot/latest/developerguide/authorizing-direct-aws.html
and the signature process is described here: https://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html
The client cert is an X.509 certificate for your device. It can be created for you by AWS IoT Core, or you will be able to upload your own as described here: https://docs.aws.amazon.com/iot/latest/developerguide/create-device-certificate.html

The string created in the signature process above can be used in your Authorization header for your request to S3, as described by this page: https://docs.aws.amazon.com/general/latest/gr/sigv4-add-signature-to-request.html. The difference between the provided upload demo and using the temporary credential workflow is that you will have to add the necessary headers, using IotHttpsClient_AddHeader after initializing the request.

isnore · March 10, 2020, 9:13pm

Hello @MuneebAWS,

So is it a matter of choosing between option number 1 vs. 2? If I create the client cert on the AWS IoT Core then do I simply store those values on my ESP32?

If this is the case, why would one choose option number 1? It seems as though the second option seems like a simpler implementation. Thanks!

MuneebAWS · March 10, 2020, 10:01pm

Hello @isnore,

If I understand you correctly, you are asking for the difference between the temporary credential workflow and creating a device client certificate? A device certificate (along with the private key) is needed to authenticate with AWS IoT Core in order to use AWS IoT services, including the credentials provider. It will not be sufficient by itself to access S3 resources.

The AWS version 4 signature scheme allows you to sign your own requests to AWS, including to S3. This can be done using temporary credentials, obtained from the AWS IoT credentials provider. The client cert is necessary for this step.

If you create the client cert from AWS IoT Core, then you will have to add the cert and private key to the credential configuration file. Since you linked the HTTPS demos from this repository, the appropriate file would be aws_clientcredential_keys.h. A tool to generate the header for you is provided here.

Let me know if you have further questions.

isnore · March 11, 2020, 9:30pm

Yes that’s what I was asking for, thanks! But, I still have a few more questions.

So as an update, I created a certificate using AWS IoT Core and added those so a configuration file. Please note, this is for my custom project so I’m not using the HTTPS demo code.

As well, I was able to successfully upload to an S3 bucket using a python script from my computer Python AWS example. What I’m confused about is how to do the same on my ESP32.

Here is the code I implemented once I have access to a wi-fi connection:

void send_http_request(void *arg){
    ulTaskNotifyTake(pdTRUE, portMAX_DELAY);

    IotHttpsClient_Init();

    // An initialized network interface.
    IotNetworkInterface_t* pNetworkInterface = &IotNetworkAfr;

// Parameters to HTTPS Client connect.
    IotHttpsConnectionInfo_t connInfo = IOT_HTTPS_CONNECTION_INFO_INITIALIZER;
    IotHttpsConnectionHandle_t connHandle = IOT_HTTPS_CONNECTION_HANDLE_INITIALIZER;
    uint8_t* pConnUserBuffer = (uint8_t*)malloc(connectionUserBufferMinimumSize);

// Set the connection configuration information.
    connInfo.pAddress = "www.amazon.com";
    connInfo.addressLen = strlen("www.amazon.com");
    connInfo.port = 443;
    connInfo.flags = 0;
    connInfo.pAlpnProtocols = "x-amzn-http-ca";
    connInfo.pCaCert = HTTPS_TRUSTED_ROOT_CA;
    connInfo.caCertLen = sizeof( HTTPS_TRUSTED_ROOT_CA );
    connInfo.userBuffer.pBuffer = pConnUserBuffer;
    connInfo.userBuffer.bufferLen = connectionUserBufferMinimumSize;
    connInfo.pClientCert = keyCLIENT_CERTIFICATE_PEM;
    connInfo.clientCertLen = sizeof( keyCLIENT_CERTIFICATE_PEM );
    connInfo.pPrivateKey = keyCLIENT_PRIVATE_KEY_PEM;
    connInfo.privateKeyLen = sizeof( keyCLIENT_PRIVATE_KEY_PEM );
    connInfo.pNetworkInterface = pNetworkInterface;

    IotHttpsReturnCode_t returnCode = IotHttpsClient_Connect(&connHandle, &connInfo);
    if( returnCode == IOT_HTTPS_OK )
    {
        // Do something with the HTTPS connection...

        // Clean up and close the HTTPS connection once it's no longer needed.
        IotHttpsClient_Disconnect(connHandle);
    }

    for(;;){}
}

but the code crashes and outputs the following:

I (4694) WIFI_TASK: SYSTEM_EVENT_STA_GOT_IP:192.168.7.104
../amazon-freertos/freertos_kernel/queue.c:1429 (xQueueSemaphoreTake)- assert failed!
abort() was called at PC 0x40088dbf on core 0
xtensa-esp32-elf-addr2line -pfiaC -e /Users/verajparuthi/esp/Pebble_ESP32_Firmware/build/esp32_app 0x40088dbf: [Errno 2] No such file or directory

ELF file SHA256: a493ea87a4267066ee586a53f0e2d303cb9b52e1797a12988562db47c0778a34

Backtrace: 0x40087c0c:0x3ffb4b20 0x40087e39:0x3ffb4b40 0x40088dbf:0x3ffb4b60 0x4012b4d5:0x3ffb4ba0 0x40128bf1:0x3ffb4bc0 0x40127e7c:0x3ffb4bf0 0x40121ef4:0x3ffb4c20 0x401227a2:0x3ffb4e80 0x400dd612:0x3ffb4ea0

I used the following values for constants:

#define HTTPS_TRUSTED_ROOT_CA                               \
    "-----BEGIN CERTIFICATE-----\n"                                      \
    "MIIDdzCCAl+gAwIBAgIEAgAAuTANBgkqhkiG9w0BAQUFADBaMQswCQYDVQQGEwJJ\n" \
    "RTESMBAGA1UEChMJQmFsdGltb3JlMRMwEQYDVQQLEwpDeWJlclRydXN0MSIwIAYD\n" \
    "VQQDExlCYWx0aW1vcmUgQ3liZXJUcnVzdCBSb290MB4XDTAwMDUxMjE4NDYwMFoX\n" \
    "DTI1MDUxMjIzNTkwMFowWjELMAkGA1UEBhMCSUUxEjAQBgNVBAoTCUJhbHRpbW9y\n" \
    "ZTETMBEGA1UECxMKQ3liZXJUcnVzdDEiMCAGA1UEAxMZQmFsdGltb3JlIEN5YmVy\n" \
    "VHJ1c3QgUm9vdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKMEuyKr\n" \
    "mD1X6CZymrV51Cni4eiVgLGw41uOKymaZN+hXe2wCQVt2yguzmKiYv60iNoS6zjr\n" \
    "IZ3AQSsBUnuId9Mcj8e6uYi1agnnc+gRQKfRzMpijS3ljwumUNKoUMMo6vWrJYeK\n" \
    "mpYcqWe4PwzV9/lSEy/CG9VwcPCPwBLKBsua4dnKM3p31vjsufFoREJIE9LAwqSu\n" \
    "XmD+tqYF/LTdB1kC1FkYmGP1pWPgkAx9XbIGevOF6uvUA65ehD5f/xXtabz5OTZy\n" \
    "dc93Uk3zyZAsuT3lySNTPx8kmCFcB5kpvcY67Oduhjprl3RjM71oGDHweI12v/ye\n" \
    "jl0qhqdNkNwnGjkCAwEAAaNFMEMwHQYDVR0OBBYEFOWdWTCCR1jMrPoIVDaGezq1\n" \
    "BE3wMBIGA1UdEwEB/wQIMAYBAf8CAQMwDgYDVR0PAQH/BAQDAgEGMA0GCSqGSIb3\n" \
    "DQEBBQUAA4IBAQCFDF2O5G9RaEIFoN27TyclhAO992T9Ldcw46QQF+vaKSm2eT92\n" \
    "9hkTI7gQCvlYpNRhcL0EYWoSihfVCr3FvDB81ukMJY2GQE/szKN+OMY3EU/t3Wgx\n" \
    "jkzSswF07r51XgdIGn9w/xZchMB5hbgF/X++ZRGjD8ACtPhSNzkE1akxehi/oCr0\n" \
    "Epn3o0WC4zxe9Z2etciefC7IpJ5OCBRLbf1wbWsaY71k5h+3zvDyny67G7fyUIhz\n" \
    "ksLi4xaNmjICq44Y3ekQEe5+NauQrz4wlHrQMz2nZQ/1/I6eYs9HRCwBXbsdtTLS\n" \
    "R9I4LtD+gdwyah617jzV/OeBHRnDJELqYzmp\n"                             \
    "-----END CERTIFICATE-----\n"

and

/**
 * @brief An #IotNetworkInterface_t that uses the functions in this file.
 */
const IotNetworkInterface_t IotNetworkAfr =
{
    .create             = IotNetworkAfr_Create,
    .setReceiveCallback = IotNetworkAfr_SetReceiveCallback,
    .send               = IotNetworkAfr_Send,
    .receive            = IotNetworkAfr_Receive,
    .receiveUpto        = IotNetworkAfr_ReceiveUpto,
    .close              = IotNetworkAfr_Close,
    .destroy            = IotNetworkAfr_Destroy
};

which is defined in ./amazon-freertos/libraries/abstractions/platform/freertos/iot_network_freertos.c

Please advise. Thanks!

Notes

This is for ESP32 and I am on the latest AWSFreeRTOS version.

MuneebAWS · March 12, 2020, 12:03am

Hi @isnore,

You mentioned that this is a custom project, but are you able to successfully run the demo code, or your code without the HTTP connection? It’s possible the issue could lie outside of the snippet you posted.

Since this is ESP32, are you using ESP-IDF? Are you able to decode the backtrace you’re getting: https://docs.espressif.com/projects/esp-idf/en/latest/api-guides/tools/idf-monitor.html ?

isnore · March 12, 2020, 2:09pm

Yes, I’ve been able to successfully connect to an access point and even make a simple https request to an endpoint using esp_http_client. However, now I removed the esp_http_client and implemented the FreeRTOSClient code.

Yes I am using the esp-idf included in the amazonfreeRTOS git repo. Sadly, I can’t seem to backtrace the result when I monitor. I get the following error and I’m not quite sure how to fix it.

I (4684) event: sta ip: 192.168.7.104, mask: 255.255.255.0, gw: 192.168.7.1
I (4694) WIFI_TASK: Event = 0007
I (4694) WIFI_TASK: SYSTEM_EVENT_STA_GOT_IP:192.168.7.104

../amazon-freertos/freertos_kernel/queue.c:1429 (xQueueSemaphoreTake)- assert failed!
abort() was called at PC 0x40088dbf on core 0
xtensa-esp32-elf-addr2line -pfiaC -e /Users/vp/esp/Pebble_ESP32_Firmware/build/esp32_app 0x40088dbf: [Errno 2] No such file or directory

ELF file SHA256: a493ea87a4267066ee586a53f0e2d303cb9b52e1797a12988562db47c0778a34

Backtrace: 0x40087c0c:0x3ffb4b20 0x40087e39:0x3ffb4b40 0x40088dbf:0x3ffb4b60 0x4012b4d5:0x3ffb4ba0 0x40128bf1:0x3ffb4bc0 0x40127e7c:0x3ffb4bf0 0x40121ef4:0x3ffb4c20 0x401227a2:0x3ffb4e80 0x400dd612:0x3ffb4ea0
xtensa-esp32-elf-addr2line -pfiaC -e /Users/vp/esp/Pebble_ESP32_Firmware/build/esp32_app 0x40087c0c: [Errno 2] No such file or directory
xtensa-esp32-elf-addr2line -pfiaC -e /Users/vp/esp/Pebble_ESP32_Firmware/build/esp32_app 0x40087e39: [Errno 2] No such file or directory
xtensa-esp32-elf-addr2line -pfiaC -e /Users/vp/esp/Pebble_ESP32_Firmware/build/esp32_app 0x40088dbf: [Errno 2] No such file or directory
xtensa-esp32-elf-addr2line -pfiaC -e /Users/vp/esp/Pebble_ESP32_Firmware/build/esp32_app 0x4012b4d5: [Errno 2] No such file or directory
xtensa-esp32-elf-addr2line -pfiaC -e /Users/vp/esp/Pebble_ESP32_Firmware/build/esp32_app 0x40128bf1: [Errno 2] No such file or directory
xtensa-esp32-elf-addr2line -pfiaC -e /Users/vp/esp/Pebble_ESP32_Firmware/build/esp32_app 0x40127e7c: [Errno 2] No such file or directory
xtensa-esp32-elf-addr2line -pfiaC -e /Users/vp/esp/Pebble_ESP32_Firmware/build/esp32_app 0x40121ef4: [Errno 2] No such file or directory
xtensa-esp32-elf-addr2line -pfiaC -e /Users/vp/esp/Pebble_ESP32_Firmware/build/esp32_app 0x401227a2: [Errno 2] No such file or directory
xtensa-esp32-elf-addr2line -pfiaC -e /Users/vp/esp/Pebble_ESP32_Firmware/build/esp32_app 0x400dd612: [Errno 2] No such file or directory

The esp32_app file is present in the build folder and it used it to run the code that executed prior to the line where it crashed. I’ve had this issue when I monitor prior to implementing the FreeRTOS http client code - so it isn’t isolated to this situation.

If perhaps I could see some example code of connecting to a bucket without using a pre-signed url that would be great! Thanks!

MuneebAWS · March 14, 2020, 1:02am

Hi @isnore,

For issues with IDF Monitor and decoding your backtrace, members of the @Espressif team should be able to help you.

As for the HTTP client code, were you able to successfully run the pre-signed URL HTTPS demos that you linked previously? Since esp_http_client is different from FreeRTOS’ implementation of its HTTP library, I would suggest first using FreeRTOS client demos, with pre-signed URLs. Afterwards, you can implement the AWS version 4 signature algorithm in your code so that the URLs do not have to be pre-signed.

mahavirj · March 16, 2020, 6:11am

@isnore I wonder why your executable is missing out on .elf suffix. Do you have your own build system changes? Do you build your application using make or CMake ? If you try generating simple crash in default AFR application, do you still run into similar backtrace decoding issue?

isnore · March 16, 2020, 4:16pm

Yes I was able to get it working with the pre-signed url.

isnore · March 16, 2020, 4:18pm

Yes I have a custom CMake project. My main CmakeLists.txt has the following code:

cmake_minimum_required(VERSION 3.13)

project(freertos_examples)

add_executable(esp32_app src/main.c)

# Tell IDF build to link against this target.
set(IDF_PROJECT_EXECUTABLE esp32_app)

# Add some extra components. IDF_EXTRA_COMPONENT_DIRS is an variable used by ESP-IDF
# to collect extra components.
get_filename_component(EXTRA_COMPONENT_DIRS components ABSOLUTE)

# Add some external components to the project
set(IDF_EXTRA_COMPONENT_DIRS ${EXTRA_COMPONENT_DIRS})

# As of now there's no offical way to redefine config files outside of Amazon FreeRTOS source tree.
# This is a temporary approach to inject an include path so that this takes precedence over the
# config file directory inside Amazon FreeRTOS.
include_directories(BEFORE amazon-freertos-configs)

# Add amazon freertos as an subdirectory. AFR_BOARD tells which board to target.
set(AFR_BOARD espressif.esp32_devkitc CACHE INTERNAL "")
add_subdirectory(amazon-freertos)

target_link_libraries(esp32_app PRIVATE AFR::demo_mqtt)

As you can see my executable name is esp32_app. I saw your post and I changed it to esp32_app.elf but that didn’t resolve the issue.

Lukeman-Mypitboard · April 8, 2021, 2:51pm

@isnore i have done the same thing

Case 1:
Created Presigned Url using python and successfully uploaded the file to S3 bucket.

Case 2:
From the ESP2 side stored the Aws IOT core root CA for MQTT authentication.
How do i create the root ca for https access to S3 bucket?
Need some help on setting up the Aws side!!!

MuneebAWS · April 9, 2021, 8:36pm

Hi @Lukeman-Mypitboard,

If you want to make HTTP requests to S3 without using a presigned URL, you can sign your requests using the AWS version 4 signature scheme. You can refer to this post above describing the process for signing requests with temporary credentials.

The Amazon Root CA certs for RSA and ECC based validation aren’t specifically for MQTT, but can be used for other IoT services. You would use the same cert for server authentication when connecting to the credentials endpoint.

Lukeman-Mypitboard · April 11, 2021, 3:15pm

Thank you for the response.

I understand the AWS version 4 signature scheme. But i am using presigned url so this method wont be suitable for me as you mentioned.

In the below mentioned file in amazon freertos SDK

github.com

aws/amazon-freertos/blob/master/vendors/espressif/boards/esp32/aws_demos/config_files/http_demo_s3_upload_config.h

/*
 * FreeRTOS V202011.00
 * Copyright (C) 2020 Amazon.com, Inc. or its affiliates.  All Rights Reserved.
 *
 * Permission is hereby granted, free of charge, to any person obtaining a copy of
 * this software and associated documentation files (the "Software"), to deal in
 * the Software without restriction, including without limitation the rights to
 * use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
 * the Software, and to permit persons to whom the Software is furnished to do so,
 * subject to the following conditions:
 *
 * The above copyright notice and this permission notice shall be included in all
 * copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
 * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
 * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
 * IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
 * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

This file has been truncated. show original

They have mentioned root CA for TLS authenticatin with S3. Is it the same root CA which is generated from AWS IOT core or a different one which i need to generate specifically for S3 service. If so how to generate root CA for S3 service?

The comments from the SDK is mentioned below for reference

/*

@brief Server’s root CA certificate for TLS authentication with S3.
The Baltimore Cybertrust Root CA Certificate is defined below.
@note This certificate should be PEM-encoded.
Must include the PEM header and footer:
“-----BEGIN CERTIFICATE-----\n”\
“…base64 data…\n”\
“-----END CERTIFICATE-----\n”

*/

Thank you for future responses

MuneebAWS · April 15, 2021, 1:32am

Hi @Lukeman-Mypitboard,

Are you unable to use the root CA that you have linked? That cert for S3 is a different one from the cert for AWS IoT I had linked previously. You do not generate your own root CA certs to connect to AWS, you should use Amazon’s.

PaulB-AWS · September 14, 2022, 7:06pm

2 posts were merged into an existing topic: S3 upload without presigned URL’s