Issue with certificates when performing mbedtls handshake with AWS

Hello,

We want our device (STM32-F446RE running FreeRTOS + SIM7000G modem) to communicate to the AWS cloud. We are trying to follow the Cellular Interface Library Demo, in particular following the diagram

We are using coreMQTT, MbedTLS 2.16 libraries, the amazon communication interface implementation for UART, and amazon UART API implementation (I was unable to add the links).
UART using 115200 baud rate.

Currently failing on mbedtls_ssl_handshake.
At first we got the error- allocation of memory failed, so we changed the value of the macro MBEDTLS_SSL_OUT_CONTENT_LEN, which determines the size of the outgoing TLS IO buffer, from 16384 to 8196.

Now we get the error- X509 - Certificate verification failed, e.g. CRL, CA or signature check failed.
We use the following config file, with our certificates inserted in the form described in the file.

/*
 * FreeRTOS V202112.00
 * Copyright (C) 2020 Amazon.com, Inc. or its affiliates.  All Rights Reserved.
 *
 * Permission is hereby granted, free of charge, to any person obtaining a copy of
 * this software and associated documentation files (the "Software"), to deal in
 * the Software without restriction, including without limitation the rights to
 * use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
 * the Software, and to permit persons to whom the Software is furnished to do so,
 * subject to the following conditions:
 *
 * The above copyright notice and this permission notice shall be included in all
 * copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
 * FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
 * COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
 * IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
 * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 *
 * https://www.FreeRTOS.org
 * https://github.com/FreeRTOS
 *
 */

#ifndef DEMO_CONFIG_H
#define DEMO_CONFIG_H

/* FreeRTOS config include. */
#include "FreeRTOSConfig.h"

/**************************************************/
/******* DO NOT CHANGE the following order ********/
/**************************************************/

/* Include logging header files and define logging macros in the following order:
 * 1. Include the header file "logging_levels.h".
 * 2. Define the LIBRARY_LOG_NAME and LIBRARY_LOG_LEVEL macros depending on
 * the logging configuration for DEMO.
 * 3. Include the header file "logging_stack.h", if logging is enabled for DEMO.
 */

#include "logging_levels.h"

/* Logging configuration for the Demo. */
#ifndef LIBRARY_LOG_NAME
    #define LIBRARY_LOG_NAME    "Cellular"
#endif

#ifndef LIBRARY_LOG_LEVEL
    #define LIBRARY_LOG_LEVEL    LOG_INFO
#endif

/* Prototype for the function used to print to console on Windows simulator
 * of FreeRTOS.
 * The function prints to the console before the network is connected;
 * then a UDP port after the network has connected. */
extern void vLoggingPrintf( const char * pcFormatString,
                            ... );

/* Map the SdkLog macro to the logging function to enable logging
 * on Windows simulator. */
#ifndef SdkLog
    #define SdkLog( message )    printf message
#endif

#include "logging_stack.h"

/************ End of logging configuration ****************/

#define democonfigUSE_TLS  1

#define mqttexampleTRANSPORT_SEND_RECV_TIMEOUT_MS         ( 500U )



/**
 * @brief The MQTT client identifier used in this example.  Each client identifier
 * must be unique; so edit as required to ensure that no two clients connecting to
 * the same broker use the same client identifier.
 *
 *!!! Please note a #defined constant is used for convenience of demonstration
 *!!! only.  Production devices can use something unique to the device that can
 *!!! be read by software, such as a production serial number, instead of a
 *!!! hard coded constant.
 *
 * #define democonfigCLIENT_IDENTIFIER				"insert here."
 */

/**
 * @brief Endpoint of the MQTT broker to connect to.
 *
 * This demo application can be run with any MQTT broker, that supports mutual
 * authentication.
 *
 * For AWS IoT MQTT broker, this is the Thing's REST API Endpoint.
 *
 * @note Your AWS IoT Core endpoint can be found in the AWS IoT console under
 * Settings/Custom Endpoint, or using the describe-endpoint REST API (with
 * AWS CLI command line tool).
 *
 * @note If you would like to setup an MQTT broker for running this demo,
 * please see `mqtt_broker_setup.txt`.
 *
 */
 #define democonfigMQTT_BROKER_ENDPOINT    "....."

/**
 * @brief The port to use for the demo.
 *
 * In general, port 8883 is for secured MQTT connections.
 *
 * @note Port 443 requires use of the ALPN TLS extension with the ALPN protocol
 * name. Using ALPN with this demo would require additional changes, including
 * setting the `pAlpnProtos` member of the `NetworkCredentials_t` struct before
 * forming the TLS connection. When using port 8883, ALPN is not required.
 *
 */
#define democonfigMQTT_BROKER_PORT    ( 8883 )

/**
 * @brief Server's root CA certificate.
 *
 * For AWS IoT MQTT broker, this certificate is used to identify the AWS IoT
 * server and is publicly available. Refer to the AWS documentation available
 * in the link below.
 * https://docs.aws.amazon.com/iot/latest/developerguide/server-authentication.html#server-authentication-certs
 *
 * @note This certificate should be PEM-encoded.
 *
 * Must include the PEM header and footer:
 * "-----BEGIN CERTIFICATE-----\n"\
 * "...base64 data...\n"\
 * "-----END CERTIFICATE-----\n"
 *
 */
#define democonfigROOT_CA_PEM "-----BEGIN CERTIFICATE-----\n"\
							  "...data...\n"\
                              "...data...\n"\                   
							  "-----END CERTIFICATE-----\n"

/**
 * @brief Client certificate.
 *
 * For AWS IoT MQTT broker, refer to the AWS documentation below for details
 * regarding client authentication.
 * https://docs.aws.amazon.com/iot/latest/developerguide/client-authentication.html
 *
 * @note This certificate should be PEM-encoded.
 *
 * Must include the PEM header and footer:
 * "-----BEGIN CERTIFICATE-----\n"\
 * "...base64 data...\n"\
 * "-----END CERTIFICATE-----\n"
 *
 * #define democonfigCLIENT_CERTIFICATE_PEM    "...insert here..."
 */
#define democonfigCLIENT_CERTIFICATE_PEM "-----BEGIN CERTIFICATE-----\n"\ 
                                         "...data...\n"\
                                         "...data...\n"\
                                         "-----END CERTIFICATE-----\n"
/**
 * @brief Client's private key.
 *
 *!!! Please note pasting a key into the header file in this manner is for
 *!!! convenience of demonstration only and should not be done in production.
 *!!! Never paste a production private key here!.  Production devices should
 *!!! store keys securely, such as within a secure element.  Additionally,
 *!!! we provide the corePKCS library that further enhances security by
 *!!! enabling securely stored keys to be used without exposing them to
 *!!! software.
 *
 * For AWS IoT MQTT broker, refer to the AWS documentation below for details
 * regarding clientauthentication.
 * https://docs.aws.amazon.com/iot/latest/developerguide/client-authentication.html
 *
 * @note This private key should be PEM-encoded.
 *
 * Must include the PEM header and footer:
 * "-----BEGIN RSA PRIVATE KEY-----\n"\
 * "...base64 data...\n"\
 * "-----END RSA PRIVATE KEY-----\n"
 *
 * #define democonfigCLIENT_PRIVATE_KEY_PEM    "...insert here..."
 */
#define democonfigCLIENT_PRIVATE_KEY_PEM "-----BEGIN RSA PRIVATE KEY-----\n"                                  \
                                         "...data...\n"\
                                         "...data...\n"\ 
										 "-----END RSA PRIVATE KEY-----\n"
/**
 * @brief An option to disable Server Name Indication.
 *
 * @note When using a local Mosquitto server setup, SNI needs to be disabled
 * for an MQTT broker that only has an IP address but no hostname. However,
 * SNI should be enabled whenever possible.
 */
#define democonfigDISABLE_SNI    ( pdFALSE )

/**
 * @brief Configuration that indicates if the demo connection is made to the AWS IoT Core MQTT broker.
 *
 * If username/password based authentication is used, the demo will use appropriate TLS ALPN and
 * SNI configurations as required for the Custom Authentication feature of AWS IoT.
 * For more information, refer to the following documentation:
 * https://docs.aws.amazon.com/iot/latest/developerguide/custom-auth.html#custom-auth-mqtt
 *
 */
// #define democonfigUSE_AWS_IOT_CORE_BROKER    ( 1 )

/**
 * @brief The username value for authenticating client to the MQTT broker when
 * username/password based client authentication is used.
 *
 * For AWS IoT MQTT broker, refer to the AWS IoT documentation below for
 * details regarding client authentication with a username and password.
 * https://docs.aws.amazon.com/iot/latest/developerguide/custom-authentication.html
 * An authorizer setup needs to be done, as mentioned in the above link, to use
 * username/password based client authentication.
 *
 * #define democonfigCLIENT_USERNAME    "...insert here..."
 */

/**
 * @brief The password value for authenticating client to the MQTT broker when
 * username/password based client authentication is used.
 *
 * For AWS IoT MQTT broker, refer to the AWS IoT documentation below for
 * details regarding client authentication with a username and password.
 * https://docs.aws.amazon.com/iot/latest/developerguide/custom-authentication.html
 * An authorizer setup needs to be done, as mentioned in the above link, to use
 * username/password based client authentication.
 *
 * #define democonfigCLIENT_PASSWORD    "...insert here..."
 */

/**
 * @brief The name of the operating system that the application is running on.
 * The current value is given as an example. Please update for your specific
 * operating system.
 */
#define democonfigOS_NAME                   "FreeRTOS"

/**
 * @brief The version of the operating system that the application is running
 * on. The current value is given as an example. Please update for your specific
 * operating system version.
 */
#define democonfigOS_VERSION                tskKERNEL_VERSION_NUMBER

/**
 * @brief The name of the hardware platform the application is running on. The
 * current value is given as an example. Please update for your specific
 * hardware platform.
 */
#define democonfigHARDWARE_PLATFORM_NAME    "WinSim"

/**
 * @brief The name of the MQTT library used and its version, following an "@"
 * symbol.
 */
#include "core_mqtt.h" /* Include coreMQTT header for MQTT_LIBRARY_VERSION macro. */
#define democonfigMQTT_LIB               "core-mqtt@"MQTT_LIBRARY_VERSION

/**
 * @brief Set the stack size of the main demo task.
 *
 * In the Windows port, this stack only holds a structure. The actual
 * stack is created by an operating system thread.
 */
#define democonfigDEMO_STACKSIZE         configMINIMAL_STACK_SIZE

/**
 * @brief Set the priority of the main demo task.
 */
#define democonfigDEMO_PRIORITY          ( tskIDLE_PRIORITY + 1 )

/**
 * @brief Size of the network buffer for MQTT packets.
 */
#define democonfigNETWORK_BUFFER_SIZE    ( 1024U )

/**
 * @brief Size of the range request from 1nce onboarding service.
 */
#define democonfigRANGE_SIZE             ( 1000U )

#endif /* DEMO_CONFIG_H */

Any help will be appreciated.

Thanks in advance,
Guy

Hi @guysa. Having 8196 as MBEDTLS_SSL_OUT_CONTENT_LEN may not be large enough. According to mbedTLS documentation, the payload can be at most 16384 bytes. Do you know why it was failing to allocate memory before reducing MBEDTLS_SSL_OUT_CONTENT_LEN? How much memory does the system have?

Thanks for the reply @tianmc1 ,
The system memory-
RAM 128KB, free space left - 15KB
FLASH 512KB, free space left - 195KB
We managed to change the heap size and allocate the required memory with the original value of 16384 for MBEDTLS_SSL_OUT_CONTENT_LEN.
And now the problem is still the certificate verification failed I described.

How did you generate these certificates? Can you first verify the certificates using some other mqtt client (mosquitto may be) on your PC?

We created a new policy on our AWS thing and took the certificates from there.
I will verify the certificates with some other mqtt client.

Any news? Were you able to verify the certificate in a different client?

We verified the certificates with AWS demo in Python.
We managed to run the code properly and send a message, with the following change:
In the call for mbedtls_ssl_conf_authmode, which sets the certificate verification mode, we changed the second argument from MBEDTLS_SSL_VERIFY_REQUIRED to MBEDTLS_SSL_VERIFY_OPTIONAL.

 *
 *  MBEDTLS_SSL_VERIFY_OPTIONAL:  peer certificate is checked, however the
 *                        handshake continues even if verification failed;
 *                        mbedtls_ssl_get_verify_result() can be called after the
 *                        handshake is complete.
 *
 *  MBEDTLS_SSL_VERIFY_REQUIRED:  peer *must* present a valid certificate,
 *                        handshake is aborted if verification failed.
 *                        (default on client)
mbedtls_ssl_conf_authmode( &( pSslContext->config ),
                               MBEDTLS_SSL_VERIFY_OPTIONAL );

So basically the handshake is still failing, any ideas?

Hey everyone,
We still can’t manage to perform the hand-shake with the MBEDTLS_SSL_VERIFY_REQUIRED option.
Appreciate any help

@guysa
1: which CA certificate have you used in your config file?

  • If both RSA and ECDSA are supported, “Starfield Services Root Certificate Authority - G2” is the easiest option.

  • In only RSA is supported, Either ( “Amazon Root CA 1” and “Amazon Root CA 2” ) or just “Starfield Services Root Certificate Authority - G2” will work.

  • If only ECDSA is supported, it’s best to include both “Amazon Root CA 3” and “Amazon Root CA 4”.

2: You can validate your combination of client key / client cert / CA cert / hostname by installing mbedtls on your computer and using the ssl_client2 utility with debug logging enabled.

3: You can enable MBEDTLS_DEBUG_C in your mbedtls_config.h file to enable debug printing via uart, which should help narrow down the issue. You can reference the transport interface implementation used in the STM32U5 reference integration for details on enabling debug prints. Make sure to add calls to mbedtls_ssl_conf_dbg and mbedtls_debug_set_threshold.

4: 16K is a quite small heap size when using mbedtls with RSA enable, so you may be encountering a malloc failure when receiving the server’s certificate chain. You might consider disabling RSA and using ECDSA instead due to the significantly lower RAM requirements.

Thanks @PaulB-AWS ,
In the config file we are using the “Amazon Root CA 1” for the server’s root CA certificate, and for the client we are using the client certificate and the private key(RSA) that we got from the our AWS thing.

I’m sorry if I wasn’t clear, we are using 64K for the heap size. The 16K is for the outgoing/ingoing buffers defined with the macros MBEDTLS_SSL_OUT_CONTENT_LEN and MBEDTLS_SSL_IN_CONTENT_LEN.

Can you enable MbedTLS debug logs and share the output?

Thanks.