MQTT mutual auth demo "Failed to perform TLS handshake"

Hello forum users,

I need help with demo example MQTT_Mutual_Auth using windows simulator. First I have to say that I ported entire demo to compile with mingw on eclipse. Then followed instructions in “mqtt_broker_setup.txt”, generated all certificates. When testing these certificates with Mosquitto broker, sub, pub, mqtt communication works fine! Now I have implemented certificates into “demo_config.h” and tested with demo. I am getting following errors:

console output:

Attempting to open interface number 1.
Successfully opened interface number 1.
0 2530 [IP-Task]
vDHCPProcess: offer 192.168.1.104
vDHCPProcess: offer 192.168.1.104
1 2630 [IP-Task]
[INFO] [MQTTDemo] [vApplicationIPNetworkEventHook:206] ---------STARTING DEMO---------
2 2630 [IP-Task]
[INFO] [MQTTDemo] [vApplicationIPNetworkEventHook:215]
IP Address: 192.168.1.104
3 2630 [IP-Task]
[INFO] [MQTTDemo] [vApplicationIPNetworkEventHook:218] Subnet Mask: 255.255.255.0
4 2630 [IP-Task]
[INFO] [MQTTDemo] [vApplicationIPNetworkEventHook:221] Gateway Address: 192.168.1.1
5 2630 [IP-Task]
[INFO] [MQTTDemo] [vApplicationIPNetworkEventHook:224] DNS Server Address: 192.168.1.1
6 2632 [DemoTask]
[INFO] [MQTTDemo] [prvConnectToServerWithBackoffRetries:662] Creating a TLS connection to 192.168.1.22:8883.
7 3334 [DemoTask]
[ERROR] [TlsTransport] [tlsHandshake:553] Failed to perform TLS handshake: mbedTLSError= X509 - Certificate verification failed, e.g. CRL, CA or signature check failed : .
8 3335 [DemoTask]
[WARN] [MQTTDemo] [prvConnectToServerWithBackoffRetries:688] Connection to the broker failed. Retrying connection with backoff and jitter.
9 3783 [DemoTask]
[INFO] [MQTTDemo] [prvConnectToServerWithBackoffRetries:662] Creating a TLS connection to 192.168.1.22:8883.
10 3933 [DemoTask]
[ERROR] [TlsTransport] [tlsHandshake:553] Failed to perform TLS handshake: mbedTLSError= X509 - Certificate verification failed, e.g. CRL, CA or signature check failed : .
11 3934 [DemoTask]
[WARN] [MQTTDemo] [prvConnectToServerWithBackoffRetries:688] Connection to the broker failed. Retrying connection with backoff and jitter.
12 4203 [DemoTask]
[INFO] [MQTTDemo] [prvConnectToServerWithBackoffRetries:662] Creating a TLS connection to 192.168.1.22:8883.
13 4333 [DemoTask]
[ERROR] [TlsTransport] [tlsHandshake:553] Failed to perform TLS handshake: mbedTLSError= X509 - Certificate verification failed, e.g. CRL, CA or signature check failed : .
14 4334 [DemoTask]
[WARN] [MQTTDemo] [prvConnectToServerWithBackoffRetries:688] Connection to the broker failed. Retrying connection with backoff and jitter.
15 4761 [DemoTask]
[INFO] [MQTTDemo] [prvConnectToServerWithBackoffRetries:662] Creating a TLS connection to 192.168.1.22:8883.
16 4934 [DemoTask]
[ERROR] [TlsTransport] [tlsHandshake:553] Failed to perform TLS handshake: mbedTLSError= X509 - Certificate verification failed, e.g. CRL, CA or signature check failed : .
17 4934 [DemoTask]
[WARN] [MQTTDemo] [prvConnectToServerWithBackoffRetries:688] Connection to the broker failed. Retrying connection with backoff and jitter.
18 6646 [DemoTask]
[INFO] [MQTTDemo] [prvConnectToServerWithBackoffRetries:662] Creating a TLS connection to 192.168.1.22:8883.
19 6833 [DemoTask]
[ERROR] [TlsTransport] [tlsHandshake:553] Failed to perform TLS handshake: mbedTLSError= X509 - Certificate verification failed, e.g. CRL, CA or signature check failed : .
20 6833 [DemoTask]
[WARN] [MQTTDemo] [prvConnectToServerWithBackoffRetries:688] Connection to the broker failed. Retrying connection with backoff and jitter.
21 11136 [DemoTask]
[INFO] [MQTTDemo] [prvConnectToServerWithBackoffRetries:662] Creating a TLS connection to 192.168.1.22:8883.
vAssertCalled( …\DemoTasks\MutualAuthMQTTExample.c, 538

mosquito broker:
1641149747: mosquitto version 2.0.14 starting
1641149747: Config loaded from mosquittoFREERTOS.conf.
1641149747: Opening ipv6 listen socket on port 8883.
1641149747: Opening ipv4 listen socket on port 8883.
1641149747: mosquitto version 2.0.14 running
1641149777: New connection from 192.168.1.104:14084 on port 8883.
1641149777: OpenSSL Error[0]: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
1641149777: Client disconnected: protocol error.
1641149777: New connection from 192.168.1.104:11234 on port 8883.
1641149777: OpenSSL Error[0]: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
1641149777: Client disconnected: protocol error.
1641149777: New connection from 192.168.1.104:18893 on port 8883.
1641149777: OpenSSL Error[0]: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
1641149777: Client disconnected: protocol error.
1641149778: New connection from 192.168.1.104:14054 on port 8883.
1641149778: OpenSSL Error[0]: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
1641149778: Client disconnected: protocol error.
1641149780: New connection from 192.168.1.104:21578 on port 8883.
1641149781: OpenSSL Error[0]: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
1641149781: Client disconnected: protocol error.
1641149782: New connection from 192.168.1.104:31607 on port 8883.
1641149782: OpenSSL Error[0]: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
1641149782: Client disconnected: protocol error.

Can you please suggest in which direction to proceed to overcome this issue?

rum

When testing these certificates with Mosquitto broker, sub, pub, mqtt communication works fine!

What were you using as the client when the connection was successful? The MQTT mutual auth demo, or something else entirely?

Now I have implemented certificates into “demo_config.h” and tested with demo. I am getting following errors

It could just be the format of the certificate - how did you convert it for inclusion into the header file.

NOTE: Adding certificates into the header file is for evaluation convenience only. Don’t use real production certificates in this way - they should be stored securely in something like a secure element or enclave.

  1. as a client I was using mosquitto_sub.exe, mosquitto_pub.exe

  2. I have simply followed example in “demo_config.h” , here is fraction of what I use

#define democonfigROOT_CA_PEM
“-----BEGIN CERTIFICATE-----\n”
“MIID6TCCAtGgAwIBAgIUaWlQIcncLfgb3t0VSd3wXOPDZDEwDQYJKoZIhvcNAQEL\n”


“-----END CERTIFICATE-----\n”

there is also backslash (line continuation) at the end of each line (does not show in post)

Curious as you have already shown the certificates to be valid.

So I can familiarize myself with the files - is this the demo you are using https://freertos.org/mqtt/mutual-authentication-mqtt-example.html, which is for this build project (before you converted to MingW): https://github.com/FreeRTOS/FreeRTOS/tree/main/FreeRTOS-Plus/Demo/coreMQTT_Windows_Simulator/MQTT_Mutual_Auth

Exactly this demo application.

Note:
I debug this application running in virtual machine. Mosquitto broker is running on host machine.

Well it looks like I messed up the formatting even if you didn’t :grinning: - so far I have this error:

7 2022 [DemoTask] [ERROR] [TlsTransport] [setRootCa:263] Failed to parse server root CA certificate: mbedTLSError= X509 - Format not recognized as DER or PEM : <No-Low-Level-Code>.
8 2022 [DemoTask] [WARN] [MQTTDemo] [prvConnectToServerWithBackoffRetries:689] Connection to the broker failed. Retrying connection with backoff and jitter.
9 2382 [DemoTask] [INFO] [MQTTDemo] [prvConnectToServerWithBackoffRetries:664] Creating a TLS connection to 192.168.0.100:8883.

After regenerating all the keys I get:

13 4601 [DemoTask] [ERROR] [TlsTransport] [tlsHandshake:552] Failed to perform TLS handshake: mbedTLSError= SSL - A fatal alert message was received from our peer : <No-Low-Level-Code>.
14 4601 [DemoTask] [WARN] [MQTTDemo] [prvConnectToServerWithBackoffRetries:689] Connection to the broker failed. Retrying connection with backoff and jitter.
15 6306 [DemoTask] [INFO] [MQTTDemo] [prvConnectToServerWithBackoffRetries:664] Creating a TLS connection to 192.168.0.100:8883.

and on the server side:

1641160618: mosquitto version 1.4.11 (build date 20/02/2017 23:24:29.40) starting
1641160618: Config loaded from mosquitto.conf.
1641160618: Opening ipv6 listen socket on port 8883.
1641160618: Opening ipv4 listen socket on port 8883.
1641160631: New connection from 192.168.0.27 on port 8883.
1641160632: OpenSSL Error: error:14089086:SSL routines:ssl3_get_client_certificate:certificate verify failed
1641160632: Socket error on client <unknown>, disconnecting.
1641160633: New connection from 192.168.0.27 on port 8883.
1641160633: OpenSSL Error: error:14089086:SSL routines:ssl3_get_client_certificate:certificate verify failed
1641160633: Socket error on client <unknown>, disconnecting.

so a bit different to yours. I’ve not verified I can connect to the broker using the mosquitto clients yet though…

…getting the same error message from the broker when I try connecting from mqtt.fx too.

Thank you Richard that you are trying to help me out. Yes, your errors are different but it is in handshake mechanism, same like my case. I wonder how to enable printout of detailed debug messages from mbedtls?

I included
#include “debug.h”

then called
mbedtls_debug_set_threshold(level);

then I need to provide “user” function through “mbedtls_ssl_conf_dbg” but I don’t find any example how to map it to FREERTOS vLoggingPrintf.

My next step is to create a successful connection from a third party tool before I do so from the FreeRTOS application. I’ve not used mosquitto_sub.exe and mosquitto_pub.exe before but will look into doing so - I normally use mqtt.fx. In the mean time there will be more people in the office today after the new year so I will also ask someone else to try the same.

Here is my status. I regenerated certificates again. It is not clear to me what should I use for “Common Name” though. In all three certificates I specify different value.
Mosquitto broker running on host PC, Mosquitto_pub on virtual machine (client name is ULSc-board). Publish is successful, on broker I get:
image

for “pub” I use switch --insecure, if I remove this switch and repeat “pub” operation, I get on broker:
image

in broker configuration is commented
#tls_version tlsv1.2
#require_certificate true

next I implement certificates to Demo application and on broker I get
image

on debug console I get

I am not sure how to proceed from this point…
How to make this demo work?

Hi @rum ,

Can you check if democonfigDISABLE_SNI in your project is set to pdTRUE? For local Mosquitto server, this option needs to be set to pdTRUE to disable SNI check. Otherwise the TLS handshake will fail to verify the certificate.

1 Like

I set

#define democonfigDISABLE_SNI ( pdTRUE )

and demo application WORKS!!!

I was reading many times text accompanying this define and did not realized that setting true it actually disables the option.

Thank you tianmc1

Hi,

I seem to have the same exact problem as @rum. I have set democonfigDISABLE_SNI to pdTRUE and still get the same error message. I’ve tried both the mqtt_basic_tls_demo and the mqtt_mutual_auth_demo and I cannot get beyond the TLS handshake. It appears that the client (this demo) is not recognizing the CA certificate sent back by the local Mosquitto MQTT server I’ve set up on my local network. I’ve checked all of the certificates multiple times. All certificates match and are properly set. Here is the output I get on both demos:

The following network interfaces are available:

Interface 1 - rpcap://\Device\NPF_{5D39A935-273A-4C2B-B435-6D00F65AE23D}
              (Network adapter 'Microsoft' on local host)

Interface 2 - rpcap://\Device\NPF_{6BEE4B80-8613-4A10-A484-613750DB53B9}
              (Network adapter 'TAP-Windows Adapter V9 for OpenVPN Connect' on local host)

Interface 3 - rpcap://\Device\NPF_{A6EB9E56-3268-486E-8EC9-EDA60EDE6DE0}
              (Network adapter 'Microsoft' on local host)

Interface 4 - rpcap://\Device\NPF_{F73E6B58-E68C-4DB4-A1BA-CF2C5CBE0E36}
              (Network adapter 'DisplayLink USB Gigabit Network Adapter' on local host)

Interface 5 - rpcap://\Device\NPF_{A464D6CF-8EA4-4B11-8806-0D87E86036D7}
              (Network adapter 'Microsoft' on local host)

Interface 6 - rpcap://\Device\NPF_{757D8571-E4C6-4E09-9731-101DC1873BE6}
              (Network adapter 'Microsoft' on local host)

Interface 7 - rpcap://\Device\NPF_{891B6912-8DE2-4F6E-9703-79CE15ED34AD}
              (Network adapter 'Intel(R) Ethernet Connection (10) I219-LM' on local host)


The interface that will be opened is set by "configNETWORK_INTERFACE_TO_USE", which
should be defined in FreeRTOSConfig.h
Attempting to open interface number 4.
Successfully opened interface number 4.
0 1482 [IP-Task] vDHCPProcess: offer 192.168.1.34
vDHCPProcess: offer 192.168.1.34
1 1542 [IP-Task] [INFO] [MQTTDemo] [vApplicationIPNetworkEventHook:190] ---------STARTING DEMO---------

2 1543 [IP-Task] [INFO] [MQTTDemo] [vApplicationIPNetworkEventHook:199]

IP Address: 192.168.1.34

3 1543 [IP-Task] [INFO] [MQTTDemo] [vApplicationIPNetworkEventHook:202] Subnet Mask: 255.255.255.0

4 1543 [IP-Task] [INFO] [MQTTDemo] [vApplicationIPNetworkEventHook:205] Gateway Address: 192.168.1.1

5 1543 [IP-Task] [INFO] [MQTTDemo] [vApplicationIPNetworkEventHook:208] DNS Server Address: 192.168.1.1



6 1543 [DemoTask] [INFO] [MQTTDemo] [prvConnectToServerWithBackoffRetries:664] Creating a TLS connection to NUC7:8883.

7 6562 [DemoTask] DNS[0x79B9]: The answer to 'NUC7' (192.168.1.30) will be stored
[ERROR] [TlsTransport] [tlsHandshake:552] 8 7143 [DemoTask] Failed to perform TLS handshake: mbedTLSError= X509 - Certificate verification failed, e.g. CRL, CA or signature check failed : <No-Low-Level-Code>.
9 7143 [DemoTask] [WARN] [MQTTDemo] [prvConnectToServerWithBackoffRetries:689] Connection to the broker failed. Retrying connection with backoff and jitter.
10 7284 [DemoTask] [INFO] [MQTTDemo] [prvConnectToServerWithBackoffRetries:664] Creating a TLS connection to NUC7:8883.

11 7324 [DemoTask] [ERROR] [TlsTransport] [tlsHandshake:552] Failed to perform TLS handshake: mbedTLSError= X509 - Certificate verification failed, e.g. CRL, CA or signature check failed : <No-Low-Level-Code>.
12 7324 [DemoTask] [WARN] [MQTTDemo] [prvConnectToServerWithBackoffRetries:689] Connection to the broker failed. Retrying connection with backoff and jitter.
13 7597 [DemoTask] [INFO] [MQTTDemo] [prvConnectToServerWithBackoffRetries:664] Creating a TLS connection to NUC7:8883.

14 7683 [DemoTask] [ERROR] [TlsTransport] [tlsHandshake:552] Failed to perform TLS handshake: mbedTLSError= X509 - Certificate verification failed, e.g. CRL, CA or signature check failed : <No-Low-Level-Code>.
15 7683 [DemoTask] [WARN] [MQTTDemo] [prvConnectToServerWithBackoffRetries:689] Connection to the broker failed. Retrying connection with backoff and jitter.
16 8395 [DemoTask] [INFO] [MQTTDemo] [prvConnectToServerWithBackoffRetries:664] Creating a TLS connection to NUC7:8883.

17 8483 [DemoTask] [ERROR] [TlsTransport] [tlsHandshake:552] Failed to perform TLS handshake: mbedTLSError= X509 - Certificate verification failed, e.g. CRL, CA or signature check failed : <No-Low-Level-Code>.
18 8483 [DemoTask] [WARN] [MQTTDemo] [prvConnectToServerWithBackoffRetries:689] Connection to the broker failed. Retrying connection with backoff and jitter.
19 12463 [DemoTask] [INFO] [MQTTDemo] [prvConnectToServerWithBackoffRetries:664] Creating a TLS connection to NUC7:8883.

20 12507 [DemoTask] [ERROR] [TlsTransport] [tlsHandshake:552] Failed to perform TLS handshake: mbedTLSError= X509 - Certificate verification failed, e.g. CRL, CA or signature check failed : <No-Low-Level-Code>.
21 12507 [DemoTask] [WARN] [MQTTDemo] [prvConnectToServerWithBackoffRetries:689] Connection to the broker failed. Retrying connection with backoff and jitter.
22 15014 [DemoTask] [INFO] [MQTTDemo] [prvConnectToServerWithBackoffRetries:664] Creating a TLS connection to NUC7:8883.

23 15063 [DemoTask] [ERROR] [TlsTransport] [tlsHandshake:552] Failed to perform TLS handshake: mbedTLSError= X509 - Certificate verification failed, e.g. CRL, CA or signature check failed : <No-Low-Level-Code>.
24 15063 [DemoTask] [ERROR] [MQTTDemo] [prvConnectToServerWithBackoffRetries:684] Connection to the broker failed, all attempts exhausted.
vAssertCalled( C:\Projects\FreeRTOS\FreeRTOS-Plus\Demo\coreMQTT_Windows_Simulator\MQTT_Mutual_Auth\DemoTasks\MutualAuthMQTTExample.c, 538

BTW, to check that I’ve set the MQTT server up properly, I use a third party MQTT client tool to see if I can connect to it. I am using MQTT Explorer. I’ve set it up for secure connection and it successfully connects to the MQTT server. Also, I’ve tried the unsecure demo, MQTT_Plain_Text, and that demo works.

Please advise,

Johnas

I also used the Python script used for the AWS set up for an IoT connection. After setting the parameters for my local MQTT server and pointing to the certificate/key files that contain the same data used in the FreeRTOS code, I was able to connect successfully to my local MQTT server. My conclusion is that the FreeRTOS code is not comparing the CA certificate correctly.

Also, I tried connecting the FreeRTOS demo code to my AWS MQTT server that I set up using my account and I still get an error although it looks like a different error. I’ll have to rerun it to see what is going on there. The Python script runs successfully. Again, all certificates/keys were manually checked and verified to be correct.

Are you using PEM format? Are you sure that you are setting the democonfigDISABLE_SNI for the correct demo? You can verify that by adding some non-compilable code like the following:

#if ( democonfigDISABLE_SNI != pdTRUE )
    /* Something that results in compilation error. */
    asdf
#endif

If that is correct, is it possible for you to capture wireshark logs? Also, can you describe your local setup so that I can try to replicate?

Hi Gaurav!

Nice to hear from you again.

Yes and yes to your questions above (you can verify it with the demo_config.h file I uploaded). BTW, the code you provided:

#if ( democonfigDISABLE_SNI != pdTRUE )
    /* Something that results in compilation error. */
    asdf
#endif

does not work because pdTRUE is define with baseType blah, blah, blah… So, I simply defined democonfigDISABLE_SNI with ( 1 ) for this test. That worked and proved that it is set correctly. I reverted the file to my original settings after this test:
#define democonfigDISABLE_SNI (pdTRUE)

My Mosquitto server is configured and enabled as shown in the screenshot below:

This server is running on a Windows machine (NUC) located on my LAN and has the local address of 192.168.1.30

Wireshark log is provided here (3 logs - see MQTTExplorerTLSMutual.pcapng):
MQTTLogs.zip (2.5 MB)

Screenshot of FreeRTOS demo program packet rejecting certificate sent by MQTT server (MQTTExplorerTLSMutual.pcapng):

I’m running the MutualAuthMQTTExample demo from FreeRTOS where I’ve modified the demo_config.h and FreeRTOSConfig.h files as per instructions. I’ve included them here for your review:
demo_config.h (5.0 KB)
FreeRTOSConfig.h (10.2 KB)

The corresponding output is what I get with all of these settings in place. The Wireshark log reflects this run as well:

Let me know if you need any further information or have any questions.

Thanks for your help,

Johnas

I followed the following instructions for setting up the broker: https://raw.githubusercontent.com/FreeRTOS/FreeRTOS/main/FreeRTOS-Plus/Demo/coreMQTT_Windows_Simulator/MQTT_Mutual_Auth/mqtt_broker_setup.txt

Here are the commands that I used to create various certs and keys:

openssl.exe req -x509 -nodes -sha256 -days 365 -newkey rsa:2048 -keyout ca.key -out ca.crt
openssl.exe req -nodes -sha256 -new -keyout server.key -out server.csr
openssl.exe x509 -req -sha256 -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 365
openssl.exe genrsa -out client.key 2048
openssl.exe req -new -out client.csr -key client.key
openssl.exe x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out client.crt -days 365

After you have created the certs, please verify that the server and client cert are correctly signed:

$ openssl.exe verify -CAfile ca.crt server.crt
server.crt: OK

$ openssl.exe verify -CAfile ca.crt client.crt
client.crt: OK

Here is my mosquitto.conf:

port 8883
cafile C:\<path_to_certs_folder>\ca.crt
certfile C:\<path_to_certs_folder>\server.crt
keyfile C:\<path_to_certs_folder>\server.key
require_certificate true
tls_version tlsv1.2
allow_anonymous true

This setup works for me with mutual auth demo. Can you try exactly these instructions from scratch and let me know?

Thanks.

Hi Gaurav,

OK, I figured out what I was doing wrong. When creating the certificates, I kept using the same organization name for all the certificates (ca root, server, client). When I initially tried to use them with this code (or any utility for that matter), I kept getting a “self-signed certificate” error. Not knowing what that meant at the time, I naturally Googled it. I found someone saying that they followed a certain procedure to get rid of that error. So, I followed the same procedure. I’m not sure if I did something else as well. It seemed to work with the MQTT tool I was using to test the MQTT broker I had set up. I even used the Python scripts from AWS pointing the endpoint to my local MQTT broker and all seemed well. When I tried this code, it would error out on the certificate comparison done by the client. I naturally thought it must be this code since all other clients were working.

So, I recreated all of the certificates as you recommended. I repeated my mistake and saw the self-signed certificate error again when I ran your verify command. In the back of my head, I had remembered reading a suggestion to use a different organization name for the different certificates, but I don’t remember a warning about using the same organization name for the certificates.

I restarted the process again from scratch. This time, I used a different name for the organization in each of the certificates. I ran your verify command again. All was well. Then I ran the code with the new certificates and it worked!

My next steps will be to move the MQTT broker to my AWS account and run the code against that. After that, I’m going to port this code to my nRF9160DK.

Thanks for your help…again.