AFR & ESP32: IotJsonUtils_FindJsonValue() API fetches an incorrect pJsonValue

Namaste Community Member,

My development environment comprises of

  • FreeRTOS version: V202002.00. (@file iot_demo_mqtt.c)
  • FreeRTOS Serializer V1.1.1 (@file iot_json_utils.c)

Requirement

  1. Publish a JSON formatted payload from the AWS IoT Console (Publish to a subscribed topic).
  2. Obtain the JSON Value corresponding to a specific Key.

Issue Description

For the above second step, I am leveraging the already available JSON Serializer library available at FreeRTOS\libraries\c_sdk\standard\serializer\src\iot_json_utils.c

Within the _mqttSubscriptionCallback() API in the @file iot_demo_mqtt.c, I am calling the IotJsonUtils_FindJsonValue() API. For my testing purpose, I am passing a hardcoded “time” string as the Key to obtain the associated Value, which is populated incorrectly.

Publish JSON Message

Below is the simple JSON formatted message I am publishing from the AWS IoT Console.

{
  "time":123456,
  "action":1
}

Firmware Code Block

static void _mqttSubscriptionCallback( void * param1,
                                       IotMqttCallbackParam_t * const pPublish )
{
    ....
    ....

    bool retVal = false;
    const char * value = NULL;
    size_t lengthOfValue = 0;
    size_t lengthOfTimeKey = strlen("time");

    retVal = IotJsonUtils_FindJsonValue( pPayload,
                (size_t) pPublish->u.message.info.payloadLength,
                "time",
                lengthOfTimeKey,
                &value,
                &lengthOfValue );

    if( false == retVal )
    {
        IotLogError( "IotJsonUtils_FindJsonValue() failed." );
    }
    else
    {
        IotLogInfo( "length of pPayload = %d", pPublish->u.message.info.payloadLength );
        IotLogInfo( "length of \"time\" string = %d", lengthOfTimeKey );
        IotLogInfo( "value = %d", value );
        IotLogInfo( "lengthOfValue = %d", lengthOfValue );
    }
}

Console Logs

Below is the console log viewed on the USB Serial port.

The variable lengthOfValue is correctly populated as 6. However, the value itself is incorrectly populated as 1073566202 or a similar number instead of the expected number 123456.

116 6553 [iot_thread] [INFO ][DEMO][65530] Incoming PUBLISH received:
Subscription topic filter: dualOutlet/topic/RFID
Publish QoS: 0
Publish payload: {
  "time":123456,
  "action":1
}
118 6553 [iot_thread] [INFO ][DEMO][65530] length of pPayload = 33
119 6553 [iot_thread] [INFO ][DEMO][65530] length of "time" string = 4
121 6554 [iot_thread] [INFO ][DEMO][65540] value = 1073566202
122 6554 [iot_thread] [INFO ][DEMO][65540] lengthOfValue = 6

Requesting assistance to obtain the correct value for the given key of my JSON payload.

Thanks | Regards,
Dipen

Namaste Community Member,

I also tried using the coreJSON library as an alternative to the JSON Serializer library I have used above. Same observation is encountered, and the “value” is incorrectly populated.

Project modifications for using coreJSON library

  1. Create a new coreJSON folder inside the path - FreeRTOS\libraries\3rdparty\
  2. Copy the core_json.c and core_json.h files in this folder.
  3. Append the below code in the CMakeLists.txt file residing inside the path - FreeRTOS\libraries\3rdparty\
# core_json - A parser strictly enforcing the ECMA-404 JSON standard, suitable for microcontrollers
if(EXISTS "${AFR_3RDPARTY_DIR}/core_json")
    afr_3rdparty_module(core_json)
    target_sources(
        3rdparty::core_json INTERFACE
        "${AFR_3RDPARTY_DIR}/core_json/core_json.h"
        "${AFR_3RDPARTY_DIR}/core_json/core_json.c"
    )
    target_include_directories(
        3rdparty::core_json INTERFACE
        "${AFR_3RDPARTY_DIR}/core_json"
    )
endif()

Associated source code

static void _mqttSubscriptionCallback( void * param1,
                                       IotMqttCallbackParam_t * const pPublish )
{
    ....
    ....


    JSONStatus_t result = JSONBadParameter;
    char * value = NULL;
    size_t valueLength = 0;

    result = JSON_Validate( pPayload, pPublish->u.message.info.payloadLength );

    if( JSONSuccess != result )
    {
        IotLogInfo("JSON_Validate failed with \"result\" as %d.", result );
    }
    else
    {
        IotLogInfo("JSON_Validate passed with \"result\" as %d.", result );
        result = JSON_Search( (char *) pPayload,
                pPublish->u.message.info.payloadLength,
                "time",
                4,
                &value,
                &valueLength );

        if( JSONSuccess != result )
        {
            IotLogInfo("JSON_Validate failed with \"result\" as %d.", result );
        }

        IotLogInfo("JSON_Search passed with \"result\" as %d.", result );
        IotLogInfo( "JSON_Search: value = %d", value );
        IotLogInfo( "JSON_Search: valueLength = %d", valueLength );
    }

Associated Publish JSON Message

Below is the simple JSON formatted message I am publishing from the AWS IoT Console.

{
  "time":123456,
  "action":1
}

Associated Console Log

Below is the console log viewed on the USB Serial port.

The variable “valueLength” is correctly populated as 10. However, the “value” variable is incorrectly populated as 1073564255 or a similar number instead of the expected number 9876543210.

147 9380 [iot_thread] [INFO ][DEMO][93790] Incoming PUBLISH received:
Subscription topic filter: dualOutlet/topic/RFID
Publish QoS: 0
Publish payload: {
"time": 9876543210,
"action": 1
}
148 9380 [iot_thread] [INFO ][DEMO][93800] JSON_Validate passed with "result" as 1.
149 9380 [iot_thread] [INFO ][DEMO][93800] JSON_Search passed with "result" as 1.
150 9380 [iot_thread] [INFO ][DEMO][93800] JSON_Search: value = 1073564255
151 9380 [iot_thread] [INFO ][DEMO][93800] JSON_Search: valueLength = 10

With both libraries showcasing the same behaviour, the issue looks to be in my firmware. Please highlight what I may be missing and, the required code change(s).

Thanks | Regards,
Dipen

Hi @dipen12,

I have posted here https://github.com/aws/amazon-freertos/issues/2864. Posting this here for visibility.

Thanks!

Carl

Namaste Community Member,

Happy to update that I have root caused the issue and, resolved the same. Below is the explanation.

Root Cause Analysis

I was expecting the “value” variable to be populated with “123456” as a number. And, was attempting to print the variable content. However, value is defined as “char * value”. This translates as below,

value[0] = 1 (i.e., ASCII Character 1 or Hexadecimal value of 0x31 to be precise)
value[1] = 2
value[2] = 3
value[3] = 4
value[4] = 5
value[5] = 6

Further, I need to NULL terminate the value variable by a count equal to the valueLength. This shall ensure the exact JSON Value corresponding to the JSON Key.

Code changes

Original code

        IotLogInfo( "JSON_Search: value = %d", value );

Required changes

        // NULL terminate the string
        value[valueLength] = 0x00;
        
        // Print as string and not an integer value
        IotLogInfo( "JSON_Search: value = %s", value );

Console Logs

Five console logs with different “values” published from the AWS IoT Console are validated as below.

--- Log 1 ---
69 2652 [iot_thread] [INFO ][DEMO][26520] Incoming PUBLISH received:
Subscription topic filter: dualOutlet/topic/RFID
Publish QoS: 0
Publish payload: {
"time":123,
"action":1
}
70 2652 [iot_thread] [INFO ][DEMO][26520] JSON_Validate passed with "result" as 1.
71 2652 [iot_thread] [INFO ][DEMO][26520] JSON_Search passed with "result" as 1.
72 2652 [iot_thread] [INFO ][DEMO][26520] JSON_Search: valueLength = 3
73 2652 [iot_thread] [INFO ][DEMO][26520] JSON_Search: value = 123

--- Log 2 ---
88 4004 [iot_thread] [INFO ][DEMO][40040] Incoming PUBLISH received:
Subscription topic filter: dualOutlet/topic/RFID
Publish QoS: 0
Publish payload: {
"time":123456,
"action":1
}
89 4004 [iot_thread] [INFO ][DEMO][40040] JSON_Validate passed with "result" as 1.
90 4004 [iot_thread] [INFO ][DEMO][40040] JSON_Search passed with "result" as 1.
91 4004 [iot_thread] [INFO ][DEMO][40040] JSON_Search: valueLength = 6
92 4004 [iot_thread] [INFO ][DEMO][40040] JSON_Search: value = 123456

--- Log 3 ---
106 5338 [iot_thread] [INFO ][DEMO][53380] Incoming PUBLISH received:
Subscription topic filter: dualOutlet/topic/RFID
Publish QoS: 0
Publish payload: {
"time":9876543210,
"action":1
}
107 5338 [iot_thread] [INFO ][DEMO][53380] JSON_Validate passed with "result" as 1.
108 5338 [iot_thread] [INFO ][DEMO][53380] JSON_Search passed with "result" as 1.
109 5338 [iot_thread] [INFO ][DEMO][53380] JSON_Search: valueLength = 10
110 5338 [iot_thread] [INFO ][DEMO][53380] JSON_Search: value = 9876543210

--- Log 4 ---
128 7014 [iot_thread] [INFO ][DEMO][70140] Incoming PUBLISH received:
Subscription topic filter: dualOutlet/topic/RFID
Publish QoS: 0
Publish payload: {
"time":98765432109876543210,
"action":1
}
129 7014 [iot_thread] [INFO ][DEMO][70140] JSON_Validate passed with "result" as 1.
130 7014 [iot_thread] [INFO ][DEMO][70140] JSON_Search passed with "result" as 1.
131 7014 [iot_thread] [INFO ][DEMO][70140] JSON_Search: valueLength = 20
132 7014 [iot_thread] [INFO ][DEMO][70140] JSON_Search: value = 98765432109876543210

--- Log 5 ---
146 8387 [iot_thread] [INFO ][DEMO][83870] Incoming PUBLISH received:
Subscription topic filter: dualOutlet/topic/RFID
Publish QoS: 0
Publish payload: {
"time":987654321098765432109876543210,
"action":1
}
147 8387 [iot_thread] [INFO ][DEMO][83870] JSON_Validate passed with "result" as 1.
148 8387 [iot_thread] [INFO ][DEMO][83870] JSON_Search passed with "result" as 1.
149 8387 [iot_thread] [INFO ][DEMO][83870] JSON_Search: valueLength = 30
150 8387 [iot_thread] [INFO ][DEMO][83870] JSON_Search: value = 987654321098765432109876543210

Additional Information

  1. The code has been tested with the coreJSON library. Similar code changes are required when using the FreeRTOS Serializer V1.1.1 (@file iot_json_utils.c)

  2. Leverage the strtoul() function to convert a string to an integer.

Hope this helps, someone !!

Thanks | Regards,
Dipen

Namaste @lundinc,

Once again, thank you and, appreciated :+1:

Thanks | Regards,
Dipen