Does the Amazon 202007.00 version of OTA ( for the esp32 ) work?

Using the OTA demo code in the 202007 version of Amazon’s FreeRTOS,

1 ) The “new” code downloads, successfully runs, and resets the processor.
2 ) The status on the AWS job changes to “In progress”.
3 ) After “rebooting”, the ESP32 tries to run “selftest” but gets the following messages in the “log”.

49 1080 [OTA Agent Task] [prvInSelfTestHandler] prvInSelfTestHandler, platform is in self-test.
50 1081 [OTA Agent Task] [prvInSelfTestHandler] Job in self test but platform state is not!

4 ) The “OTA Agent Task” fails the job and the device is reset.

51 1085 [OTA Agent Task] [prvPublishStatusMessage] Msg: {“status”:“FAILED”,“statusDetails”:{“reason”:“rejected: 0x26000000”}}
52 1085 [OTA Agent Task] [INFO ][MQTT][10850] (MQTT connection 0x3ffd8eb8) MQTT PUBLISH operation queued.
53 1085 [OTA Agent Task] [INFO ][MQTT][10850] (MQTT connection 0x3ffd8eb8, PUBLISH operation 0x3ffd9d78) Waiting for operation completion.
54 1100 [OTA Agent Task] [INFO ][MQTT][11000] (MQTT connection 0x3ffd8eb8, PUBLISH operation 0x3ffd9d78) Wait complete with result SUCCESS.
55 1100 [OTA Agent Task] [prvPublishStatusMessage] ‘FAILED’ to $aws/things/A12345-67891/jobs/AFR_OTA-ota3_Orange/update
56 1100 [OTA Agent Task] [prvResetDevice] Attempting forced reset of device…

5 ) The status on the AWS job changes to “Failed”.
6 ) The ESP32 reboots and continues to run the “new” code.

In the aws_iot_ota_agent.c file, the prvInSelfTestHandler “generates” the two “contradictory” log statements that the “platform is in self-test” followed by the “Job in self test but platform state is not!” shown in “item 3” above.

This is due to the “failing” tof the prvInSelftest function. In version 202002.00 of the Amazon FreeRTOS library, this was checked with the "prvStartSelfTestTimer function.

Has anyone verified this change actually works on the ESP32?

And if so, what should I possibly be checking to see why it’s not for me? Or what could I possibly be doing wrong?

Any “feedback” would greatly be appreciated!

Tom

Hello Tom,

We are trying to reproduce and investigate this issue.

The prvInSelftest function in aws_iot_ota_agent.c checks the following,

if( xOTA_Agent.xPALCallbacks.xGetPlatformImageState( xOTA_Agent.ulServerFileID ) == eOTA_PAL_ImageState_PendingCommit )

and in my code the “xOTA_Agent.xPALCallbacks.xGetPlatformImageState( xOTA_Agent.ulServerFileID” returns a “2” which from

typedef enum
{
eOTA_PAL_ImageState_Unknown = 0,
eOTA_PAL_ImageState_PendingCommit,
eOTA_PAL_ImageState_Valid,
eOTA_PAL_ImageState_Invalid,
} OTA_PAL_ImageState_t;

would be "eOTA_PAL_ImageState_Valid, and not the “PendingCommit” that is being checked for.

So is it checking for the wrong “ImageState”? Or is the value wrong after the processor reboots? Or something else?

Tom

Some additional information that might be helpful is the “log” just before it fails ( see below ) where it appears the downloaded JSON code is in “self_test” mode and the software tries to “Set image as testing!” even though later it is not detected.

I also believe the “error message” of “Job in self test but platform state is not!” which is from the prvInSelfTestHandler function is “turned around” where the “Platform state is in self test, but the job is not being detected as being in self test”. The prvInSelfTestHandler function is not even called unless the platform is in self test.

Again, thanks for any assistance you can give!

Tom

I (10061) ota_pal: prvPAL_SetPlatformImageState, 1
W (10061) ota_pal: Set image as testing!
29 1001 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ clientToken: 0:A12345-67891 ]
30 1001 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ timestamp: 1596247745 ]
31 1001 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ jobId: AFR_OTA-ota_Orange4 ]
32 1001 [OTA Agent Task] [prvParseJSONbyModel] Identified parameter [ self_test ]
33 1001 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ updatedBy: 1 ]
34 1001 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ protocols: [“MQTT”] ]
35 1001 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ streamname: AFR_OTA-093ea117-cf77-45d1-a9ba-9272cf22c0c6 ]
36 1001 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ filepath: / ]
37 1001 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ filesize: 860080 ]
38 1001 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ fileid: 0 ]
39 1001 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ certfile: C:\Program Files\OpenSSL-Win64\bin ]
40 1001 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ sig-sha256-ecdsa: MEQCIARFdJfRWDytEgzvBZ24bfg5CGk7… ]
41 1001 [OTA Agent Task] [prvParseJobDoc] In self test mode.
42 1001 [OTA Agent Task] [prvValidateUpdateVersion] The update version is newer than the version on device.
43 1001 [OTA Agent Task] [prvParseJobDoc] Setting image state to Testing for file ID 0
44 1001 [OTA Agent Task] [prvPublishStatusMessage] Msg: {“status”:“IN_PROGRESS”,“statusDetails”:{“self_test”:“active”,“updatedBy”:“0x3”}}
45 1003 [OTA Agent Task] [INFO ][MQTT][10030] (MQTT connection 0x3ffdabf4) MQTT PUBLISH operation queued.
46 1003 [OTA Agent Task] [INFO ][MQTT][10030] (MQTT connection 0x3ffdabf4, PUBLISH operation 0x3ffd1148) Waiting for operation completion.
47 1019 [OTA Agent Task] [INFO ][MQTT][10190] (MQTT connection 0x3ffdabf4, PUBLISH operation 0x3ffd1148) Wait complete with result SUCCESS.
I (10251) ota_pal: prvPAL_GetPlatformImageState
I (10251) esp_ota_ops: aws_esp_ota_get_boot_flags: 1
I (10261) esp_ota_ops: [0] aflags/seq:0x0/0x9, pflags/seq:0x0/0x8
I (10261) ota_pal: prvPAL_GetPlatformImageState
I (10271) esp_ota_ops: aws_esp_ota_get_boot_flags: 1
48 1019 [OTA Agent Task] [prvPublishStatusMessage] ‘IN_PROGRESS’ to $aws/things/A12345-67891/jobs/AFR_OTA-ota_Orange4/update
49 1020 [OTA Agent Task] [prvExecuteHandler] Called handler. Current State [WaitingForJob] Event [ReceivedJobDocument] New state [CreatingFile]
50 1020 [OTA Agent Task] [prvInSelfTestHandler] prvInSelfTestHandler, platform is in self-test.
51 1021 [OTA Agent Task] [prvStartSelfTest] PLATFORM STATE = 2
I (10291) esp_ota_ops: [0] aflags/seq:0x0/0x9, pflags/seq:0x0/0x8
I (10321) ota_pal: prvPAL_SetPlatformImageState, 3
W (10321) ota_pal: Set image as invalid!
I (10331) esp_ota_ops: aws_esp_ota_get_boot_flags: 1
I (10341) esp_ota_ops: [0] aflags/seq:0x0/0x9, pflags/seq:0x0/0x8
W (10341) ota_pal: Image not in self test mode 0
I (10351) esp_ota_ops: aws_esp_ota_get_boot_flags: 1
I (10351) esp_ota_ops: [0] aflags/seq:0x0/0x9, pflags/seq:0x0/0x8
52 1027 [OTA Agent Task] [prvInSelfTestHandler] Job in self test but platform state is not!

If you’re not able to recreate my issue, could you please show me a “log” of this successfully completing? That way I’ll have something to compare to to help determine where my code may be failing.

Tom

Hello Tom,

We could not reproduce this issue. Can you please confirm if you have any changes in your repository based on 202007 release?

The changes I’ve made are:

1 ) #define CONFIG_OTA_UPDATE_DEMO_ENABELD in aws_demo_config.h

2 ) Specifying MQTT ( not HTTP ) in aws_ota_agent_config.h

3 ) Specifying WiFi ( not BLE ) in aws_iot_network_config.h

4 ) Setting the WiFi’s SSID, password, security, and thing name in aws_credentials.h

5 ) Setting the “keys” in aws_credentials_keys.h

6 ) Re-defining the partition table from

image

to

image

for the 8MB WROVER module I’m using.

7 ) Setting the “code signing certificate” in aws_ota_codesigner_certificate.h.

8 ) Setting the “APP_VERSION” for both the “old” and “new” code builds.

Otherwise, I don’t believe I’ve made any changes.

BUT, I am calling the DEMO_RUNNER_RunDemos() function as shown below from a “setup()” routine that I will be using to merge my Arduino code into. Most of this code has been “cut and pasted” from the FreeRTOS 202007 demo code, but if you see anything I don’t have setup correctly, please let me know!

Tom

/* Logging Task Defines. */
#define mainLOGGING_MESSAGE_QUEUE_LENGTH ( 32 )
#define mainLOGGING_TASK_STACK_SIZE ( configMINIMAL_STACK_SIZE * 4 )
#define mainDEVICE_NICK_NAME “Espressif_Demo”

//#define configSUPPORT_STATIC_ALLOCATION 1

QueueHandle_t spp_uart_queue = NULL;

static void prvMiscInitialization( void );

// extern HardwareSerial Serial;
// HardwareSerial Serial(0);

extern “C” bool Iot_CreateDetachedThread( IotThreadRoutine_t threadRoutine,
void * pArgument,
int32_t priority,
size_t stackSize );

extern “C” BaseType_t xLoggingTaskInitialize( uint16_t usStackSize,
UBaseType_t uxPriority,
UBaseType_t uxQueueLength );

// extern “C” void runDemoTask( void * pArgument );

/* Can run ‘make menuconfig’ to choose the GPIO to blink,
or you can edit the following line and set a number here.
*/
#define CONFIG_BLINK_GPIO 33
#define BLINK_GPIO (gpio_num_t)CONFIG_BLINK_GPIO

#ifndef LED_BUILTIN
#define LED_BUILTIN 33 // 33 = GREEN, 32 = ORANGE
#endif

#if defined( democonfigOTA_UPDATE_TASK_STACK_SIZE )
#undef democonfigDEMO_STACKSIZE
#define democonfigDEMO_STACKSIZE democonfigOTA_UPDATE_TASK_STACK_SIZE
#endif
#if defined( democonfigOTA_UPDATE_TASK_TASK_PRIORITY )
#undef democonfigDEMO_PRIORITY
#define democonfigDEMO_PRIORITY democonfigOTA_UPDATE_TASK_TASK_PRIORITY
#endif

// Forward declaration of demo entry function to be renamed from #define in aws_demo_config.h /
int DEMO_entryFUNCTION( bool awsIotMqttMode,
const char * pIdentifier,
void * pNetworkServerInfo,
void * pNetworkCredentialInfo,
const IotNetworkInterface_t * pNetworkInterface );

/* Forward declaration of network connected DEMO callback to be renamed from #define in aws_demo_config.h */
#ifdef DEMO_networkConnectedCallback
void DEMO_networkConnectedCallback( bool awsIotMqttMode,
const char * pIdentifier,
void * pNetworkServerInfo,
void * pNetworkCredentialInfo,
const IotNetworkInterface_t * pNetworkInterface );
#else
#define DEMO_networkConnectedCallback ( NULL )
#endif

/* Forward declaration of network disconnected DEMO callback to be renamed from #define in aws_demo_config.h */
#ifdef DEMO_networkDisconnectedCallback
void DEMO_networkDisconnectedCallback( const IotNetworkInterface_t * pNetworkInterface );
#else
#define DEMO_networkDisconnectedCallback ( NULL )
#endif

// Copied from “…/amazon-freertos/vendors/espressif/esp-idf/components/esp32/freertos_hook.c”
// void IRAM_ATTR esp_vApplicationTickHook();
extern “C” void IRAM_ATTR esp_vApplicationTickHook();
extern “C” void esp_vApplicationIdleHook();

void setup() {
Serial.begin(115200);

pinMode(LED_BUILTIN, OUTPUT);


prvMiscInitialization();


if( SYSTEM_Init() == pdPASS )
	{
		vDevModeKeyProvisioning();
		
		ESP_ERROR_CHECK( esp_bt_controller_mem_release( ESP_BT_MODE_CLASSIC_BT ) );
    ESP_ERROR_CHECK( esp_bt_controller_mem_release( ESP_BT_MODE_BLE ) );

		DEMO_RUNNER_RunDemos();   		
	}

}

void loop() {
digitalWrite(LED_BUILTIN, !digitalRead(LED_BUILTIN));
Serial.println(“Hello, from loop!”);
delay(2500);
}

////////////////////////////////////////////////////////////////////////////////

#if CONFIG_FREERTOS_UNICORE
#define ARDUINO_RUNNING_CORE 0
#else
#define ARDUINO_RUNNING_CORE 1
#endif

void loopTask(void *pvParameters)
{
setup();
for(;:wink: {
// micros(); //update overflow
loop();
}
}

// int main()
// nt app_main(void)
extern “C” void app_main(void)
{
initArduino();
xTaskCreatePinnedToCore(loopTask, “loopTask”, 8192, NULL, 1, NULL, ARDUINO_RUNNING_CORE);
// return 0;
}

////////////////////////////////////////////////////////////////////////////////
// *

//------------------------------------------------------------//

extern void vApplicationIPInit( void );
static void prvMiscInitialization( void )
{
// Initialize NVS /
esp_err_t ret = nvs_flash_init();

if( ( ret == ESP_ERR_NVS_NO_FREE_PAGES ) || ( ret == ESP_ERR_NVS_NEW_VERSION_FOUND ) )
{
    ESP_ERROR_CHECK( nvs_flash_erase() );
    ret = nvs_flash_init();
}

ESP_ERROR_CHECK( ret );

// #if BLE_ENABLED
// NumericComparisonInit();
// spp_uart_init();
// #endif

// Create tasks that are not dependent on the WiFi being initialized. /

//extern “C” xLoggingTaskInitialize( mainLOGGING_TASK_STACK_SIZE,
xLoggingTaskInitialize( mainLOGGING_TASK_STACK_SIZE,
tskIDLE_PRIORITY + 5,
mainLOGGING_MESSAGE_QUEUE_LENGTH );

#if AFR_ESP_LWIP
// configPRINTF( (“Initializing lwIP TCP stack\r\n”) );
tcpip_adapter_init();
#else
// configPRINTF( (“Initializing FreeRTOS TCP stack\r\n”) );
vApplicationIPInit();
#endif
}

//-----------------------------------------------------------/

// extern void esp_vApplicationTickHook();
extern “C” void esp_vApplicationTickHook();
extern “C” void IRAM_ATTR vApplicationTickHook()
{
esp_vApplicationTickHook();
}

//-----------------------------------------------------------/
// extern void esp_vApplicationIdleHook();
extern “C” void esp_vApplicationIdleHook();
extern “C” void vApplicationIdleHook()
{
esp_vApplicationIdleHook();
}

//-----------------------------------------------------------/

// void vApplicationDaemonTaskStartupHook( void )
extern “C” void vApplicationDaemonTaskStartupHook( void )
{
}

#if !AFR_ESP_LWIP
//-----------------------------------------------------------/
// void vApplicationIPNetworkEventHook( eIPCallbackEvent_t eNetworkEvent )
extern “C” void vApplicationIPNetworkEventHook( eIPCallbackEvent_t eNetworkEvent )
{
uint32_t ulIPAddress, ulNetMask, ulGatewayAddress, ulDNSServerAddress;
system_event_t evt;

if( eNetworkEvent == eNetworkUp )
{
    // Print out the network configuration, which may have come from a DHCP
    //  server.
    FreeRTOS_GetAddressConfiguration(
        &ulIPAddress,
        &ulNetMask,
        &ulGatewayAddress,
        &ulDNSServerAddress );

    evt.event_id = SYSTEM_EVENT_STA_GOT_IP;
    evt.event_info.got_ip.ip_changed = true;
    evt.event_info.got_ip.ip_info.ip.addr = ulIPAddress;
    evt.event_info.got_ip.ip_info.netmask.addr = ulNetMask;
    evt.event_info.got_ip.ip_info.gw.addr = ulGatewayAddress;
    esp_event_send( &evt );
}

}
#endif

Hello Tom,

Thank you for sharing the changes you have made and as we could not reproduce this issue I suspect it could be due to one of these changes.

I think we should look at the custom partition table that you have defined. Can you try with default partition table ? or you can use the same configurations for nvs, otadata, phy_init as in default partition table and increase the size of ota_o and ota_1 partition for your platform without offsets as that will be added by espressif tool. After application partitions you can add any other partition required.

Please let me know how this works, meanwhile I will try to get 8 MB board for testing this.

I just noticed your reply and I’ll try what you recommend on Monday.

But I think I did find what’s causing my problem, but not sure why.

In my log shown above ( and copied below ) in the prvParseJobDoc function the software trys “Setting image state to Testing for file ID” ( as shown on the last line of this posting ) by using

( void ) prvSetImageStateWithReason( eOTA_ImageState_Testing, xErrVersionCheck );

which calls the following function

/* Call the platform specific code to set the image state. */
xErr = xOTA_Agent.xPALCallbacks.xSetPlatformImageState( xOTA_Agent.ulServerFileID, eState );

which is defined as prvPAL_SetPlatformImageState

if( pxCallbacks->xSetPlatformImageState != NULL )
{
    xOTA_Agent.xPALCallbacks.xSetPlatformImageState = pxCallbacks->xSetPlatformImageState;
}
else
{
    xOTA_Agent.xPALCallbacks.xSetPlatformImageState = prvPAL_DefaultSetPlatformImageState;
}

which for “eOTA_ImageState_Testing” does NOT set the flag as shown below.

    case eOTA_ImageState_Testing:
        ESP_LOGW( TAG, "Set image as testing!" );
        return kOTA_Err_None;

The flags stays at 0 and the following steps fail.

So how is this flag supposed to actually get set to “1” ( eOTA_ImageState_Testing )???

W (10061) ota_pal: Set image as testing!
29 1001 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ clientToken: 0:A12345-67891 ]
30 1001 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ timestamp: 1596247745 ]
31 1001 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ jobId: AFR_OTA-ota_Orange4 ]
32 1001 [OTA Agent Task] [prvParseJSONbyModel] Identified parameter [ self_test ]
33 1001 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ updatedBy: 1 ]
34 1001 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ protocols: [“MQTT”] ]
35 1001 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ streamname: AFR_OTA-093ea117-cf77-45d1-a9ba-9272cf22c0c6 ]
36 1001 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ filepath: / ]
37 1001 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ filesize: 860080 ]
38 1001 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ fileid: 0 ]
39 1001 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ certfile: C:\Program Files\OpenSSL-Win64\bin ]
40 1001 [OTA Agent Task] [prvParseJSONbyModel] Extracted parameter [ sig-sha256-ecdsa: MEQCIARFdJfRWDytEgzvBZ24bfg5CGk7… ]
41 1001 [OTA Agent Task] [prvParseJobDoc] In self test mode.
42 1001 [OTA Agent Task] [prvValidateUpdateVersion] The update version is newer than the version on device.
43 1001 [OTA Agent Task] [prvParseJobDoc] Setting image state to Testing for file ID 0

Sorry, it’s not defined as “prvPAL_SetPlatformImageState” but the prvPAL_DefaultSetPlatformImageState function it is defined as does call prvPAL_SetPlatformImageState.

static OTA_Err_t prvPAL_DefaultSetPlatformImageState( uint32_t ulServerFileID,
OTA_ImageState_t eState )
{
( void ) ulServerFileID;
( void ) eState;
return prvPAL_SetPlatformImageState( eState );
}

So I made the changes to the prvPAL_SetPlatformImageState function in aws_ota_pal.c to “force” the setting of the eOTA_ImageState_Testing mode and the OTA “works”.

So, as I previously asked, how is this flag “supposed” to get set to 1? I don’t see any code in the FreeRTOS demo code that actually does this. So what am I missing?

1 Like

So I was able to get it working without making the changes to my “custom” partition table shown above. So I’m curious whether changes to the partition table would possibly cause the “test mode” to not get set?

And what’s stored in the partition table’s “phy_init” that I don’t have in my table? I haven’t found any documentation or software that would show what it’s used for.

Tom, you are right. It worked for me. Thank you.

Currently, in “In self test mode”, boot flag is not written with “ESP_OTA_IMG_PENDING_VERIFY”, but in “Beginning self-test”, the status is read from boot flags, hence it was not matching and lead to “Rejecting new image and rebooting:The job is in the self-test state while the platform is not.”

I (14063) esp_ota_ops: aws_esp_ota_get_boot_flags: 1
I (14073) esp_ota_ops: [1] aflags/seq:0xffffffff/0x2, pflags/seq:0x2/0x1
E (14073) OTA: Platform is not in selftest 3
W (14083) OTA: Rejecting new image and rebooting:The job is in the self-test state while the platform is not.
I (14093) OTA: otaPal_SetPlatformImageState, 3
W (14093) OTA: Set image as invalid!
I (14103) esp_ota_ops: aws_esp_ota_get_boot_flags: 1
I (14113) esp_ota_ops: [1] aflags/seq:0xffffffff/0x2, pflags/seq:0x2/0x1
W (14113) OTA: Image not in self test mode -1

In case anybody comes across this as I did 2 years later, take a look at the bootloader Kconfig option help for BOOTLOADER_APP_ROLLBACK_ENABLE.
Project Configuration - ESP32 - — ESP-IDF Programming Guide latest documentation (espressif.com)

Also discussed here:
Do you NEED to enable rollback? - ESP32 Forum

Disabling this option “fixes” the issue.

2 Likes

Thank you for sharing your solution!

Your insights are invaluable, and we’re grateful for your willingness to share them.