ESP32 OTA "Job in self test but platform state is not!"

FreeOurToes wrote on December 13, 2018:

Hi,

I have been trying OTA for ESP32. Everything sets up correctly. I receive OTA all blocks, device restarts and starts running for 5-10 seconds and then in the middle of it, it reboots to previous fw version.

The only detail I am able to get out of debugging is that platform is not in self-test mode.

30 746 [OTA Task] [prvOTAUpdateTask] Job in self test but platform state is not!

In aws_ota_agent.c comments says that it could be an attack platform image state.
But not sure what to make of it…

thanks


11 716 [OTA Task] [prvSubscribeToJobNotificationTopics] OK: $aws/things/esp32_tests_gg/jobs/$next/get/accepted
I (7368) ota_pal: prvPAL_GetPlatformImageState
I (7368) esp_ota_ops: aws_esp_ota_get_boot_flags: 1
I (7368) esp_ota_ops: [0] aflags/seq:0x3/0x2, pflags/seq:0x1/0x2
I (7378) ota_pal: prvPAL_GetPlatformImageState
I (7378) esp_ota_ops: aws_esp_ota_get_boot_flags: 1
I (7388) esp_ota_ops: [0] aflags/seq:0x3/0x2, pflags/seq:0x1/0x2
12 722 [OTA Task] [prvSubscribeToJobNotificationTopics] OK: $aws/things/esp32_tests_gg/jobs/notify-next
13 725 [OTA Task] [OTA_CheckForUpdate] Request #0
I (7458) ota_pal: prvPAL_SetPlatformImageState, 1
W (7458) ota_pal: Set image as testing!
14 731 [OTA Task] [prvParseJSONbyModel] Extracted parameter [ clientToken: 0:esp32_tests_gg ]
15 731 [OTA Task] [prvParseJSONbyModel] Extracted parameter [ jobId: AFR_OTA-esp32_test_9 ]
16 731 [OTA Task] [prvParseJSONbyModel] Identified parameter [ self_test ]
17 731 [OTA Task] [prvParseJSONbyModel] Extracted parameter [ updatedBy: 655374 ]
18 731 [OTA Task] [prvParseJSONbyModel] Extracted parameter [ streamname: AFR_OTA-### ]
19 731 [OTA Task] [prvParseJSONbyModel] Extracted parameter [ filepath: /device/updates ]
20 731 [OTA Task] [prvParseJSONbyModel] Extracted parameter [ filesize: 626064 ]
21 731 [OTA Task] [prvParseJSONbyModel] Extracted parameter [ fileid: 0 ]
22 731 [OTA Task] [prvParseJSONbyModel] Extracted parameter [ certfile: aws_ota_codesigner_certificate.h ]
23 731 [OTA Task] [prvParseJSONbyModel] Extracted parameter [ sig-sha256-ecdsa: ###... ]
24 731 [OTA Task] [prvParseJobDoc] In self test mode.
25 731 [OTA Task] [prvUpdateJobStatus] Msg: {"status":"IN_PROGRESS","statusDetails":{"self_test":"active","updatedBy":"0xa000f"}}
26 732 [OTA] [OTA_AgentInit] Ready.
27 736 [OTA Task] [prvUpdateJobStatus] 'IN_PROGRESS' to $aws/things/esp32_tests_gg/jobs/AFR_OTA-esp32_test_9/update
28 736 [OTA Task] [prvOTA_Close] Context->0x3ffbb520
29 742 [OTA Task] [prvUnSubscribeFromDataStream] OK: $aws/things/esp32_tests_gg/streams/AFR_OTA-1##/data/cbor
I (7568) ota_pal: prvPAL_GetPlatformImageState
I (7598) esp_ota_ops: aws_esp_ota_get_boot_flags: 1
I (7598) esp_ota_ops: [0] aflags/seq:0x3/0x2, pflags/seq:0x1/0x2
I (7608) ota_pal: prvPAL_SetPlatformImageState, 3
W (7618) ota_pal: Set image as invalid!
I (7618) esp_ota_ops: aws_esp_ota_get_boot_flags: 1
I (7628) esp_ota_ops: [0] aflags/seq:0x3/0x2, pflags/seq:0x1/0x2
W (7628) ota_pal: Image not in self test mode 3
I (7638) esp_ota_ops: aws_esp_ota_get_boot_flags: 1
I (7638) esp_ota_ops: [0] aflags/seq:0x3/0x2, pflags/seq:0x1/0x2
30 746 [OTA Task] [prvOTAUpdateTask] Job in self test but platform state is not!
31 750 [OTA Task] [prvUpdateJobStatus] Msg: {"status":"FAILED","statusDetails":{"reason":"rejected: 0x26000000"}}
32 755 [OTA Task] [prvUpdateJobStatus] 'FAILED' to $aws/things/esp32_tests_gg/jobs/AFR_OTA-esp32_test_9/update
33 755 [OTA Task] [prvResetDevice] Attempting forced reset of device...
I (8198) wifi: state: run -> init (0)
I (8198) wifi: n:1 0, o:1 0, ap:255 255, sta:1 0, prof:1
I (8198) wifi: pm stop, total sleep time: 0/3802069

I (8238) wifi: flush txq
I (8238) wifi: stop sw txq
I (8238) wifi: lmac stop hw txq
I (8238) WIFI: SYSTEM_EVENT_STA_DISCONNECTED
ets Jun  8 2016 00:22:57


gordon-AWS wrote on December 14, 2018:

Hello,

It looks like the OTA image self-test fails, which causes a rollback.

Can you check that the OTA image you are downloading has a version number greater than the image on your device?

Also, which version of Amazon FreeRTOS are you using? We have recently made some fixes to improve the reliability of OTA, so please make sure you are using the latest version.

FreeOurToes wrote on December 14, 2018:

Thanks, I managed to have a successful OTA.

Ok, So today I downloaded the newest version from github.

Configured credentials. did,

make erase_flash

then

make flash monitor -j9

. incremented the build number in fw version.

make all

Uploaded newest bin, did all jazz in the console and it worked.

But I am sure I was increment-ing build version before, because I did like 10 tries of different things and and code I was using was from this week too.
But I noticed one difference.
So after some unsuccessful tries (few things that I was changing were “Pathname of firmware image on device” field in ota from example /device/update to just having “/”.)
In debugging window I started to have an error about partitions.

E (108) boot: ota data partition invalid and no factory, will try all partitions
 

Even after flashing new firmware this error existed, but after doing make erase_flash, and then flashing new firmware it went away and OTA was success.


I (30) boot: ESP-IDF v3.1-dev-322-gf307f41-dirty 2nd stage bootloader
I (30) boot: compile time 10:47:09
I (31) boot: Enabling RNG early entropy source...
I (36) boot: SPI Speed      : 40MHz
I (40) boot: SPI Mode       : DIO
I (44) boot: SPI Flash Size : 4MB
I (48) boot: Partition Table:
I (52) boot: ## Label            Usage          Type ST Offset   Length
I (59) boot:  0 nvs              WiFi data        01 02 00010000 00006000
I (67) boot:  1 otadata          OTA data         01 00 00016000 00002000
I (74) boot:  2 phy_init         RF data          01 01 00018000 00001000
I (82) boot:  3 ota_0            OTA app          00 10 00020000 00100000
I (89) boot:  4 ota_1            OTA app          00 11 00120000 00100000
I (97) boot:  5 storage          Unknown data     01 82 00220000 00010000
I (104) boot: End of partition table
E (108) boot: ota data partition invalid and no factory, will try all partitions
I (116) esp_image: segment 0: paddr=0x00020020 vaddr=0x3f400020 size=0x15a44 ( 88644) map
I (156) esp_image: segment 1: paddr=0x00035a6c vaddr=0x3ffb0000 size=0x02454 (  9300) load
I (160) esp_image: segment 2: paddr=0x00037ec8 vaddr=0x40080000 size=0x00400 (  1024) load


couper321 wrote on April 04, 2019:

Hey there, I am having the similar issue:

29 1007 Job in self test but platform state is not!
30 1011 Msg: {“status”:“FAILED”,“statusDetails”:{“reason”:“rejected: 0x26000000”}}

But on boot, the firmware seems to have updated (the printed out version number increases) except this job reports back a failure into the AWS console, did you end up solving this or having this as an effect?

Edited by: couper321 on Apr 3, 2019 6:22 PM

embeddedx wrote on May 08, 2019:

Yes, I have the same behavior, any solutions?

[prvOTAUpdateTask] Job in self test but platform state is not!
40 1318 [OTA Task] [prvUpdateJobStatus] Msg: {“status”:“FAILED”,“statusDetails”:{“reason”:“rejected: 0x26000000”}}
41 1332 [OTA] State: Ready Received: 1 Queued: 1 Processed: 0 Dropped: 0

TiglathIII wrote on June 05, 2019:

Hi @couper321 it looks like you are exactly 2 months ahead of me. Having got past the connecting issues I then hit this issue that I have now solved thanks in part to @FreeOurToes plus one other obscure change hinted at being needed for another device.

So, to re-cap I did the following:

  1. Downloaded the latest Amazon FreeRTOS firmware (as of today 5th June).
  2. Erased my ESP32 device and re-imaged it with this new firmware, checking that it waits for OTA update jobs.
  3. Incremented the APP_VERSION_BUILD variable as directed.
  4. Then there was a little note for the Microchip Curiosity PIC32MZEF device to also increment the OTA sequence number in file \demos\common\ota\bootloader\utility\user-config\ota-descriptor.config. I found I had to do this for my ESP32 device also, otherwise I seemed to always get the previous version firmware downloaded.
  5. Built the new aws_demos.bin binary image, Uploaded to my S3 bucket, created a new FreeRTOS OTA job, being careful to create a new code signed image from the newly uploaded binary image.
  6. Ran make monitor to see what was happening on the ESP32 device.

Hope this helps :slight_smile:

Edited by: TiglathIII on Jun 5, 2019 3:15 PM