Can We Use FreeRTOS OTA Library for Custom OTA Jobs?

[Jhon]

Hey everyone,

I’m working with FreeRTOS and AWS IoT and exploring the OTA (Over-the-Air) library that FreeRTOS provides. I know it’s designed to work with AWS IoT for managing firmware updates, but I was wondering:

Can we use this library for a custom OTA job within AWS IoT?
For example, if we want more control over update handling, scheduling, or validation, is there a way to customize the OTA process while still leveraging AWS IoT?

My goal is to create a simple OTA process—without any code signing—and stream the update directly through MQTT.

Has anyone tried modifying the OTA workflow while staying within AWS IoT? Any best practices or potential pitfalls?

Looking forward to your thoughts!

[kstribrn (Kody Stribrny)]

You absolutely can use a custom OTA job! We introduced the Modular OTA library (or more the set of libraries) for this exact purpose. These libraries make working with an OTA process simper by providing clean APIs for things such parsing job documents. You can find more on this here.

[Jhon]

I was actually referring to this demo:
FreeRTOS OTA Agent-Orchestrator Demo

But now I think I might be looking at the wrong demo, right? Since this one is based on the FreeRTOS OTA job, it seems tightly integrated with AWS IoT’s standard OTA process.

Do you have a demo or example of a custom OTA job that still uses the FreeRTOS OTA library but allows more flexibility (e.g., no code signing, MQTT streaming, etc.)?

[kstribrn (Kody Stribrny)]

That demo is supposed to more closely resemble our older OTA Agent library. I don’t recommend using that library or pattern as it can make extensibility tricky. Instead I’d probably start from the simple orchestrator demo and build up from there.

You would still use the same component libraries as the simple or OTA agent orchestrator demos.

[kstribrn (Kody Stribrny)]

I tested out the simple orchestrator with a custom document - it won’t handle it out of the gate (not a shocker) but it was able to get the job doc and decline it. Try that demo subbing in your custom job document and modify it as needed. This should be the quickest way to get started.

[Jhon]

@kstribrn

Try that demo subbing in your custom job document and modify it as needed.

Could you please guide me on how to do this and provide a custom job document?

I noticed that my custom job is being rejected at jobHandlerChain. Here’s the relevant code snippet:

bool otaDemo_handleIncomingMQTTMessage( char * topic,
                                        size_t topicLength,
                                        uint8_t * message,
                                        size_t messageLength )
{
    bool handled = false;
    int32_t fileId = 0;
    int32_t blockId = 0;
    int32_t blockSize = 0;

    handled = jobMetadataHandlerChain( topic, topicLength );

    if( !handled )
    {
        char thingName[ MAX_THING_NAME_SIZE + 1 ] = { 0 };
        size_t thingNameLength = 0U;

        mqttWrapper_getThingName( thingName, &thingNameLength );

        /*
         * AWS IoT Jobs library:
         * Checks if a message comes from the start-next/accepted reserved topic.
         */
        handled = Jobs_IsStartNextAccepted( topic, topicLength, thingName, thingNameLength );

        if( handled )
        {
            handled = jobHandlerChain( ( char * ) message, messageLength );
        }
        else
        {
            /*
             * MQTT streams Library:
             * Checks if the incoming message contains the requested data block. It is performed by
             * comparing the incoming MQTT message topic with MQTT streams topics.
             */
            handled = mqttDownloader_isDataBlockReceived( &mqttFileDownloaderContext,
                                                          topic,
                                                          topicLength );

    

[kstribrn (Kody Stribrny)]

The demo jobHandlerChain fails because the otaParser_parseJobDocFile function is specifically checking for a FreeRTOS OTA job (see this line). What you’ll need to do is replace the OTA parser in the job handler chain with your own your custom document parser.

[kstribrn (Kody Stribrny)]

I created a rough example you can find under my fork here which parses a custom job document. Further work would be required to actually process the document but that isn’t important for this example because you know best how to process your own job.

The job document below was generated by selecting the “Create custom job” option and using the “AWS-Download-File” template. I only used this template because I did not want to create my own. The Job Document shown in the console is…

{
  "version": "1.0",
  "steps": [
    {
      "action": {
        "name": "Download-File",
        "type": "runHandler",
        "input": {
          "handler": "download-file.sh",
          "args": [
            "https://en.wikipedia.org/wiki/%22Hello,_World!%22_program",
            "/"
          ],
          "path": ""
        },
        "runAsUser": ""
      }
    }
  ]
}

The example parses the version, name, and handler fields. There is a lot of hardcoding but this should give you a good starting point.

Example link

[Jhon]

I am using the Modular OTA library.
After setting the OTA PAL interfaces and OTA_Init(), everything looks fine before running the OTA_EventProcessingTask.

However, when this task runs, I’m getting a hard fault, and I see function pointers pointing to invalid addresses.

[kstribrn (Kody Stribrny)]

Looks like the link to the demo on the announcement page is broken. The demo you’ve referenced uses the old OTA agent library. The new, modular OTA demos can be found here.

The old OTA agent demo will not support your use case. Switching to the modular demo will allow you to modify the code more easily to handle you custom job.

[kstribrn (Kody Stribrny)]

I’ll fix this redirect and work to get the old OTA content removed from the website.

[Jhon]

Hey Kody Stribrny,

Never mind - I fixed the hard fault error. It was a mistake in my code.

However, the link to the demo on the announcement page appears to be broken.

As per your recommendation, I’m now using the new modular OTA library.

I find it a bit confusing, especially regarding OTA_Init().
Are all the fields in the OtaAppBuffer_t structure required for a custom OTA job?

For reference, here’s how I’m initializing the buffer:

/**
 * @brief The buffer passed to the OTA Agent from the application during initialization.
 */
static OtaAppBuffer_t ota_buffer =
{
    .pUpdateFilePath    = update_file_path,
    .updateFilePathsize = OTA_MAX_FILE_PATH_SIZE,
    .pCertFilePath      = cert_file_path,
    .certFilePathSize   = OTA_MAX_FILE_PATH_SIZE,
    .pDecodeMemory      = decode_mem,
    .decodeMemorySize   = config_OTA_FILE_BLOCK_SIZE,
    .pFileBitmap        = bitmap,
    .fileBitmapSize     = OTA_MAX_BLOCK_BITMAP_SIZE,
    .pUrl               = update_url,
    .urlSize            = OTA_MAX_URL_SIZE,
    .pAuthScheme        = auth_scheme,
    .authSchemeSize     = OTA_MAX_AUTH_SCHEME_SIZE
};

Are all of these fields strictly necessary, or can some be omitted for custom OTA implementations?

[kstribrn (Kody Stribrny)]

@Jhon sorry to say but you’re still using the old OTA agent based off the link you’ve provided. The ota-for-aws-iot-embedded-sdk Github library is the old OTA agent, not the modular OTA library.

Please see this page for the new modular OTA library demos.

[Jhon]

@kstribrn

{
  "version": "1.0",
  "steps": [
    {
      "action": {
        "name": "Download-File",
        "type": "runHandler",
        "input": {
          "handler": "download-file.sh",
          "args": [
            "https://en.wikipedia.org/wiki/%22Hello,_World!%22_program",
            "/"
          ],
          "path": ""
        },
        "runAsUser": ""
      }
    }
  ]
}

Thank you @kstribrn for your support. Could you please guide me on how to create an MQTT file streaming setup for a custom OTA job?

[kstribrn (Kody Stribrny)]

The simple-ota demo will be the best guide here. It’s important to remember that both the jobs and mqtt-streams libraries assemble the requests for you but don’t send the requests. To send the requests your code will need to call some sort of MQTT (or whatever messaging protocol you’re using) library which will send the assembled payloads. In the simple-ota demo the MQTT library used is CoreMQTT.

To get started you’ll need to:

1. Register a callback to handle an incoming MQTT message. In the simple demo this is done by creating the otaDemo_handleIncomingMQTTMessage function and registering it as part of the MQTT_Init function.
2. Establish the MQTT file stream context by calling mqttDownloader_init.

From here you can create whatever request/response handlers that you’d like. Your application will probably send the first message to as part of fetching the job through a stream. Your MQTT callback handler will then process the response (coming as an MQTT message) and orchestrate further requests for additional ‘blocks’ as needed.

[Jhon]

@kstribrn

Your MQTT callback handler will then process the response (coming as an MQTT message) and orchestrate further requests for additional ‘block

But doesn’t that require code signing? I needed to skip that for now.

[kstribrn (Kody Stribrny)]

No code signing is required since you’ll be creating a custom AWS IoT Job. Going without is probably fine for a short lived POC however it is strongly recommended to use code signing for longer term/production solutions.

Choosing an S3 file in the AWS IoT Jobs ‘Custom Job’ console workflow should show that code signing would be disabled for the custom job. An example of this is shown in the screenshot below.

[Jhon]

Thank you!

The firmware is downloading now.
But I have one more doubt:

The .bin file size before uploading to the S3 bucket was around 10 KB, but when I check it in S3, it’s somehow only 6 KB. Even when I download it back from S3, it’s still 6 KB.

Is it being compressed automatically?

[aggarg (Gaurav Aggarwal)]

@Jhon

The .bin file size before uploading to the S3 bucket was around 10 KB, but when I check it in S3, it’s somehow only 6 KB.

It should not happen. How did you upload to S3? Can you check again and ensure that the upload operation is not interrupted in the middle.

[Jhon]

My bad, it’s working now.
Thank you