My journey with FreeRTOS so far

Kernel
1

Hi Richard, AWS team & fellow FreeRTOS enthusiasts,

I have some questions after having journeyed with FreeRTOS for some months. Firstly, I would like to say it is an excellently designed system and one of the best APIs I have ever worked with.

My environment is as follows:

  • FreeRTOSv202406.01-LTS (Kernel v11.1.0)
  • MCUXpresso IDE v24.9 [Build 25] [2024-09-26]
  • NXP Arm Cortex-M7 (MKV58F1M0xxx24)

My requirement is to develop an extremely stable embedded application that needs to run for years unattended. Because of this, I chose static memory allocation for the whole application. In fact, it was instructed by my boss. The application has several comms peripherals:

  1. SPI → Wiznet W5500 → TCP, UDP
  2. I2C
  3. CAN
  4. CANopen
  5. UART
    Note: Only 1, 3, 4, 5 are coded in FreeRTOS at the moment

It also has the following tasks at the moment:

  1. UART user interface
  2. TCP user interface
  3. Error handling
  4. Error logging
  5. Data logging
  6. Supervisory control (decision making)
  7. Dynamic control (1 kHz PID control loop of a large servo motor)
    Note: Only 1 and 6 are coded in FreeRTOS at the moment

My problems with this setup basically involve:

  • Regular hanging of the entire application.
  • Sometimes if it runs, all functionality ceases and I’m forced to press “pause” on the IDE. Then a very long stack trace appears, typically ending in a SIGTRAP, with the running pointer pointing to a FreeRTOS file, often tasks.c. Recently it got stuck where the number of idle tasks are being compared to the number of cores. But they should both be 1. I set number of cores to 1 in FreeRTOSConfig.c, as per my MCU.

My observations are that even though MCUXpresso has “views” for FreeRTOS, e.g., a task list, it lists nothing while the application is running. Only after pressing “pause” does the IDE populate the task list, and all tasks are blocked except the idle task. But this scenario cannot be correct because the IDE just halted the whole MCU.

Having a tool like Percepio Tracealyzer would be great but our company is in South Africa, and we cannot afford it.

Is there a way to get this done with MCUXpresso alone? (I must also note that printf sometimes also hangs the system but at least I can print to the MCUXpresso console).

There are 3 things in FreeRTOS objects that I want to share my experiences with and ask your opinion about if you don’t mind:

  1. Tasks. I have basically tried periodic tasks and event-driven tasks (with a semaphore or queue unblocking it). In my experience, the event-driven tasks are way more stable and predictable. All my periodic tasks hang regularly. What am I doing wrong?
  2. Inter-task communications. I have had working queues and binary semaphores. But they also recently started hanging the whole application. I have given up on task notifications completely, I know they are great because they bypass the kernel, but I just couldn’t get it working. Am I doing something wrong here?
  3. Design philosophy. I started with many tasks as I think the FreeRTOS design philosophy is: each task performs only one – well – task. This is evidenced from my list of comms pheripherals and tasks above. However, I am now leaning towards the opposite side of the spectrum: as few tasks as possible. The reason for this is that I simply cannot get the inter-task communications to work. So it seems I am going to have a massive supervisory control task and then a few smaller ones. But this leans in the direction of bare-metal design and is against the FreeRTOS philosophy, right?

Note: I am careful to choose task priorities well and also assign enough stack memory wherever it is required.

I am sorry for the long post, but I am really at my wit’s end. I would really appreciate anyone’s advice. Maybe there is something wrong with my setup causing the application to hang regularly?

Thank you so much for reading this and any replies you might have.

2

Questions around your questions:

For #1 - how are you setting up your periodic tasks? It is possible that the lower priority periodic task isn’t getting time to execute as it’s starved by higher priority periodic tasks.

For #2 - What is the state of the queues and semaphores when lock up happens? The most basic case of hanging could occur if the queue was either empty or full and you have specified an indefinite wait time (aka portMAX_DELAY) for the read or write operation respectively.

For #3 - While FreeRTOS does advocate the use of tasks, there are reasons to create tasks sparingly. Actions such as context switches do cost some time so minimizing these can make your application more efficient.

Overall tasks exist to make your development easier. If you find having 5 tasks easier to model than 50 tasks, you’re probably not alone, and should go ahead with those fewer tasks. I’ve never personally worked on a project with more than maybe 10-20 tasks. If you have significantly more than that, it’s possible your tasks are too granular and context switching may be occurring very frequently.

1 Like
3

I’d suggest to comment out all the functionality and start adding one by one:

  1. Add first task and get it working.
  2. Then add another task and get both of them working.
  3. Get inter task communication working.
  4. Add remaining tasks one by one.

This step-by-step approach would enable to narrow down and debug your problem easily.

4
  1. Example of one of my event-driven tasks:
/* Task definitions */
#define RTC_TASK_PRIORITY        4
#define RTC_TASK_STACK_SIZE      ( configMINIMAL_STACK_SIZE )  
#define RTC_TASK_QUEUE_ITEM_SIZE ( sizeof(RTC_Time) )
#define RTC_TASK_QUEUE_LENGTH    ( 2 )

/* Declare the queue(s) */
StaticQueue_t xRTCTaskQueueBuffer1, xRTCTaskQueueBuffer2;
QueueHandle_t xRTCTaskQueue1, xRTCTaskQueue2;

/* Allocate memory for the queues' storage */
uint8_t ucRTCTaskQueueStorage1[RTC_TASK_QUEUE_LENGTH * RTC_TASK_QUEUE_ITEM_SIZE];

uint8_t ucRTCTaskQueueStorage2[RTC_TASK_QUEUE_LENGTH * RTC_TASK_QUEUE_ITEM_SIZE];

/* Task handle and buffer */
StaticTask_t xRTCTaskBuffer;
StackType_t xRTCTaskStack[RTC_TASK_STACK_SIZE];

/* Task function */
void vRTCTask(void *pvParameters) 
{
    TickType_t xLastWakeTime;
    uint32_t count;

    /* Task loop */
    for (;;) 
    {
        /* Wait for the next tick from outside FreeRTOS */
        if (rtcTaskTick) 
        {
            count++;
            rtcTaskTick = false;

            /* Send the updated time incl ten_ms to log tasks */
            BaseType_t currentTimeQToLogSent = xQueueSend(xRTCTaskQueue1,
                                                          &current_time, pdMS_TO_TICKS(5));
            //printf("current_time send to logging tasks\r\n");

            int prev_second = current_time->second;
            vUpdateCurrentTime();
            if (current_time->second > prev_second) 
            {
                current_time->ten_ms = 0; // Not used in UITask

                /* Send the updated time to UITask */
                BaseType_t currentTimeQToUISent = xQueueSend(xRTCTaskQueue2,
                                                             &current_time, pdMS_TO_TICKS(50));

                // gotoxy(37, 6);
                // printf("current_time sent to UITask\r\n");
                //
                // printf("%04d-%02d-%02d %02d:%02d:%02d\r\n",
                //    current_time->year, current_time->month, current_time->day, current_time->hour, current_time->minute, current_time->second);

            }
        }
    }
}

/* Initialization function */
void vInitRTCTask(void) 
{

    InitTimer0(16);
    vInitCurrentTime();

    /* Create the queues statically */
    xRTCTaskQueue1 = xQueueCreateStatic(RTC_TASK_QUEUE_LENGTH,
                                        RTC_TASK_QUEUE_ITEM_SIZE, ucRTCTaskQueueStorage1,
                                        &xRTCTaskQueueBuffer1);

    xRTCTaskQueue2 = xQueueCreateStatic(RTC_TASK_QUEUE_LENGTH,
                                        RTC_TASK_QUEUE_ITEM_SIZE, ucRTCTaskQueueStorage2,
                                        &xRTCTaskQueueBuffer2);

    if (xRTCTaskQueue1 == NULL) 
    { // || xRTCTaskQueue2 == NULL) {
        fprintf(stderr, "Failed to create the RTC task queues.\n");
        return;
    }

    // Create the task statically
    TaskHandle_t xRTCTaskHandle = xTaskCreateStatic(vRTCTask,   // Task function
                                                    "RTC Task",               // Task name
                                                    RTC_TASK_STACK_SIZE,       // Stack size
                                                    (void*) 1,                  // Task parameter
                                                    RTC_TASK_PRIORITY,        // Priority
                                                    xRTCTaskStack,            // Stack buffer
                                                    &xRTCTaskBuffer           // Task buffer
                                                    );

    if (xRTCTaskHandle == NULL) 
    {
        fprintf(stderr, "Failed to create the RTC task.\n");
    } else 
    {
        // printf("The RTC task was created successfully.\n");
    }
}

Example of one of my periodic tasks:

/* Task definitions */
#define CAN_TASK_PRIORITY        5
#define CAN_TASK_STACK_SIZE      (4 * configMINIMAL_STACK_SIZE) // 8192 bytes
#define CAN_TASK_QUEUE_ITEM_SIZE ( sizeof(uint8_t) )
#define CAN_TASK_QUEUE_LENGTH    ( 8 )

/* Declare the semaphore(s) */
StaticSemaphore_t xCANTaskSemaphoreBuffer1, xCANTaskSemaphoreBuffer2;
SemaphoreHandle_t xCANTaskSemaphore1, xCANTaskSemaphore2;

/* Declare the mutexes */
StaticSemaphore_t xCANTaskMutexBuffer1, xCANTaskMutexBuffer2;
SemaphoreHandle_t xCANTaskMutex1, xCANTaskMutex2;

/* Declare the queue(s) */
StaticQueue_t xCANTaskQueueBuffer1, xCANTaskQueueBuffer2;
QueueHandle_t xCANTaskQueue1, xCANTaskQueue2;

/* Allocate memory for the queues' storage */
uint8_t ucCANTaskQueueStorage1[CAN_TASK_QUEUE_LENGTH * CAN_TASK_QUEUE_ITEM_SIZE];
uint8_t ucCANTaskQueueStorage2[CAN_TASK_QUEUE_LENGTH * CAN_TASK_QUEUE_ITEM_SIZE];

/* Task handle and buffer for static task */
StaticTask_t xCANTaskBuffer;
StackType_t xCANTaskStack[CAN_TASK_STACK_SIZE];

/* Task function */
void vCANTask(void *pvParameters) 
{
    /* The parameter value is expected to be 1 as 1 is passed in the
     pvParameters value in the call to xTaskCreateStatic() below. */
    configASSERT(((uintptr_t )pvParameters) == 1);

    /* Task loop */
    for (;;) 
    {

        /* Wait for an event */
        uint32_t receivedValue;

        if (xTaskNotifyWait(0, 0xFFFFFFFF, &receivedValue, pdMS_TO_TICKS(20)) == pdPASS)
        {
            uint16_t message_id;
            flexcan_frame_t* frame;
            decodeNotificationData(receivedValue, &message_id, &frame);
            gotoxy(20, 30); printf("In vCANTask. message_id = %lu; frame address = 0x%08X", message_id, frame);

        }

        /* Test transmit */
        tx_mode_and_status_broadcast();
        vTaskDelay(pdMS_TO_TICKS(40));
    }
}

/* Initialization function */
void vInitCANTask(void) 
{
    /* Create the semaphores statically */
    xCANTaskSemaphore1 = xSemaphoreCreateBinaryStatic(
        &xCANTaskSemaphoreBuffer1);
    xCANTaskSemaphore2 = xSemaphoreCreateBinaryStatic(
        &xCANTaskSemaphoreBuffer2);

    if (xCANTaskSemaphore1 == NULL || xCANTaskSemaphore2 == NULL) 
    {
        fprintf(stderr, "Failed to create the CAN task semaphores.\r\n");
        return;
    }

    /* Create the mutexes statically */
    xCANTaskMutex1 = xSemaphoreCreateMutexStatic(&xCANTaskMutexBuffer1);
    xCANTaskMutex2 = xSemaphoreCreateMutexStatic(&xCANTaskMutexBuffer2);

    if (xCANTaskMutex1 == NULL || xCANTaskMutex2 == NULL) 
    {
        fprintf(stderr, "Failed to create the CAN task mutexes.\r\n");
        return;
    }

    /* Create the queues statically */
    xCANTaskQueue1 = xQueueCreateStatic(CAN_TASK_QUEUE_LENGTH,
                                        CAN_TASK_QUEUE_ITEM_SIZE, ucCANTaskQueueStorage1,
                                        &xCANTaskQueueBuffer1);

    xCANTaskQueue2 = xQueueCreateStatic(CAN_TASK_QUEUE_LENGTH,
                                        CAN_TASK_QUEUE_ITEM_SIZE, ucCANTaskQueueStorage2,
                                        &xCANTaskQueueBuffer2);

    if (xCANTaskQueue1 == NULL || xCANTaskQueue2 == NULL) 
    {
        fprintf(stderr, "Failed to create the CAN task queues.\r\n");
        return;
    }

    /* Create the task statically */
    TaskHandle_t xCANTaskHandle = xTaskCreateStatic
        (
            vCANTask,            // Task function
            "CAN Task",          // Task name
            CAN_TASK_STACK_SIZE, // Stack size
            (void*) 1,           // Task parameter
            CAN_TASK_PRIORITY,   // Priority
            xCANTaskStack,       // Stack buffer
            &xCANTaskBuffer      // Task buffer
        );

    configASSERT(xCANTaskHandle != NULL); 

    if (xCANTaskHandle == NULL) 
    {
        fprintf(stderr, "Failed to create the CAN task.\r\n");
    } else 
    {
        //printf("The CAN task was created successfully.\r\n");

        fprintf(stderr, "vInitCANTask called.\r\n");
        fprintf(stderr, "xCANTaskHandle: %p\n", xCANTaskHandle);

    }
}
  1. Regarding the block time, I never use portMAX_DELAY, I always specify a realistic value in
  2. Thank you for your advice on the number of tasks.
5

Hi Gaurav,

I commented out all my task initialisation functions except one:

vInitCANTask();

The code for this task is in my other reply. It starts to run okay, but then does nothing. When I pause MCUXpresso, I get the following stack trace:

rcu_sw JLink Debug [GDB SEGGER Interface Debugging]	
	rcu_sw.axf	
		Thread #1 57005 (Suspended : Signal : SIGTRAP:Trace/breakpoint trap)	
			prvIdleTask() at tasks.c:5,797 0x10001af0	
			Timer0Task_NestedDefaultTask() at timer0.c:137 0x10001778	
	arm-none-eabi-gdb (13.2.90.20231008)

The line in tasks.c referred to starts with:

            if( listCURRENT_LIST_LENGTH( &( pxReadyTasksLists[ tskIDLE_PRIORITY ] ) ) > ( UBaseType_t ) configNUMBER_OF_CORES )
            {
                taskYIELD();
            }
            else
            {
                mtCOVERAGE_TEST_MARKER();
            }

Of course I have configNUMBER_OF_CORES set to 1 in FreeRTOSConfig.h as per my MCU.

This error occurs with the others by themselves too. Other errors occur with the other tasks and inter-task communications too, all in isolation, as you suggested.

6

How do you verify that it does nothing? Can you put a breakpoint in it and see if it is running? If not, reduce the task definition to the following and examine in debugger if the counter variable is incrementing:

void Task( void * param )
{
    volatile uint32_t counter = 0;

    for( ;; )
    {
        counter += 1;

        vTaskDelay( pdMS_TO_TICKS( 1000 ) );
    }
}
7

If you don’t have configNUMBER_OF_CORES (in your case 1) ready tasks in the system, something has either corrupted your system, or you have an idle hook function that blocked, which isn’t allowed.

8

Hi Richard,

There is no vApplicationIdleHook(); in my system.

9

Hi Gaurav,

I implemented that simple task that you suggested. However, things are progressing from bad to worse. Here is the stack trace:

rcu_sw JLink Debug [GDB SEGGER Interface Debugging]	
	rcu_sw.axf	
		Thread #1 57005 (Suspended : Signal : SIGTRAP:Trace/breakpoint trap)	
			prvCreateStaticTask() at tasks.c:1,265 0x100074f0	
			xTaskCreateStatic() at tasks.c:1,322 0x10007558	
			prvCreateIdleTasks() at tasks.c:3,610 0x10001c90	
			vTaskStartScheduler() at tasks.c:3,679 0x10001cb2	
			main() at rcu_sw.c:1,283 0x1000165c	
	arm-none-eabi-gdb (13.2.90.20231008)

The last location in tasks.c is:

        configASSERT( puxStackBuffer != NULL );

As far as I know, my stack buffer is fine as shown in a previous listing.

10

That indicates that you likely ran out of heap to create the idle tasks.

11

Maybe I’m ignorant here, but I don’t think I need any heap. I have

#define configSUPPORT_DYNAMIC_ALLOCATION   0

in FreeRTOSConfig.h

Everything I do in FreeRTOS is done by means of static memory allocation, and my tasks and other other objects’ stack sizes are more than enough.

12

I guess you have implemented vApplicationGetIdleTaskMemory. Can you share its definition?

13

The problem is you have not provided a version of vApplicationGetIdleTaskMemory, so the default version still uses the heap to allocate the needed memory (I think).

14

My definition is as follows:

void vApplicationGetIdleTaskMemory(StaticTask_t** ppxIdleTaskTCBBuffer,
                                   StackType_t** ppxIdleTaskStackBuffer,
                                   uint32_t* pulIdleTaskStackSize)
{
    static StaticTask_t xIdleTaskTCB;
    static StackType_t xIdleStack[configMINIMAL_STACK_SIZE];

    *ppxIdleTaskTCBBuffer = &xIdleTaskTCB;
    *ppxIdleTaskStackBuffer = xIdleStack;
    *pulIdleTaskStackSize = configMINIMAL_STACK_SIZE;
}
15

Please try the following:

void vApplicationGetIdleTaskMemory(StaticTask_t** ppxIdleTaskTCBBuffer,
                                   StackType_t** ppxIdleTaskStackBuffer,
                                   uint32_t* pulIdleTaskStackSize)
{
    static StaticTask_t xIdleTaskTCB;
    static StackType_t xIdleStack[configMINIMAL_STACK_SIZE];

    *ppxIdleTaskTCBBuffer = &xIdleTaskTCB;
    *ppxIdleTaskStackBuffer = &( xIdleStack[ 0 ] );
    *pulIdleTaskStackSize = configMINIMAL_STACK_SIZE;
}

If the above does not work, you need to examine the call stack and see where puxStackBuffer is becoming NULL.

If static allocation is enabled, it is mandatory to either supply vApplicationGetIdleTaskMemory or set configKERNEL_PROVIDED_STATIC_MEMORY. The default version (provided when configKERNEL_PROVIDED_STATIC_MEMORY is set to 1) does not use heap- FreeRTOS-Kernel/tasks.c at main · FreeRTOS/FreeRTOS-Kernel · GitHub.

16

@aggarg I’m not sure how helpful this will be because we don’t use MCUXpresso. However, we have used FreeRTOS with static memory allocation, W5500 UART, I2C and CAN for many years (although we don’t use W5500 and CAN in the same build configuration) with Arm Cortex M4F and M7 MCUs. The project is open source and you are welcome to browse it (start from GitHub - Duet3D/RepRapFirmware: OO C++ RepRap Firmware).

Some observations that may be helpful:

  • We use configASSERT to keep assertions enabled. It’s easy in a complex system to call a FreeRTOS function that may suspend the calling task accidentally when the scheduler is locked or from an ISR.
  • We define configCHECK_FOR_STACK_OVERFLOW as 2 although unfortunately when we do get stack overflows, it often doesn’t trap them before the crash.
  • We have a fatal error handler that logs hard faults, memory access violations, assertion failures and stack overflows to flash memory when they happen, with the log including a partial stack trace. We have a diagnostic function to display those logged values after the MCU has reset. This makes it much easier to identify the cause of a crash when you don’t have a debugger attached.
  • If you use DMA then be very careful how it interacts with cache memory. Depending on the core revision of the Arm M7 MCU that you use, you may need to restrict which cache mode you use and you may need to restrict DMA transfers to/from non-cached memory.

HTH David

2 Likes
17

Thank you for sharing @dc42 !