Task stuck in ready state on cortex-M33

[zexalistic (Yihao Liu)]

Hi, I use freeRTOS kernel version 10.5.1 with Keil_v5 on Cortex M33.

My project has 3 static tasks and 1 timer task which is never run. Task 1 priority is 29, task 2 priority is 31, task 3 priority is 30. Task 1 and task 3 are for loops with osDelay 500 ticks and 100 ticks perspectively. Task 2 is a simple task waiting for the semaphore given from I2C slave interrupt via xSemaphoreGiveFromISR. Only receiving certain message will the semaphore be given. After taking the semaphore, task 2 will use another I2C port as a master and send/receive 4 bytes, then go back taking messages. This project is mainly used to relay data from PC via I2C slave port onto another device.

Everything works fine when I run the system to transmit data at the beginning. After transmitting thousands of bytes, somehow the task 2 will get stuck at ready state and never jump out. The system will get stuck randomly when sending/receiving bytes.

The attached graph is captured by the system analyzer of Keil v5. The tasks list from top to bottom are task1,2,3, idle task and Tmr Svc. Light blue is ready state, dark blue is running state and white is block state.

As seen from the graph, task 3 leaves running state and all task are in block state, then task 2 enters ready state and then never leaves. Each time the system get stuck, there will be such a task switching issue, which seems like the system fails to run idle task after task 3 leaves running state.

Does anyone have any idea why the stuck issue happens?

Thanks,
Yihao

[tony-josi-aws (Tony Josi)]

@zexalistic

Have you checked that the stacks of these tasks are not overflowing, especially for task 2? Could you try increasing the stack size and try to reproduce the issue?

You can also try enabling configCHECK_FOR_STACK_OVERFLOW if not already.

[aggarg (Gaurav Aggarwal)]

Please define configASSERT if not done already. When the system is stuck, can you break it in debugger and examine TCBs and task lists to see if you see any memory corruption.

[zexalistic (Yihao Liu)]

I set the total heap size as 102400, and set task 1,2,3 stack size as 4000. I believe the stask size is large enough. I also enabled check for stack overflow for method two.
It is strange that there’s no hardfault, and other tasks can switch normally.

[danielglasser]

@zexalistic

Everything works fine when I run the system to transmit data at the beginning. After transmitting thousands of bytes, somehow the task 2 will get stuck at ready state and never jump out. The system will get stuck randomly when sending/receiving bytes.

Under what conditions does task 2 go back to waiting on the semaphore?

From the description, the semaphore is not checked again until it’s finished relaying data from the inbound i2c buffer to the outbound I2c controller buffer. Without seeing the code, it’s hard to say for sure what’s going on, but if there is a boundary condition on the copy loop exit or out-bound I2C controller busy wait condition, you may be spinning without pending on anything.

How do you handle outbound i2c buffer-full conditions?

[zexalistic (Yihao Liu)]

Hi,

Thank you for your idea. I did check these things in my former test, but I don’t think the task is stuck at out-bound I2C busy wait condition
There’s no osDelay/vTaskDelay function within out-bound I2C r/w function. Each time I only send/receive 2 bytes(fix number) so I can avoid i2c buffer overflow condition. I use a while loop to check I2C ready state flag until the transmission is ready. When I stop in debug mode, the PC pointer is stopped at freertos task switching function, instead of that while loop. If the task is waiting for some busy bit, it should stop at the while loop.

Here’s how semaphore is transmitted:

Thanks,
Yihao

[danielglasser]

In “I2C2_EV_IRQHandler()”, I think you need:

1. Add a local variable with type BaseType_t, for instance, “xTaskWoken”; this will used to tell the kernel whether to call the scheduler when the IRQ handler exits
2. In the call to “xSemaphoreGiveFromISR()”, replace the second parameter with “&xTaskWoken”; if a task is waiting on the semaphore, “xTaskWoken” will be set to “pdTRUE”, otherwise it will be set to “pdFALSE”
3. Immediately after the call to “xSemaphoreGiveFromISR()”, add a call to “portYeild_FROM_ISR(xTaskWoken)”; this tells the scheduler whether to perform a context switch when the interrupt handler returns.

I don’t know if this will solve your problem, but I recommend trying it. In the latest “Mastering the FreeRTOS™ Real Time Kernel” (see New FreeRTOS Kernel book released), example 7.1 shows how this works and probably explains it better than I can.

[aggarg (Gaurav Aggarwal)]

Here is the code for @danielglasser’s suggestion:

BaseType_t xHigherPriorityTaskWoken = pdFALSE;
xSemaphoreGiveFromISR( cmis_msg.sem, &( xHigherPriorityTaskWoken ) );
portYIELD_FROM_ISR( xHigherPriorityTaskWoken );

[aggarg (Gaurav Aggarwal)]

Thank you for reporting back!

[zexalistic (Yihao Liu)]

Sorry, I did more test today and the issue appear again. But on the other side, at least I can see Freertos is trying to do the task switch for task with higher priority.

I have already add these code:
BaseType_t xHigherPriorityTaskWoken = pdFALSE;
xSemaphoreGiveFromISR( cmis_msg.sem, &( xHigherPriorityTaskWoken ) );
portYIELD_FROM_ISR( xHigherPriorityTaskWoken );

Here’s the new waveform captured by the system analyzer of Keil v5.

As we can see, task 2 preempt task 3 when task 3 running and task 2 is in ready state. However, task 2 fails to enter running state and then task 3 continue to run. Later the task 2 get stuck in ready state and other tasks can switch normally.

[danielglasser]

@zexalistic

As we can see, task 2 preempt task 3 when task 3 running and task 2 is in ready state. However, task 2 fails to enter running state and then task 3 continue to run. Later the task 2 get stuck in ready state and other tasks can switch normally.

I’m not sure what might cause this. Assuming task 2 has the highest priority, when it becomes ready, the FreeRTOS scheduler should perform the switch at the next scheduling event, which should be the invocation of portYEILD_FROM_ISR() in the interrupt handler. You could add a “taskYIELD()” in the main loops of tasks 1 and 3 and see if anything happens differently. It might be worthwhile examining the TCBs for all 3 tasks and make sure something hasn’t overwritten the priority field within task 2’s TCB.

Again, without the source code, I cannot diagnose the problem. Even with the source code (which I can understand if you can’t provide it), I might be unable to diagnose the problem, since I don’t have the Keil tools and at the moment cannot even bring an STM32 development board to my lab (or my desk). I can say that I’ve not encountered this behavior, and the most recent FreeRTOS project I had used 8 or 9 tasks (not counting the idle and timer tasks) of which 5 were high-priority I/O tasks, each servicing 1 peripheral type and blocking on events (rather than semaphores in my case) that were set from the ISRs for their peripherals.

[aggarg (Gaurav Aggarwal)]

@zexalistic

Later the task 2 get stuck in ready state and other tasks can switch normally.

In addition to the suggestions from @danielglasser, please also ensuer that configUSE_PREEMPTION and configUSE_TIME_SLICING are set to 1 in your FreeRTOSConfig.h.

[zexalistic (Yihao Liu)]

Thank you. I double checked, configUSE_PREEMPTION and configUSE_TIME_SLICING were all set to 1. And I define configUSE_PORT_OPTIMISED_TASK_SELECTION as 0

[zexalistic (Yihao Liu)]

I am checking TCB now. I find that uxTopReadyPriority is 0 when task 2 get stuck, although task 2’s priority in TCB is 31 and task 2 is in ready list. I am thinking about the reason why uxTopReadyPriority fails to be set as 31 when the last time task 2 is triggered.

When I manually set the value of uxTopReadyPriority as 31 and run, task 2 will be run and go back to block state.

[zexalistic (Yihao Liu)]

I manually changed the priority of IDLE task as 5 and repeat the stuck issue. Then I found uxTopReadyPriority was stuck at 5. It seems like uxTopReadyPriority was somehow overwritten by idle task…

[zexalistic (Yihao Liu)]

I think we have the same issue, and you are in that list

Quetions about the task scheduler

Do you have configASSERT() defined?

I am going to try Yueming’s solution next week

[aggarg (Gaurav Aggarwal)]

@zexalistic

It seems like uxTopReadyPriority was somehow overwritten by idle task…

Can you try to use data breakpoint to catch what is overwriting uxTopReadyPriority?

[zexalistic (Yihao Liu)]

Hi,

The cause of problem is exactly the same with

Quetions about the task scheduler

Hi Jeff, Here is my configASSERT() #define configASSERT(x) EFM_ASSERT(x) #define EFM_ASSERT(expr) ((expr) ? ((void)0) : assertEFM(__FILE__, __LINE__)) 66 void assertEFM(const char *file, int line) 67 { 68 (void)file; /* Unused parameter */ 69 (void)line; /* Unused parameter */ 70 71 while (true) { 72 } 73 } Thanks for your help! My freertos is "V10.4.3".

We both use Cortex-M33 core and here’s the implementation of sys tick irq:

void SysTick_Handler( void ) /* PRIVILEGED_FUNCTION /
{
uint32_t ulPreviousMask;
ulPreviousMask = portSET_INTERRUPT_MASK_FROM_ISR();
{
/ Increment the RTOS tick. /
if( xTaskIncrementTick() != pdFALSE )
{
/ Pend a context switch. */
portNVIC_INT_CTRL_REG = portNVIC_PENDSVSET_BIT;
}
}
portCLEAR_INTERRUPT_MASK_FROM_ISR( ulPreviousMask );
}

Although Sys tick irq suspends task scheduler, it is not run under critical section which can be interrupted by a higher IRQ(i2c irq). The problem is exactly the same as described by Yueming:

configUSE_PORT_OPTIMISED_TASK_SELECTION is 0 then task selection is performed in a generic way. There might be a racing condition on uxTopReadyPriority.
Please check this failed case:

1. Tick IRQ is a low priority IRQ.
2. When Tick IRQ finished this line “UBaseType_t uxTopPriority = uxTopReadyPriority;” , let’s assume uxTopReadyPriority and uxTopPriority is 5 now.
   {
   ==> UBaseType_t uxTopPriority = uxTopReadyPriority;
   /* Find the highest priority queue that contains ready tasks. /
   while( listLIST_IS_EMPTY( &( pxReadyTasksLists[ uxTopPriority ] ) ) )
   {
   configASSERT( uxTopPriority );
   –uxTopPriority;
   }
   / listGET_OWNER_OF_NEXT_ENTRY indexes through the list, so the tasks of
3. the same priority get an equal share of the processor time. /
   listGET_OWNER_OF_NEXT_ENTRY( pxCurrentTCB, &( pxReadyTasksLists[ uxTopPriority ] ) );
   uxTopReadyPriority = uxTopPriority;
   } / taskSELECT_HIGHEST_PRIORITY_TASK */
4. A higher priority IRQ occurs, tick IRQ is switched out. The higher priority IRQ adds a ready task into the ready list and sets uxTopReadyPriority to this task’s priority, let’s assume it is 6.
5. The higher priority IRQ finished and tick IRQ contine it’s code. uxTopReadyPriority will be set to a value <= 5.
6. The task scheduler cannot call priority 6 task on next tick IRQ. That would cause some issues.

uxTopReadyPriority is written twice and the last time it is written by tick IRQ and set as the priority of idle task. If I use taskSELECT_HIGHEST_PRIORITY_TASK(), the task scheduler will only check the ready list for priority 0(idle task).Unless there is another task trigger event that rewrite uxTopReadyPriority back to the overridden value, the task will always be stuck in ready list and never run.

Therefore, my solution is to abandon uxTopReadyPriority and taskSELECT_HIGHEST_PRIORITY_TASK(). I instead search all the ready list. Since there are not too many tasks, the cost is affordable. And the porblem is solved in this way.

[jefftenney (Jeff)]

@zexalistic

4. A higher priority IRQ occurs, tick IRQ is switched out. The higher priority IRQ adds a ready task into the ready list

This is impossible with official FreeRTOS code. At the beginning of SysTick_Handler(), portSET_INTERRUPT_MASK_FROM_ISR() masks all interrupts (even higher priority interrupts) that are allowed to make FreeRTOS API calls. As a result, the following statement isn’t right:

@zexalistic

Although Sys tick irq suspends task scheduler, it is not run under critical section which can be interrupted by a higher IRQ(i2c irq).

Can you post your FreeRTOSConfig.h?

[zexalistic (Yihao Liu)]

Here’s the FreeRTOSConfig.h(Sorry I can not upload files, so I paste it here)

/* --------------------------------------------------------------------------
 * Copyright (c) 2013-2022 Arm Limited. All rights reserved.
 *
 * SPDX-License-Identifier: Apache-2.0
 *
 * Licensed under the Apache License, Version 2.0 (the License); you may
 * not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 * www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an AS IS BASIS, WITHOUT
 * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 *
 * --------------------------------------------------------------------------
 *
 * $Revision:   V10.4.0
 *
 * Project:     CMSIS-FreeRTOS
 * Title:       FreeRTOS configuration definitions
 *
 * --------------------------------------------------------------------------*/

#ifndef FREERTOS_CONFIG_H
#define FREERTOS_CONFIG_H

/*-----------------------------------------------------------
 * Application specific definitions.
 *
 * These definitions should be adjusted for your particular hardware and
 * application requirements.
 *
 * THESE PARAMETERS ARE DESCRIBED WITHIN THE 'CONFIGURATION' SECTION OF THE
 * FreeRTOS API DOCUMENTATION AVAILABLE ON THE FreeRTOS.org WEB SITE.
 *
 * See http://www.freertos.org/a00110.html
 *----------------------------------------------------------*/

#if (defined(__ARMCC_VERSION) || defined(__GNUC__) || defined(__ICCARM__))
#include <stdint.h>

#include "RTE_Components.h"
#include CMSIS_device_header
#endif

//-------- <<< Use Configuration Wizard in Context Menu >>> --------------------

//  <o>Minimal stack size [words] <0-65535>
//  <i> Stack for idle task and default task stack in words.
//  <i> Default: 128
#define configMINIMAL_STACK_SIZE                ((uint16_t)(128))

//  <o>Total heap size [bytes] <0-0xFFFFFFFF>
//  <i> Heap memory size in bytes.
//  <i> Default: 8192
#define configTOTAL_HEAP_SIZE                   ((size_t)102400)

//  <o>Kernel tick frequency [Hz] <0-0xFFFFFFFF>
//  <i> Kernel tick rate in Hz.
//  <i> Default: 1000
#define configTICK_RATE_HZ                      ((TickType_t)1000)

//  <o>Timer task stack depth [words] <0-65535>
//  <i> Stack for timer task in words.
//  <i> Default: 80
#define configTIMER_TASK_STACK_DEPTH            1024

//  <o>Timer task priority <0-56>
//  <i> Timer task priority.
//  <i> Default: 40 (High)
#define configTIMER_TASK_PRIORITY               40

//  <o>Timer queue length <0-1024>
//  <i> Timer command queue length.
//  <i> Default: 5
#define configTIMER_QUEUE_LENGTH                10

//  <o>Preemption interrupt priority
//  <i> Maximum priority of interrupts that are safe to call FreeRTOS API.
//  <i> Default: 16
#define configMAX_SYSCALL_INTERRUPT_PRIORITY    16

//  <q>Use time slicing
//  <i> Enable setting to use timeslicing.
//  <i> Default: 1
#define configUSE_TIME_SLICING                  1

//  <q>Use tickless idle
//  <i> Enable low power tickless mode to stop the periodic tick interrupt during idle periods or
//  <i> disable it to keep the tick interrupt running at all times.
//  <i> Default: 0
#define configUSE_TICKLESS_IDLE                 0

//  <q>Idle should yield
//  <i> Control Yield behaviour of the idle task.
//  <i> Default: 1
#define configIDLE_SHOULD_YIELD                 1

//  <o>Check for stack overflow
//    <0=>Disable <1=>Method one <2=>Method two
//  <i> Enable or disable stack overflow checking.
//  <i> Callback function vApplicationStackOverflowHook implementation is required when stack checking is enabled.
//  <i> Default: 0
#define configCHECK_FOR_STACK_OVERFLOW          2

//  <q>Use idle hook
//  <i> Enable callback function call on each idle task iteration.
//  <i> Callback function vApplicationIdleHook implementation is required when idle hook is enabled.
//  <i> Default: 0
#define configUSE_IDLE_HOOK                     0

//  <q>Use tick hook
//  <i> Enable callback function call during each tick interrupt.
//  <i> Callback function vApplicationTickHook implementation is required when tick hook is enabled.
//  <i> Default: 0
#define configUSE_TICK_HOOK                     0

//  <q>Use deamon task startup hook
//  <i> Enable callback function call when timer service starts.
//  <i> Callback function vApplicationDaemonTaskStartupHook implementation is required when deamon task startup hook is enabled.
//  <i> Default: 0
#define configUSE_DAEMON_TASK_STARTUP_HOOK      0

//  <q>Use malloc failed hook
//  <i> Enable callback function call when out of dynamic memory.
//  <i> Callback function vApplicationMallocFailedHook implementation is required when malloc failed hook is enabled.
//  <i> Default: 0
#define configUSE_MALLOC_FAILED_HOOK            0

//  <o>Queue registry size
//  <i> Define maximum number of queue objects registered for debug purposes.
//  <i> The queue registry is used by kernel aware debuggers to locate queue and semaphore structures and display associated text names.
//  <i> Default: 0
#define configQUEUE_REGISTRY_SIZE               0

// <h>Event Recorder configuration
//  <i> Initialize and setup Event Recorder level filtering.
//  <i> Settings have no effect when Event Recorder is not present.

//  <q>Initialize Event Recorder
//  <i> Initialize Event Recorder before FreeRTOS kernel start.
//  <i> Default: 1
#define configEVR_INITIALIZE                    1

//  <e>Setup recording level filter
//  <i> Enable configuration of FreeRTOS events recording level
//  <i> Default: 1
#define configEVR_SETUP_LEVEL                   1

//  <o>Tasks functions
//  <i> Define event recording level bitmask for events generated from Tasks functions.
//  <i> Default: 0x05
//    <0x00=>Off <0x01=>Errors <0x05=>Errors + Operation <0x0F=>All
#define configEVR_LEVEL_TASKS                   0x05

//  <o>Queue functions
//  <i> Define event recording level bitmask for events generated from Queue functions.
//  <i> Default: 0x05
//    <0x00=>Off <0x01=>Errors <0x05=>Errors + Operation <0x0F=>All
#define configEVR_LEVEL_QUEUE                   0x05

//  <o>Timer functions
//  <i> Define event recording level bitmask for events generated from Timer functions.
//  <i> Default: 0x05
//    <0x00=>Off <0x01=>Errors <0x05=>Errors + Operation <0x0F=>All
#define configEVR_LEVEL_TIMERS                  0x05

//  <o>Event Groups functions
//  <i> Define event recording level bitmask for events generated from Event Groups functions.
//  <i> Default: 0x05
//    <0x00=>Off <0x01=>Errors <0x05=>Errors + Operation <0x0F=>All
#define configEVR_LEVEL_EVENTGROUPS             0x05

//  <o>Heap functions
//  <i> Define event recording level bitmask for events generated from Heap functions.
//  <i> Default: 0x05
//    <0x00=>Off <0x01=>Errors <0x05=>Errors + Operation <0x0F=>All
#define configEVR_LEVEL_HEAP                    0x05

//  <o>Stream Buffer functions
//  <i> Define event recording level bitmask for events generated from Stream Buffer functions.
//  <i> Default: 0x05
//    <0x00=>Off <0x01=>Errors <0x05=>Errors + Operation <0x0F=>All
#define configEVR_LEVEL_STREAMBUFFER            0x05
//  </e>
// </h>

// <h> Port Specific Features
// <i> Enable and configure port specific features.
// <i> Check FreeRTOS documentation for definitions that apply for the used port.

//  <q>Use Floating Point Unit
//  <i> Using Floating Point Unit (FPU) affects context handling.
//  <i> Enable FPU when application uses floating point operations.
//  <i> Default: 1
#define configENABLE_FPU                      1

//  <q>Use M-Profile Vector Extension
//  <i> Using M-Profile Vector Extension (MVE) affects context handling.
//  <i> Enable MVE when application uses signal processing and ML algorithms.
//  <i> Default: 0
#define configENABLE_MVE                      0

//  <q>Use Memory Protection Unit
//  <i> Using Memory Protection Unit (MPU) requires detailed memory map definition.
//  <i> This setting is only releavant for MPU enabled ports.
//  <i> Default: 0
#define configENABLE_MPU                      0

//  <q> Use TrustZone Secure Side Only
//  <i> This settings prevents FreeRTOS contex switch to Non-Secure side.
//  <i> Enable this setting when FreeRTOS runs on the Secure side only.
#define configRUN_FREERTOS_SECURE_ONLY        0

//  <q>Use TrustZone Security Extension
//  <i> Using TrustZone affects context handling.
//  <i> Enable TrustZone when FreeRTOS runs on the Non-Secure side and calls functions from the Secure side.
//  <i> Default: 1
#define configENABLE_TRUSTZONE               0 //    hy 9/27   1

//  <o>Minimal secure stack size [words] <0-65535>
//  <i> Stack for idle task Secure side context in words.
//  <i> This setting is only relevant when TrustZone extension is enabled.
//  <i> Default: 128
#define configMINIMAL_SECURE_STACK_SIZE       ((uint32_t)128)
// </h>

//------------- <<< end of configuration section >>> ---------------------------

/* Defines needed by FreeRTOS to implement CMSIS RTOS2 API. Do not change! */
#define configCPU_CLOCK_HZ                      (SystemCoreClock)
#define configSUPPORT_STATIC_ALLOCATION         1
#define configSUPPORT_DYNAMIC_ALLOCATION        1
#define configUSE_PREEMPTION                    1
#define configUSE_TIMERS                        1
#define configUSE_MUTEXES                       1
#define configUSE_RECURSIVE_MUTEXES             1
#define configUSE_COUNTING_SEMAPHORES           1
#define configUSE_TASK_NOTIFICATIONS            1
#define configUSE_TRACE_FACILITY                1
#define configUSE_16_BIT_TICKS                  0
#define configUSE_PORT_OPTIMISED_TASK_SELECTION 0
#define configMAX_PRIORITIES                    56
#define configKERNEL_INTERRUPT_PRIORITY         255

/* Defines that include FreeRTOS functions which implement CMSIS RTOS2 API. Do not change! */
#define INCLUDE_xEventGroupSetBitsFromISR       1
#define INCLUDE_xSemaphoreGetMutexHolder        1
#define INCLUDE_vTaskDelay                      1
#define INCLUDE_xTaskDelayUntil                 1
#define INCLUDE_vTaskDelete                     1
#define INCLUDE_xTaskGetCurrentTaskHandle       1
#define INCLUDE_xTaskGetSchedulerState          1
#define INCLUDE_uxTaskGetStackHighWaterMark     1
#define INCLUDE_uxTaskPriorityGet               1
#define INCLUDE_vTaskPrioritySet                1
#define INCLUDE_eTaskGetState                   1
#define INCLUDE_vTaskSuspend                    1
#define INCLUDE_xTimerPendFunctionCall          1

/* Map the FreeRTOS port interrupt handlers to their CMSIS standard names. */
#define xPortPendSVHandler                      PendSV_Handler
#define vPortSVCHandler                         SVC_Handler

/* Ensure Cortex-M port compatibility. */
#define SysTick_Handler                         xPortSysTickHandler

#if (defined(__ARMCC_VERSION) || defined(__GNUC__) || defined(__ICCARM__))
/* Include debug event definitions */
#include "freertos_evr.h"
#endif

#endif /* FREERTOS_CONFIG_H */