Massive stack overflow on task when kernel starts. Any ideas?

This is my environment:

  • FPGA with Arm M33 based build (custom implementation, right now just UART and DMA, basic interrupts are defined). Core clock is 20 Mhz right now
  • JLINK mini jtag debugger
  • Developing using CMSIS v2 with FreeRTOS inside Keil

Currently, I have a very basic implementation just for testing,

int main(void){
	SystemCoreClockUpdate();
	//xTaskCreate(task_uart, "UART Test", 1024, NULL, 50, NULL);

	vTaskStartScheduler();
	while(1);
}

Please note, I am debugging with the JLINK at 20 Mhz as well, I run to main, then step through the code above, I also just let it run without step through, either way this overflow happens. The kernel starts properly and reaches the point it should (never returning).

So the problem, when the kernel starts, the tick count never iterates in the RTOS Kernel view and the running task shows a massive stack overflow. The task shows “running” but never actually executes (the PC never gets to that thread). This happens with all tasks, I tried with a test UART task as highest priority that had very minimal code in it, still a massive stack overflow and PC pointing to wrong place. I have tried increasing the tasks stack size to very large amounts, still no luck. I am sure that the heap and stack allocation are large enough, RAM space is huge and the idle task/timer tasks defined by FreeRTOS have more than enough stack space.

I have manually read from all the NVIC pending registers to see if some unhandled interrupt when causing the CPU to hang but none are pending. I verified the SysTick is working as well through JLINK commander. Is there something I am missing here? Anyone have experience with this issue?

In your example above you have the task commented out. I assume you actually are running that task.

Can you determine if the tick interrupt is actually running? The tick interrupt must run and execute the xTaskIncrementTick() function. It may be that your simplistic device lacks the timer used for the tick generation.

Hi,

I enabled the Tick Hook and then created a basic method void vApplicationTickHook(void) in main and this is never entered after the kernel begins. Since I am using a JLINK, I made use of the JLINK commander app and read directly from the SysTick Control and Status Register located on the data sheet. From this read, it seems they are properly being set and enabled by the RTOS and the SYST_CVR register value is changing as well.

Walking through the steps of the RTOS scheduler start, I took note of the following:

PendSV, CallSV and SysTick are made the same priority as the kernel via the System Handler Priority Register 3. 
port.c:

#define portNVIC_SYSTICK_CTRL_REG             ( *( ( volatile uint32_t * ) 0xe000e010 ) )
#define portNVIC_SYSTICK_LOAD_REG             ( *( ( volatile uint32_t * ) 0xe000e014 ) )
#define portNVIC_SYSTICK_CURRENT_VALUE_REG    ( *( ( volatile uint32_t * ) 0xe000e018 ) )
#define portNVIC_SHPR3_REG                    ( *( ( volatile uint32_t * ) 0xe000ed20 ) )
#define portNVIC_SYSTICK_ENABLE_BIT           ( 1UL << 0UL )
#define portNVIC_SYSTICK_INT_BIT              ( 1UL << 1UL )
#define portNVIC_SYSTICK_CLK_BIT              ( 1UL << 2UL )
#define portNVIC_SYSTICK_COUNT_FLAG_BIT       ( 1UL << 16UL )
#define portNVIC_PEND_SYSTICK_CLEAR_BIT       ( 1UL << 25UL )
#define portNVIC_PEND_SYSTICK_SET_BIT         ( 1UL << 26UL )
#define portMIN_INTERRUPT_PRIORITY            ( 255UL )
#define portNVIC_PENDSV_PRI                   ( portMIN_INTERRUPT_PRIORITY << 16UL )
#define portNVIC_SYSTICK_PRI                  ( portMIN_INTERRUPT_PRIORITY << 24UL )
...
portNVIC_SHPR3_REG |= portNVIC_PENDSV_PRI;
portNVIC_SHPR3_REG |= portNVIC_SYSTICK_PRI;
.
/* Stop and reset the SysTick. */
    portNVIC_SYSTICK_CTRL_REG = 0UL;
    portNVIC_SYSTICK_CURRENT_VALUE_REG = 0UL;

    /* Configure SysTick to interrupt at the requested rate. */
    portNVIC_SYSTICK_LOAD_REG = ( configSYSTICK_CLOCK_HZ / configTICK_RATE_HZ ) - 1UL;
    portNVIC_SYSTICK_CTRL_REG = portNVIC_SYSTICK_CLK_BIT_CONFIG | portNVIC_SYSTICK_INT_BIT | portNVIC_SYSTICK_ENABLE_BIT;

So I ran the following test in main.c

#define SYSTICKCTRL             ( *( ( volatile uint32_t * ) 0xe000e010 ) )
#define SYSTICKLOAD		( *( (volatile uint32_t *) 0xe000e014))
#define portNVIC_SYSTICK_ENABLE_BIT           ( 1UL << 0UL )
#define portNVIC_SYSTICK_INT_BIT              ( 1UL << 1UL )
#define portNVIC_SYSTICK_CLK_BIT              ( 1UL << 2UL )
int main (void) {
	SystemCoreClockUpdate();
	SYSTICKLOAD = ( SystemCoreClock / configTICK_RATE_HZ ) - 1UL;
    SYSTICKCTRL = portNVIC_SYSTICK_CLK_BIT | portNVIC_SYSTICK_INT_BIT | portNVIC_SYSTICK_ENABLE_BIT;
	uint32_t ticks = SysTick->VAL;
	for(int i = 0; i < 10; i++){
	}
	ticks = SysTick->VAL;
	vTaskStartScheduler();

	while(1);
}

This successfully showed the SysTick value was updating, so it is leaving me a bit stumped. I checked the value of the System Handler Priority Register 3 and it returns E0E00000, [31:24] are used for the SysTick, this would mean the priority is set as 66 for the SysTick. Does any of this seem off?

You said you are using M33. Which FreeRTOS port are you using? Can you share FreeRTOSConfig.h file?

Yes an M33, with no TZ, FPU or DSP. Here is the FreeRTOSConfig.h file I am using.
FreeRTOSConfig.h (10.8 KB)

Which files in the portable folder are you using? Are you using these ones?

When you break the code in the debugger, what is the call stack?

Yes, except I am using FreeRTOS Kernel V10.5.1. This is what the call stack looks like, if I just let it run without stepping through main:

However, if I step into main and into vTaskStartScheduler() then I see this once I reach vStartFirstTask()

I am suspicious of the debugger here. It doesn’t look like the timer task actually has a stack overflow at all. According to your pictures, the top of stack is 0x20012F20, and the (lower) limit is 0x20012E30. No overflow there – in fact, 0xF0 bytes available.

Two suggestions for your FreeRTOSConfig.h file:

  1. Add a definition for configASSERT(), like this:
    #define configASSERT( x ) if( ( x ) == 0 ) { taskDISABLE_INTERRUPTS(); for( ;; ); }

  2. Fix this line:

#define configMAX_SYSCALL_INTERRUPT_PRIORITY    16

Typically it would be built it as follows (or similar):

#define configPRIO_BITS  3
#define configLIBRARY_MAX_SYSCALL_INTERRUPT_PRIORITY  1
#define configMAX_SYSCALL_INTERRUPT_PRIORITY  (configLIBRARY_MAX_SYSCALL_INTERRUPT_PRIORITY << (8 - configPRIO_BITS))

The resulting value of configMAX_SYSCALL_INTERRUPT_PRIORITY is 32 here, not 16. The value 16 is not valid and will break all of FreeRTOS’s critical sections. Your PE implements only 3 bits of interrupt priority, and the value 16 (0x10) doesn’t set any of them (check against 0xE0). The value 32 (0x20) is valid. It allows you to use interrupt priorities 1 through 7 and safely make FreeRTOS API calls from those ISRs. ISRs for interrupt priority 0 (the highest priority) are not be allowed to make FreeRTOS API calls.

Here is how we know your PE implements only 3 bits:

Hi Jeff,

I attempted to increase the configMAX_SYSCALL_INTERRUPT_PRIORITY to 32, but that did not help. I read this “For ports that implement both configKERNEL_INTERRUPT_PRIORITY and configMAX_SYSCALL_INTERRUPT_PRIORITY:
configKERNEL_INTERRUPT_PRIORITY sets the interrupt priority used by the RTOS kernel itself. configMAX_SYSCALL_INTERRUPT_PRIORITY sets the highest interrupt priority from which interrupt safe FreeRTOS API functions can be called.” from https://www.freertos.org/a00110.html#kernel_priority
Looking through my FreeRTOSConfig.h file, I see #define configKERNEL_INTERRUPT_PRIORITY 255 so does this mean my port implements both?

Furthermore, I did the following test:

uint32_t read_systick_priority(void) {
    uint32_t shpr3 = *((volatile uint32_t*) 0xE000ED20);
		return shpr3;
	
}

int main(void){
	SystemCoreClockUpdate();

	//xTaskCreate(task_uart, "UART Test", 1024, NULL, 50, NULL);
		uint32_t systick_priority = read_systick_priority();
	vTaskStartScheduler();
	while(1);
}

This returns 0, however when I use JLINK commander (command line based utility that can be used for verifying proper functionality of J-Link) and read from 0xE000ED20 directly, I am seeing E0E00000. This is not good, it seems inside keil, even before the RTOS kernel begins, I am having problems reading properly from processor. I am not sure why this is, I have verified the JTAG settings 100 times and am using the same speed in Keil as I am in JLINK commander.

You should leave it set to 32. If you set it to 16 you will hit a configASSERT().

No, it’s just generic FreeRTOSConfig.h contents for Cortex M. The CM33 port does not actually use configKERNEL_INTERRUPT_PRIORITY at all.

I don’t think so. If you read SHPR3 after reset, you’ll get 0x00000000. But if you read it after FreeRTOS startup code executes, you’ll get 0xE0E00000. That’s expected. The CM33 port code sets the SysTick and PendSV priorities to the minimum (7 in your case). Priority 7 shows up as 0xE0.

Did defining configASSERT() do any good?

Oh, i just remembered several important configASSERT() statements in the port code were not present in that version. They came along later. Any chance you could update the kernel and port files to the current version?

configASSERT is not defined in the FreeRTOSConfig.h that you shared. Please add and share your findings.

Hi, 10.5.1 is the latest that keil shows for the ARM::CMSIS-FreeRTOS bundle

I tried using the #define configASSERT( x ) if( ( x ) == 0 ) { taskDISABLE_INTERRUPTS(); for( ;; ); } to disable interrupts and the code never stops, the running task is still not entered and does not execute. Tick count shows as zero.

One this I did notice, when reading the SysTick Calibration Value Register, it returned 4007A11F

NOREF: 0 (Reference clock provided)
SKEW: 1 (TENSMS value is inexact or not given, this can affect suitability of SysTick) Is this anything to explore?
TENMS: 7A11F

No, I don’t think that’s anything to worry about

I see. It would take some manual effort then for you to update to 11.x, but I think it is worthwhile. Your application seems to be encountering a fault of some kind that stops execution, and the debugger isn’t helping because it is showing the main-stack context, not the user stack (task) context. Version 11.x of the kernel is likely to hit an assert statement before you get to that point, and the assert statement will identify the issue. Hopefully. :slightly_smiling_face:

I would like to try and manually update it, do you have a link to that new FreeRTOSCOnfig.h for the more recent kernel or could you point me in the right direction? I found the following below, is this a more recent example?

https://github.com/FreeRTOS/FreeRTOS/blob/main/FreeRTOS/Demo/CORTEX_MPU_M33F_Simulator_Keil_GCC/Config/FreeRTOSConfig.h

Your existing FreeRTOSConfig.h will work as-is when you upgrade the kernel files. You can download them from Releases · FreeRTOS/FreeRTOS-Kernel · GitHub. You should be able to replace any bundle file in your project with a like-named file from the 11.x download. That is the manual effort I was talking about. If you are using CMSIS-OS then there is a small chance the CMSIS layer might need a little tweaking but it shouldn’t be much if any. (The download will not include any CMSIS layer, as FreeRTOS does not provide one.)

Okay, thank you, I was able to upgrade to 11.1.0 pretty painlessly, I added the line #define configASSERT( x ) if( ( x ) == 0 ) { taskDISABLE_INTERRUPTS(); for( ;; ); } to my FreeRTOSConfig.h file at line 49. I then started the debugging, stepped through main(), call SystemCoreClockUpdate(); and then create my test UART task, then I start the scheduler via vTaskStartScheduler();

After about 30 seconds, I hit “Stop” and this is what the call stack looked like

Judging from the Disassembly window, I see the PC is at 0x200029A0 which appears to be the configAssert for the PendSV_Handler?

Seems like you have not correctly installed FreeRTOS interrupt handlers? Can you share your vector table (should be somewhere in the startup code)?

This is the startup file I am using, provided by CMSIS
startup_ARMCM33.c (7.2 KB)

I should note, I am executing from flash, so maybe it is an issue with the VTOR not being directly set in FreeRTOS? If that was the case though, how come the PC, LR and SP all end up at the correct location?