Link Register, LR, on GCC/Cortex-M3 seems to be corrupted before running the first task

Xavier · November 4, 2021, 5:30am

Hi, I’m trying the GCC Cortex-M3 port on the SAM3X8E chip for the Arduino platform with this simple program:

#include <FreeRTOS.h>
#include <task.h>

void hello_world_task( void* pvParameters )
{
   (void) pvParameters;
   pinMode( 13, OUTPUT );
   bool led_state = false;

   while( 1 )
   {
      vTaskDelay( pdMS_TO_TICKS( 250 ) );
      digitalWrite( 13, (led_state = !led_state) );
   }
}

void setup()
{
   xTaskCreate(
         hello_world_task,
         "HWT",
         configMINIMAL_STACK_SIZE*4,
         (void*) 0,
         tskIDLE_PRIORITY,
         NULL );

   /* (1) */ SysTick->CTRL = SysTick_CTRL_CLKSOURCE_Msk | SysTick_CTRL_TICKINT_Msk | SysTick_CTRL_ENABLE_Msk; 
   __enable_irq();

   vTaskStartScheduler();   
}

void loop() {}

(1) SysTick() triggers an interrupt before the first task is created, so the kernel crashes. That’s way I needed to disable the SysTick interrupt and then to enable it here (as well as the whole interrupt system).

The scheduler already starts, but when it’s time to start the first and only task it seems that the SVCHandler() corrupts its address, so I get a fault handler (or at least that’s not the address of the task).

In the attached image you can see that the value of the LR register (register 14) before executing the instruction BX is wrong, so a fault handler is triggered.

The LR register holds the value FFFF FFF9 when landing in vPortSVCHandler after the call to the function prvPortStartFirstTask(), as shown:

Is that normal? What must I look for? Any hint? Thanks!

hs2 · November 4, 2021, 6:04am

Doing this

isn’t normal since this is managed by FreeRTOS by default.
I think it might trigger the scheduler before being initialized started and can cause any weird problem…
What if you’re omitting this part ?

Xavier · November 4, 2021, 6:16am

Hi,

If I ommit such code then the kernel crashes because, as I said, the SysTick_handler() is triggered and it asks for a context switch, but at that moment there is any created tasks, and the kernel doesn’t handle such situation.

Instead of editing the Cortex M3 CMSIS library or the GCC port, I opted for disabling and enabling the interrupts and clearing the flags at my side.

EDIT:

This is the main() function in which I disable the interrupts:

int main( void )
{
   __disable_irq();
  
	// Initialize watchdog
	watchdogSetup();

	init();           // enables SysTick()

   SysTick->CTRL = 0; // disables SysTick()

   SCB->ICSR |= ( 1 << 25 );
   // https://developer.arm.com/documentation/dui0552/a/cortex-m3-peripherals/system-control-block/interrupt-control-and-state-register

	initVariant();

//	delay(1); // interrupts are disabled at this moment

#if defined(USBCON)
	USBDevice.attach();
#endif

	setup();

	for (;;)
	{
		loop();
		if (serialEventRun) serialEventRun();
	}

	return 0;
}

And this is how SysTick_Config() looks:

static __INLINE uint32_t SysTick_Config(uint32_t ticks)
{
  if (ticks > SysTick_LOAD_RELOAD_Msk)  return (1);            /* Reload value impossible */

  SysTick->LOAD  = (ticks & SysTick_LOAD_RELOAD_Msk) - 1;      /* set reload register */
  NVIC_SetPriority (SysTick_IRQn, (1<<__NVIC_PRIO_BITS) - 1);  /* set Priority for Cortex-M0 System Interrupts */
  SysTick->VAL   = 0;                                          /* Load the SysTick Counter Value */
  SysTick->CTRL  = SysTick_CTRL_CLKSOURCE_Msk |
                   SysTick_CTRL_TICKINT_Msk   |
                   SysTick_CTRL_ENABLE_Msk;                    /* Enable SysTick IRQ and SysTick Timer */
  return (0);                                                  /* Function successful */
}

hs2 · November 4, 2021, 6:48am

Out of reset the SysTick is disabled and doesn’t fire interrupts. If something else than FreeRTOS manipulates the SysTick owned by FreeRTOS you’re in trouble.
There are platforms resp. libraries (like STM HAL) using the SysTick for other purposes. There an other HW timer is used to tick FreeRTOS.
You’ve to ensure that either the SysTick is reserved for FreeRTOS and don’t touch it or configure your application and FreeRTOS using another ticker.

rtel · November 4, 2021, 8:35pm

The function that configures the tick interrupt is a weak symbol, so you can define your own and so use any timer for the tick interrupt. You also have to install the handler, of course.

aggarg · November 4, 2021, 8:42pm

Since you are returning from interrupt handler, that is expected.

Can you examine the faulting instruction and the cause of the fault? This page contains some instructions about how to do that - Debugging and diagnosing hard faults on ARM Cortex-M CPUs

Xavier · November 5, 2021, 3:06am

Hi,

For the Arduino Due platform (the one that uses the Cortex-M3 chip) the systick is configured early and that makes that an event is fired before the first task is created, and because of that the kernel crashes.

I can, but I don’t want to, mess around with Cortex-M or FreeRTOS libraries, that’s way I disabled and re-enabled the systick in the way I did it.

The systick is configured and started inside the init() function:

int main( void )
{
    __disable_irq();

  	// Initialize watchdog
	watchdogSetup();

	init();   // systick is configured and started here

    // ... more code as previously shown

    // somewhere around here a systick event has been fired

    // I create tasks and start the kernel somewhere here
}

And this is the init() function:

void init( void )
{
  SystemInit();

  // Set Systick to 1ms interval, common to all SAM3 variants
  if (SysTick_Config(SystemCoreClock / 1000))
  {
    // Capture error
    while (true);
  }

  // more unrelated code here

Yes, it is. The original systick handler (among the others handlers needed by the kernel) now are disabled and looks like this (inside the SAM3x8E library cortex_handlers.c file):

void SVC_Handler       (void) __attribute__ ((weak, alias("__halt")));
void PendSV_Handler    (void) __attribute__ ((weak, alias("__halt")));
void SysTick_Handler   (void) __attribute__ ((weak, alias("__halt")));

#if 0 /* FreeRTOS will implement them */
void SVC_Handler       (void) { svcHook(); }
void PendSV_Handler    (void) {	pendSVHook(); }

void SysTick_Handler(void)
{
	if (sysTickHook())
		return;

	tickReset();

	// Increment tick count each ms
	TimeTick_Increment();
}
#endif

And the expected kernel systick handler is almost the same of the Cortex-M3 port (I needed to add the instructions that Arduino requires, but I don’t think that’s a problem):

void xPortSysTickHandler( void )
{
    // needed by Arduino:
	if (sysTickHook()) return; // does nothing
	tickReset();               // doesn't reset anything
	TimeTick_Increment();      // increments an internal Arduino tick counter (for millis and micros functions)



    /* The SysTick runs at the lowest interrupt priority, so when this interrupt
     * executes all interrupts must be unmasked.  There is therefore no need to
     * save and then restore the interrupt mask value as its value is already
     * known. */
    portDISABLE_INTERRUPTS();
    {
        /* Increment the RTOS tick. */
        if( xTaskIncrementTick() != pdFALSE )
        {
            /* A context switch is required.  Context switching is performed in
             * the PendSV interrupt.  Pend the PendSV interrupt. */
            portNVIC_INT_CTRL_REG = portNVIC_PENDSVSET_BIT;
        }
    }
    portENABLE_INTERRUPTS();
}

However, for the first task to run the systick can be disabled (just for the task to start running! problems then will arise), so that the problem I’m facing is not related to the systick.

Xavier · November 5, 2021, 3:13am

Hi,

If I ommit this part the kernel crashes because there was an systick event triggered before the first task is created.

void xPortSysTickHandler( void )
{
    // Arduino stuff:
	if (sysTickHook()) return;
	tickReset();
	TimeTick_Increment();



    /* The SysTick runs at the lowest interrupt priority, so when this interrupt
     * executes all interrupts must be unmasked.  There is therefore no need to
     * save and then restore the interrupt mask value as its value is already
     * known. */
    portDISABLE_INTERRUPTS();
    {

        /* 1 */ if( xTaskIncrementTick() != pdFALSE )  
        {
            /* A context switch is required.  Context switching is performed in
             * the PendSV interrupt.  Pend the PendSV interrupt. */
            portNVIC_INT_CTRL_REG = portNVIC_PENDSVSET_BIT;
        }
    }
    portENABLE_INTERRUPTS();
}

1: xTaskIncrementTick() is called before any task is created and before the kernel is running, that’s way the kernel crashes. This handler must be disabled and then re-enabled as late as possible.

Xavier · November 5, 2021, 3:30am

@rtel, @hs2 , @aggarg

Hi,

Let’s summarize:

In my firsts attempts to make FreeRTOS to work along Arduino Due a systick event got fired before the first task was created, and the kernel crashes because of a call to the function xTaskIncrementTick() inside the systick handler.
I “fixed” that problem delaying the start of the systick timer for as long as possible, and that crash dissapeared.
But then I got a crash when the function that starts the first task (prvPortStartFirstTask()) calls the SVC handler, as mentioned before, because of an invalid value in the LR register. You guys are the experts, but I don’t think this issue is related to the systick timer (for now).

I’ll wait for your kind answers, and in the middle I’ll try with another Cortex-M3 chip (LPC1549) that is already running FreeRTOS to look what’s going on there and then compare results with the SAM3X8E chip.

Kind regards!

hs2 · November 5, 2021, 11:38am

If the SysTick is accidentally enabled by some Arduino code and you re-use it as you mentioned you probably also should NVIC_ClearPendingIRQ( SysTick_IRQn ) to avoid that it fires delayed but unexpectly.
And you should be sure not using any Arduino code perhaps relying on the SysTick…
Even though I’m not familiar with Arduino I wonder if it can be configured NOT using the SysTick. I guess there are some useful hints in the Arduino forums regarding using it with FreeRTOS. You’re for sure not the 1st one doing this. So chances are that there is a clean, reliable solution for it.

jefftenney · November 5, 2021, 1:29pm

That LR value in SVCHandler is correct, even though it looks suspicious. It’s an exception return value, and it will be slightly adjusted prior to use at the end of the handler.

aggarg · November 5, 2021, 5:48pm

You need to ensure that interrupts are not enabled before the scheduler is started. I’d suggest to modify the following code sequence to never enable SysTick in the first place as it leaves a window in which SysTick can still fire:

   init();           // enables SysTick()
   SysTick->CTRL = 0; // disables SysTick()

This is not because of the incorrect LR value. When you step through the SVC_Handler assembly, can you check what value is being programmed into PSP msr psp, r0 (looking at the image you shared, it should be 0x20067EF0)? And after you have that value , can you read 8 words/uint32_t starting from that memory. The value of the 7th word, i.e. sp[6] is the location where BX LR should jump to. Does the address and the code at the location look legitimate?

Thanks.

richard-damon · November 5, 2021, 6:43pm

An alternate to keeping the systick interrupt disabled (because you have a need for other uses) is to add a check if the scheduler has been started before calling the FreeRTOS tick code in the interrupt.

Xavier · November 6, 2021, 3:17am

aggarg:

You need to ensure that interrupts are not enabled before the scheduler is started. I’d suggest to modify the following code sequence to never enable SysTick in the first place as it leaves a window in which SysTick can still fire:

Hi,

I already did that as shown in this previous entry post.

=========
@jefftenney, @hs2, @richard-damon, @rtel:

Thank you for the hint:

PSP holds a strange value: 0x0000 0B00, and in fact the debugger gave me a clue that I overlooked:

I will trace the whole program from the beginning in order to trace back the issue. ASAP I got news you’ll be notified. Thank you all!

BTW1: Today I learnt that BX 0xffff ffd0 means “return to user space from kernel space”; after all the value is correct. It’s mentioned it in tons of places, but I learnt it specially from here.

BTW2: The GCC/Cortex-M3 port works as expected in the LPC1549 chip.

aggarg · November 6, 2021, 5:57am

That seems like a memory corruption. Either the pxCurrentTCB pointer was overwritten or the content of a TCB was corrupted.

Xavier · November 6, 2021, 7:42am

@jefftenney, @hs2, @richard-damon, @rtel, @aggarg:

FOUND IT!

There is a thing in function prvInitialiseNewTask() when it tries to align the top of the stack. The offending lines are these (actually it is the second one):

  #if ( portSTACK_GROWTH < 0 )
        {
            /*1*/ pxTopOfStack = &( pxNewTCB->pxStack[ ulStackDepth - ( uint32_t ) 1 ] );
            /*2*/ pxTopOfStack = ( StackType_t * ) ( ( ( portPOINTER_SIZE_TYPE ) pxTopOfStack ) & ( ~( ( portPOINTER_SIZE_TYPE ) portBYTE_ALIGNMENT_MASK ) ) ); 
   ...

Calculates the top of the stack. OK. For example, it calculates the value 0x20071E24 (value extracted from the debug session as shown below).
Alignes the top of the stack. FAIL. It should be 0x20071E20, but it messes up and one gets 0x00001E20.

After instruction labeled as 1) is executed we get this:

After instruction labeled as 2) is executed we get this:

Releases

I’m using the 2020 release: FreeRTOS Kernel V10.4.3, but the newer one (downloaded as FreeRTOSv202107.00) FreeRTOS Kernel V10.4.4 has the very same code.

And as I mentioned, I was doing tests on other chip (LPC1549) for which its BSP was shipped with an older FreeRTOS release, “FreeRTOS V8.0.1 - Copyright (C) 2014 Real Time Engineers Ltd.”, and it works as expected. In it, the aforementioned lines look like:

#if( portSTACK_GROWTH < 0 )
{
	pxTopOfStack = pxNewTCB->pxStack + ( usStackDepth - ( uint16_t ) 1 );
	pxTopOfStack = ( StackType_t * ) ( ( ( portPOINTER_SIZE_TYPE ) pxTopOfStack ) & ( ( portPOINTER_SIZE_TYPE ) ~portBYTE_ALIGNMENT_MASK  ) );
...

The differences I can see are:

the uint16_t casting,
the location of the ~ operator.

In both cases, the older and newer releases I got:

#ifndef portPOINTER_SIZE_TYPE
    #define portPOINTER_SIZE_TYPE    uint32_t
#endif

and

#define portBYTE_ALIGNMENT 8

How did I make it work?

After executing the instruction labeled as 2) I manually corrected the value for both tasks, mine (hello world) and the Idle task, and the LED on-board of the Arduino Due board blinked as expected.

I won’t mark the thread as resolved until you guys confirm or deny this discovery (it’s too late to edit tasks.c to workaround this issue, but tomorrow I’ll do).

Kind regards!

simonjwright · November 6, 2021, 1:29pm

I’m using 10.0.1, same code as shown here for 10.4.3, works fine on Arduino Due with GCC 8 … 11 for arm-eabi.

I haven’t actually checked this precise issue with the debugger, but it would have had to show!

rtel · November 6, 2021, 3:30pm

A couple of comments:

Reading on my phone so can’t see the details but if there was something fundamentally wrong with that C code I would expect all the Cortex-M devices in our test lab to fail every night, and to of heard about it in the forum before. Can you please post your compiler version and command line so we can try with your exact setup.

Going back to the problem of the interrupt executing between creating a task and starting the scheduler. That smells of a misconfiguration because creating the task before the scheduler starts will itself mask the SysTick interrupt priority. If it doesn’t my first guess would be the FreeRTOS configuration file does not correctly describe the number of priority bits implemented by the processor, or the C startup code is not initializing the variables correctly.

Xavier · November 7, 2021, 3:09am

The Arduino platform ships an older release of GCC for its Due board:

.arduino15/packages/arduino/tools/arm-none-eabi-gcc/4.8.3-2014q1/bin/arm-none-eabi-g++

Although the BSP that I’m using with the the alternative chip (LPC1549) for comparisons is kind of old, the gcc tool-chain (included in the software mcuxpresso) is much newer than that of Arduino:

./arm-none-eabi-gcc --version
arm-none-eabi-gcc (GNU Arm Embedded Toolchain 10-2020-q4-major) 10.2.1 20201103 (release)

Such old Arduino GCC release has a bug, and if it isn’t a bug, then I don’t know what it is. This line (the one that mess things up) in tasks.c:

pxTopOfStack = ( StackType_t * ) ( ( ( portPOINTER_SIZE_TYPE ) pxTopOfStack ) & ( ~( ( portPOINTER_SIZE_TYPE ) portBYTE_ALIGNMENT_MASK ) ) );

has this assembly output:

00080506   LDR            R3, [R7, #32]
00080508   UXTH           R3, R3          /*1*/
0008050A   MOV            R2, R3
0008050C   MOVW           R3, #0xFFF8
00080510   ANDS           R3, R2
00080512   STR            R3, [R7, #32]

(1) This instruction clears the 16 most significant bits of whatever R3 is holding!

I tried splitting the instruction in subinstructions to isolate and fix the problem, but such instruction is always generated no matter what I try:

           StackType_t* x = ~((portPOINTER_SIZE_TYPE)portBYTE_ALIGNMENT_MASK);
     000804DC   MVN            R3, #7
     000804E0   STR            R3, [R7, #28]
            StackType_t* y = pxTopOfStack;
     000804E2   LDR            R3, [R7, #32]
     000804E4   STR            R3, [R7, #24]
            pxTopOfStack = (StackType_t)(((portPOINTER_SIZE_TYPE)x) & ((portPOINTER_SIZE_TYPE)y));
     000804E6   LDR            R3, [R7, #28]
     000804E8   UXTH           R2, R3         // here...
     000804EA   LDR            R3, [R7, #24]
     000804EC   UXTH           R3, R3         // and here...
     000804EE   ANDS           R3, R2
     000804F0   UXTH           R3, R3         // and here again.
     000804F2   STR            R3, [R7, #32]

The newer GCC release (the one that I’m using with the LPC1549 chip) generates this assembly code:

810:	697b	ldr	r3, [r7, #20]
812:	f023 0307	bic.w	r3, r3, #7
816:	617b	str	r3, [r7, #20]

for the same instruction (that I replicate here):

pxTopOfStack = ( StackType_t * ) ( ( ( portPOINTER_SIZE_TYPE ) pxTopOfStack ) & ( ((portPOINTER_SIZE_TYPE ) ~portBYTE_ALIGNMENT_MASK ) );

Arduino guys don’t want to update the GCC release, so I’ll updated myself, and you’ll be notified ASAP I got some news.

Thank you all!

Xavier · November 7, 2021, 6:31am

I solved it and the LED is blinking!

It’s not a compiler problem (I tried the newest and the problem was still there, but I’m gonna keep it, thoug), it’s a problem with the constant portPOINTER_SIZE_TYPE. I needed to re-define it in the FreeRTOSConfig.h file:

#if defined(ARDUINO_ARCH_SAM)

#ifdef portPOINTER_SIZE_TYPE
#undef portPOINTER_SIZE_TYPE
#endif

#define portPOINTER_SIZE_TYPE uint32_t

// more defs

Yes, it was already defined in FreeRTOS.h but that wasn’t enough.

Right now it’s too late to keep talking about it, but tomorrow I’ll explain you what I did and what I was trying.

Greetings!

(… and the LED is still blinking )

EDIT:

(For the next explanation I was using the latest GCC tool-chain from ARM: 10.3-2021.10)

While I was trying to narrow the issue down I did ugly things, like this:

pxTopOfStack = ( StackType_t * ) ( ( ( portPOINTER_SIZE_TYPE ) pxTopOfStack ) & ( ( ( portPOINTER_SIZE_TYPE ) 0xFFFFFFF8 ) ) );

or like this:

pxTopOfStack = ( StackType_t * ) ( ( ( portPOINTER_SIZE_TYPE ) pxTopOfStack ) & ( ( ( uint32_t) ~portBYTE_ALIGNMENT_MASK ) ) );

The fact was that, in any case, the compiler treated the constant portBYTE_ALIGNMENT_MASK as a 16 bit symbol:

uint32_t x = (portPOINTER_SIZE_TYPE)~portBYTE_ALIGNMENT_MASK; // 0xFFF8, ALWAYS, no matter what I tried!

when portBYTE_ALIGNMENT_MASK was declared as:

#if portBYTE_ALIGNMENT == 8
    #define portBYTE_ALIGNMENT_MASK    ( 0x0007 )
#endif

The constant portPOINTER_SIZE_TYPE is declared as (in FreeRTOS.h):

#ifndef portPOINTER_SIZE_TYPE
    #define portPOINTER_SIZE_TYPE    uint32_t
#endif

but it losts its meaning somewhere.

Still doing ugly things that I’m not proud of, while trying to get the 32 bits result: 0xFFFFFFF8 the next idea crossed my mind:

#define portPOINTER_SIZE_TYPE uint32_t

           StackType_t* x = (StackType_t*)((portPOINTER_SIZE_TYPE)(~portBYTE_ALIGNMENT_MASK));
           pxTopOfStack = (StackType_t*)( &( pxNewTCB->pxStack[ ulStackDepth - ( uint32_t ) 1 ] ) );
           pxTopOfStack = ( StackType_t * ) ( ( ( portPOINTER_SIZE_TYPE ) pxTopOfStack ) & ( ( ( portPOINTER_SIZE_TYPE ) ~x ) ) );

and voilá! I got it, there was a problem with the constant portPOINTER_SIZE_TYPE.

And as I shouldn’t leave such definition in this file, I found that re-defining it in FreeRTOSConfig.h still works:

#if defined(ARDUINO_ARCH_SAM)

#ifdef portPOINTER_SIZE_TYPE
#undef portPOINTER_SIZE_TYPE
#endif

#define portPOINTER_SIZE_TYPE uint32_t

// more Cortex-M3 defs...

Then I cleaned the code up, and I tried back with the older GCC release (4.8.3) and the workaround was still working, so I could say that is “safe” to use the GCC release that’s shipped with the Arduino platform.

Now I’ll look for the place or places in which such constant might be changed to 16 bits.

Meanwhile, THANK YOU ALL!