LPC1763, FreeRTOS 8.1.2: Stack Overflow Hook triggers on basic task.

system · April 9, 2015, 11:26am

mikaeljanssonbe wrote on Thursday, April 09, 2015:

I have two tasks; “main” and “uart_task”

I create the main task first and then uart_task. Both contain while (true); loops, i.e. doing nothing but busywaiting.

Eventually I end up in the stack overflow hook, with pxCurrentTCB->pxTopOfStack <= pxCurrentTCB->pxStack, i.e. the first check in vTaskSwitchContext(). When I look at the pxCurrentTCB the first time I’m in uart_task(), it looks as it should, but the second time, it’s something different.

My suspicion is that one of the ISRs (PendSV, SysTick, SVC) is doing something incorrect with the stack.

Any clues? What other information do you need to help me?

rtel · April 9, 2015, 11:46am

rtel wrote on Thursday, April 09, 2015:

Can you please post:

the code that creates the tasks,
the code that implements the tasks (I know you said they didn’t do
anything but loop),
The start of the interrupt vector table

Can you also confirm that the FreeRTOS interrupts are actually
executing. You can do that by looking at the xTickCount value in
tasks.c - is it incrementing at the expected rate?

system · April 9, 2015, 3:02pm

mikaeljanssonbe wrote on Thursday, April 09, 2015:

The FreeRTOS interrupts are executing. Verified by setting breakpoints in the handlers.

I use newlib’s malloc and implement _sbrk() such that start of heap is 0x20080000, i.e. SRAM Bank 1. It is 16 KB large, meaning it ends at 0x20088000. pxCurrentTCB.pxStack and .pxTopOfStack is within this range.

basicvectors tosBasicVectors = {
  (void *) &chipRAMend,                 //  0: stack pointer (0x10007fe0)
  (vector) sysStartup,                  //  1: entry point to start
  (vector) UNEX2,                       //  2: NMI
  (vector) HardFault_Handler,                       //  3: HardFault
  (vector) UNEX4,                       //  4: MemManage
  (vector) UNEX5,                       //  5: BusFault
  (vector) UNEX6,                       //  6: UsageFault
  (vector) 0,                           //  7: (reserved)
  (vector) 0,                           //  8: (reserved)
  (vector) 0,                           //  9: (reserved)
  (vector) 0,                           // 10: (reserved)
  (vector) NVIC_SVC_HANDLER,                      // 11: SVC
  (vector) UNEX12,                      // 12: DebugMonitor
  (vector) 0,                           // 13: (reserved)
  (vector) NVIC_PEND_SV_HANDLER,          // 14: PendSVC
  (vector) NVIC_SYSTICK_HANDLER         // 15: SysTick [offset -1]
  };

Tasks:

void main(void *param) {
while (true);
}
void task_uart(void *param) {
while (true);
}

FreeRTOSConfig.h:

#define configUSE_PREEMPTION		1
#define configUSE_IDLE_HOOK			0
#define configMAX_PRIORITIES		( 5 )
#define configUSE_TICK_HOOK			1
#define configCPU_CLOCK_HZ			( ( unsigned long ) 96000000 )
#define configTICK_RATE_HZ			( ( TickType_t ) 1000 )
#define configMINIMAL_STACK_SIZE	( ( unsigned short ) 80 )
#define configMAX_TASK_NAME_LEN		( 12 )
#define configUSE_TRACE_FACILITY	1
#define configUSE_16_BIT_TICKS		0
#define configIDLE_SHOULD_YIELD		0
#define configUSE_CO_ROUTINES 		0
#define configUSE_MUTEXES			1

#define configMAX_CO_ROUTINE_PRIORITIES ( 2 )

#define configUSE_COUNTING_SEMAPHORES 	0
#define configUSE_ALTERNATIVE_API 		0
#define configCHECK_FOR_STACK_OVERFLOW	2
#define configUSE_RECURSIVE_MUTEXES		1
#define configQUEUE_REGISTRY_SIZE		10
#define configGENERATE_RUN_TIME_STATS	0

/* Set the following definitions to 1 to include the API     function, or zero
to exclude the API function. */

#define INCLUDE_vTaskPrioritySet			1
#define INCLUDE_uxTaskPriorityGet			1
#define INCLUDE_vTaskDelete					1
#define INCLUDE_vTaskCleanUpResources		0
#define INCLUDE_vTaskSuspend				1
#define INCLUDE_vTaskDelayUntil				1
#define INCLUDE_vTaskDelay					1
#define INCLUDE_uxTaskGetStackHighWaterMark	1

I create the tasks with a layer of code that I’d like to wait pasting in, if possible. It’s basically a simple wrapper that lets me associate other data along with a task, using xTaskCreate() with the standard parameters. Stack size of 256 bytes gives me a stack overflow, stack size of 1024 gives me a hard fault.

Something else I’ve seen is that structures like pxCurrentTCB but also stacktraces, sometimes contain 0xA5A5A5A5, ie. FILL_CHAR, in fields such as pvOwner that should be another value. Could it be related?

rtel · April 9, 2015, 3:15pm

rtel wrote on Thursday, April 09, 2015:

I can never be 100% certain, but I’m as close as its possible to be to
100% that the issue is not in our code.

When you say you are using newlib’s malloc (which we really don’t
recommend on that size of processor), have you made it reentrant? That
can be done in several ways - either by using heap_3.c and making sure
nothing calls malloc() directly, or overloading malloc() and free() and
having them direct to one of the FreeRTOS heap managers, or by setting
configUSE_NEWLIB_REENTRANT to 1 in FreeRTOS (this latter method is
provided for convenience of some users, but is not supported by us).

If you want to place the heap at a certain address I would recommend
using heap_5.c.

Your vector table does not seem to call the FreeRTOS interrupt handlers
directly. If you have some kind of wrapper, or modify the stack in any
way before calling the FreeRTOS handlers, then you are likely to be
corrupting the task’s context.

Useful links:

http://www.freertos.org/a00111.html
http://www.freertos.org/FAQHelp.html [note the bit about interrupt vectors]

Regards.

system · April 9, 2015, 3:48pm

mikaeljanssonbe wrote on Thursday, April 09, 2015:

Hi!

Yes, I hope FreeRTOS is tested on LPc1763 or similar, so I don’t think that’s the problem. But I don’t know what could be wrong.

About the sys vector, I forgot to include the defines:

#define NVIC_SVC_HANDLER     vPortSVCHandler
#define NVIC_PEND_SV_HANDLER xPortPendSVHandler
#define NVIC_SYSTICK_HANDLER xPortSysTickHandler

We use heap_3.c, configUSE_NEWLIB_REENTRANT=1 and implement __mallock_lock()/__malloc_unlock() as disable/enable interrupts, to comply with Newlib’s malloc() locking mechanisms. Does this suffice?

From what I can tell by breaking at malloc() and vTaskStartScheduler(), no allocations happen after IDLE task is created. How can malloc() be the cause?

What I am seeing here is that PSP (mon regs in gdb) is outside of pxCurrentTCB’s stack, and at task switch time, pxCurrentTCB.pxTopOfStack is overwritten with PSP, and subsequently the stack overflow test fails in vTaskSwitchContext().

Example stack overflow and pxCurrentTCB. (it fails top of stack <= stack):

vApplicationStackOverflowHook (pxTask=0x20080498, pcTaskName=0x200804cc "task_uart") at FreeRTOSCommonHooks.c:91
(gdb) bt
#0  vApplicationStackOverflowHook (pxTask=0x20080498, pcTaskName=0x200804cc "task_uart") at FreeRTOSCommonHooks.c:91
#1  0x00002584 in vTaskSwitchContext () at tasks.c:2155
#2  0x00001b9e in xPortPendSVHandler () at port.c:440
Backtrace stopped: frame did not save the PC

(gdb) p *pxCurrentTCB
$1 = {pxTopOfStack = 0x20080150, xGenericListItem = {xItemValue = 50, pxNext = 0x10000ca8 <xDelayedTaskList1+8>, pxPrevious = 0x10000ca8 <xDelayedTaskList1+8>, pvOwner = 0x20080498, pvContainer = 0x10000ca0 <xDelayedTaskList1>},
xEventListItem = {xItemValue = 3, pxNext = 0x0 <tosBasicVectors>, pxPrevious = 0x200804e0, pvOwner = 0x20080498, pvContainer = 0x0 <tosBasicVectors>},
uxPriority = 2, pxStack = 0x200804f0, pcTaskName = "task_uart\000\000", uxTCBNumber = 2, uxTaskNumber = 329, uxBasePriority = 2, uxMutexesHeld = 0}
(gdb)

Am I missing something here, or shouldn’t PSP always be within the current process’s stack, i.e. between pxStack and pxTopOfStack?

rtel · April 9, 2015, 4:04pm

rtel wrote on Thursday, April 09, 2015:

Your application task stacks are huge, while the stack for the idle task
is minimal, so it does not look like it is a genuine stack overflow,
just something going amiss with the stack that is triggering the stack
overflow hook. Yes, the stack pointer should be between the start and
end of the application tasks stack.

While you are not calling malloc() after the scheduler has started,
having configUSE_NEWLIB_REENTRANT will be having an effect during the
context switch. Can you please try with configUSE_NEWLIB_REENTRANT set
to 0.

Also the mallock_lock()/malloc_unlock() are probably safe if they are
manipulating the global interrupt enable bit (FreeRTOS doesn’t touch
that so there won’t be a clash of expectations), however please try
without those defined too. I don’t think that will make a difference
as, as we both mentioned already, at that time malloc() should be be
being called anyway.

Is there anything special about RAM bank1? For example, does it need to
be enabled, or is it shared with the debugger, or anything like that.
When we ported FreeRTOS+TCP to the LPC18xx there were four (five?)
different RAM banks, and everything worked fine until the heap touched
the highest bank in the address map (heap_5 was used to spread the heap
across different RAM banks). I can’t remember what the issue was or
what we did to fix it, but we had to do something.

Regards.

system · April 9, 2015, 4:42pm

mikaeljanssonbe wrote on Thursday, April 09, 2015:

Hi again,

Nothing special about RAM bank 1. We’ve historically used bank 0 for other data.
I changed these parameters:

include/not include __malloc_lock()/__malloc_unlock()
SRAM bank 0 instead of bank 1
Set configUSE_NEWLIB_REENTRANT to 0 and 1.

There was no difference. I do have a GDB session that clearly shows the problem I’m seeing:

Start at rtos_main(), the first thing called after setting up the data segment, clearing BSS etc.:

..., rtos_main () at rtos.c:724
(gdb) mon regs
R0 = 00000000, R1 = 00000003, R2 = 05FA0200, R3 = 00000000
R4 = 3456ABCD, R5 = 3456ABCD, R6 = 12345678, R7 = 10007FC8
R8 = A5A5A5A5, R9 = A5A5A5A5, R10= A5A5A5A5, R11= A5A5A5A5
R12= A5A5A5A5, R13= 10007FC0, MSP= 10007FC0, PSP= 2007C1A0
R14(LR) = 00001737, R15(PC) = 00001136
XPSR 01000000, APSR 00000000, EPSR 01000000, IPSR 00000000

Note the value of PSP at this point.

Break again at task_uart - started as a task by xTaskCreate() and vStartScheduler(). But PSP here is (almost) the same as before FreeRTOS started, and definitely not within pxStack-pxTopOfStackRange:

Breakpoint 1, task_uart (blob=0x0 <tosBasicVectors>) at application.c:97
97              ;
(gdb) mon regs
R0 = 00000000, R1 = 00000114, R2 = 00000000, R3 = 2007C1C4
R4 = A5A5A5A5, R5 = A5A5A5A5, R6 = A5A5A5A5, R7 = 2007C1C0
R8 = A5A5A5A5, R9 = A5A5A5A5, R10= A5A5A5A5, R11= A5A5A5A5
R12= A5A5A5A5, R13= 2007C1C0, MSP= 10007FC0, PSP= 2007C1C0
R14(LR) = 00000E9B, R15(PC) = 0000030A
XPSR 01000000, APSR 00000000, EPSR 01000000, IPSR 00000000
CFBP 02000000, CONTROL 2000000, FAULTMASK 00, BASEPRI 00, PRIMASK 00
(gdb) p pxCurrentTCB.pxTopOfStack
$1 = (volatile StackType_t *) 0x2007c5a8
(gdb) p pxCurrentTCB.pxStack
$2 = (StackType_t *) 0x2007c4f0

Stop at the next interrupt called and check regs (PSP in particular) vs. pxTopOfStack:

(gdb) b xPortSysTickHandler 
Breakpoint 2 at 0x1b48: file port.c, line 474.
(gdb) disable 1
(gdb) c
Continuing.
Breakpoint 2, xPortSysTickHandler () at port.c:474
474     {
(gdb) mon regs
R0 = 00000000, R1 = 00000114, R2 = 00000000, R3 = 2007C1C4
R4 = A5A5A5A5, R5 = A5A5A5A5, R6 = A5A5A5A5, R7 = 2007C1C0
R8 = A5A5A5A5, R9 = A5A5A5A5, R10= A5A5A5A5, R11= A5A5A5A5
R12= A5A5A5A5, R13= 10007FC0, MSP= 10007FC0, PSP= 2007C1A0
R14(LR) = FFFFFFFD, R15(PC) = 00001B48
XPSR 0100000F, APSR 00000000, EPSR 01000000, IPSR 0000000F
CFBP 00000000, CONTROL 00, FAULTMASK 00, BASEPRI 00, PRIMASK 00
(gdb) p pxCurrentTCB.pxTopOfStack
$1 = (volatile StackType_t *) 0x2007c5a8

Everything in order. Continue to xPortPendSVHandler invoked by systick IRQ and verify pxTopOfStack is correct:

(gdb) b vTaskSwitchContext 
Breakpoint 3 at 0x2452: file tasks.c, line 2116.
(gdb) b xPortPendSVHandler
Breakpoint 4 at ...
(gdb) c
Continuing.
Breakpoint 4, xPortPendSVHandler () at port.c:440
440             __asm volatile
(gdb) p pxCurrentTCB.pxTopOfStack
$5 = (volatile StackType_t *) 0x2007c5a8

Step through the handler:

(gdb) layout asm
(gdb) # si a few times

B+ |0x1b04 <xPortPendSVHandler>             mrs    r0, PSP
   |0x1b08 <xPortPendSVHandler+4>           isb    sy
   |0x1b0c <xPortPendSVHandler+8>           ldr    r3, [pc, #52]
   |0x1b0e <xPortPendSVHandler+10>          ldr    r2, [r3, #0]
   |0x1b10 <xPortPendSVHandler+12>          stmdb  r0!, {r4, r5, r6, r7, r8, r9, r10, r11}
  >|0x1b14 <xPortPendSVHandler+16>          str    r0, [r2, #0]

Here, r0 contains PSP and we’re just about to write r0 to [r2], which is pxCurrentTCB:

(gdb) mon regs
R0 = 2007C180, R1 = 10000C3C, R2 = 2007C498, R3 = 10000C38
R4 = A5A5A5A5, R5 = A5A5A5A5, R6 = A5A5A5A5, R7 = 2007C1C0
R8 = A5A5A5A5, R9 = A5A5A5A5, R10= A5A5A5A5, R11= A5A5A5A5
R12= A5A5A5A5, R13= 10007FC0, MSP= 10007FC0, PSP= 2007C1A0
R14(LR) = FFFFFFFD, R15(PC) = 00001B14
XPSR 2100000E, APSR 20000000, EPSR 01000000, IPSR 0000000E
(gdb) p pxCurrentTCB.pxTopOfStack
$6 = (volatile StackType_t *) 0x2007c5a8

Do the store:

(gdb) si
(gdb) p pxCurrentTCB.pxTopOfStack
$7 = (volatile StackType_t *) 0x2007c180
(gdb)

Presto! pxTopOfStack overwritten by PSP. Which probably is what the code is supposed to do, but since PSP is never updated to match pxTopOfStack, it’s the same as it was before FreeRTOS started.

system · April 9, 2015, 7:44pm

edwards3 wrote on Thursday, April 09, 2015:

Start at rtos_main(), the first thing called after setting up the data segment, clearing BSS etc.:

Note the value of PSP at this point.

At that point PSP has not been used and probably just contains whatever it contained last time, unless you did a hard reset.

Break again at task_uart - started as a task by xTaskCreate() and vStartScheduler(). But PSP here is (almost) the same as before FreeRTOS started, and definitely not within pxStack-pxTopOfStackRange:

Is this inside the task_uart() function? At that point the PSP should be set correctly, but it is not even in the correct RAM bank. What is below the RAM bank 0 in the memory map? Another RAM bank or nothing?

What is TCB.pxTopOfStack for uart_task immediately after uart_task is created and before the scheduler has started? I’m guessing it is never where you think it is in the first place.

You might need to get a handle to the task to find its TCB (the handle is a pointer to the TCB). Alternatively when you create uart_task you will go through a function called xTaskGenericCreate() that sets pxNewTCB->pxTopOfStack, you can see the value there.

system · April 9, 2015, 8:56pm

mikaeljanssonbe wrote on Thursday, April 09, 2015:

You might need to get a handle to the task to find its TCB (the handle is a pointer to the TCB). Alternatively when you create uart_task you will go through a function called xTaskGenericCreate() that sets pxNewTCB->pxTopOfStack, you can see the value there

Can I look at pxCurrentTCB when I’m inside task_uart() for its TCB? This is what I’ve done so far.

At that point PSP has not been used and probably just contains whatever it contained last time, unless you did a hard reset.

Yes, which is what I’m trying to show later in the gdb session - PSP keeps being the same value, more or less, even when I’m in task_uart(). From my understanding, PSP should always match pxCurrentTCB.pxTopOfStack, give or take however many bytes there are in a frame - correct?

Is this inside the task_uart() function? At that point the PSP should be set correctly

Yup, it’s inside the task_uart() function. I’m printing pxCurrentTCB since to my understanding it’s the TCB associated with the current task… but PSP is not set correctly. This, I think, is the entire problem. Why isn’t PSP in the pxStack…pxTopOfStack range? I don’t have the sources available right now, but shouldn’t PSP be set to pxCurrentTCB.pxTopOfStack set in xPortSVCHandler() invoked by the service call in prvPortStartFirstTask()?

but it is not even in the correct RAM bank. What is below the RAM bank 0 in the memory map? Another RAM bank or nothing?

It points to chip RAM. Memory layout is 32K chip RAM, SRAM bank 0, SRAM bank 1. It’s really all continuous memory. chipRAMend is where the PSP is initially setup by the start vector. Above chipRAMend is another 32K of RAM, off the AHB bus.

So the initial PSP is OK… it’s just that it never changes to the task’s own stack pointer.

system · April 9, 2015, 9:22pm

edwards3 wrote on Thursday, April 09, 2015:

So when you say “Break again at task_uart” and show the PSP as being 0x2007C1A0 I’m taking that as the PSP value when task_uart starts running for the first time, before it enters the while(1) loop. So that is the PSP value before the task has ever been switched out. If that is the case then its stack is already wrong before the PendSV handler executes to switch away from it, which is why I guessed the stack was not where you expected it to be right from the start. If that is not misinterpreting your post then its not the interrupts that are the problem, but how the task is allocated a stack in the first place.

system · April 10, 2015, 4:49am

mikaeljanssonbe wrote on Friday, April 10, 2015:

Hi guys!

Apparently, doing something exactly and “exactly” is a wee bit different. My uart_task() had a local buffer after the while(true); which I remember being 64 bytes big. Well, it wasn’t…

For anyone reading this later: if FreeRTOS says the stack overflowed, it could very well be the case that the stack did overflow. Check your data!

Thanks a lot for your time, help & support! I really appreciate it. Very helpful community!