Queue crash problem

rousea wrote on Wednesday, October 27, 2010:

I am developing a project using an STM32 ARM processor and FreeRTOS.  there are 3 USARTS, and the RX ISR for each gives a semphore which is taken by task  vUSART_RXcharReceived:

void vUSART_RXcharReceived(void* pvParameters){
	portTickType	TimeOut;
	u8	Port;
	TimeOut = portMAX_DELAY;
	Port = *(u8*)pvParameters;
	//	Make sure the semaphore is cleared initially
	xSemaphoreTake(prvHIO.USART_RXcharReceived[Port], 0);
	while (1){
		if (xSemaphoreTake(prvHIO.USART_RXcharReceived[Port], TimeOut) == pdPASS){	//	If we received a char...
			TimeOut = 35000/SysConfig.COMport[Port].BaudRate;	//	Time for 3.5 chars
			if (TimeOut < 2)
				TimeOut = 2;
		}else{		//	End of message
			TimeOut = portMAX_DELAY;
			xQueueSend(HIO.CheckMessage, &Port, portMAX_DELAY);

There are three insances of this task, one for each USART, and as you see they wait for  a break in transmission then add an item to queue HIO.CheckMessage to identify which port now contains a message.

The queue is received by vCheckMessage:

void vCheckMessage(void* pvParameters){
	s16	MsgLen, Response = DONE;
	u8	DiagnosticsDisplayed;
	u8	Port;
	HIO.CheckMessage = xQueueCreate(15, sizeof(u8));
	while (1){
		xQueueReceive(HIO.CheckMessage, &Port, portMAX_DELAY);

I am testing it using traffic on only one USART. The programme works correctly for the first six messages, then fails, giving a Hard Fault exception, which means it has tried to access an invalid memory address.  The fault always occurs as the code is running xQueueSend(HIO.CheckMessage, &Port, portMAX_DELAY) in vUSART_RXcharReceived(().  I have pinpointed to error to task vListRemove( ) in FreeRTOS list.c. This copies a value to a pointer

pxList = ( xList * ) pxItemToRemove->pvContainer;

However, when the system fails it is because the value of this pointer is 0 (NULL), so when it is subsequently used the code attempts to write to Flash code space.

I obviously cannot change the code within FreeRTOS, yet I can’t see anything I am doing wrong to cause the problem.  When the failure occurs I can look at the memory location of  pxItemToRemove and see that all elements appear to hold sensiboe values except pvContainer, which is 0.  I can’t put a watch on it to trap when it is set to 0, since it is constantly being updated when the code is running.

Any suggestions?

Alan Rouse

richard_damon wrote on Wednesday, October 27, 2010:

While I am not familiar with this particular port, the issue sounds like what can happen if the interrupt isn’t setup right and doesn’t respect the Critical Section code. The key here is that the priority of the interrupt needs to be set properly. If it can interrupt during a Critical Section or causes unexpected interrupt nesting then the interrupts can corrupt the data structures.

The other possibility is that an interrupt is using a non FromISR routine which can also cause this sort of issue.

rousea wrote on Wednesday, October 27, 2010:

Thanks for your ideas.  I have already repeatedly checked to confirm that all semaphore and queue accesses from ISR’s use the correct call format.  However, the queue that is failing has no connection with interrupts.  It is simply passing a queue from one task to another.

Regarding your first thought, do you mean the FreeRTOS priorities or the ARM priorities? On the ARM the Systick intterupt is higehst (0) and the USART interrupts are lowest (4).  On FreeRTOS the USART task priority is set to 3, so it is higher than the CheckMessage() task, which is set to priority 2. I believe that taskENTER_CRITICAL() disables all interrupts, so I can’t understand why there should be any problem if an interrupt occurs during critical code sections.

Since posting the problem I checked which version of FreeRTOS is being used and found it is V5.4.2.  I have now downloaded the latest version, 6.1.0 in the hope that it may fix the problem.  However, I can’t yet get it to work:  vTaskStartScheduler() calls vPortStartFirstTask(), which comprises a small number of assembler instructions.  When it executes the last instruction, svc 0, the processor jumps to the default handler exception.  If I revert back to FreeRTOS V5.4.2 the programme starts properly, but exhibits the original problem.  It would appear, therefore, that there is a compatibility problem between V5 and V6 of FreeRTOS.

rtel wrote on Thursday, October 28, 2010:

Regarding your first thought, do you mean the FreeRTOS priorities or the ARM priorities?

I think richard_damon is referring to the interrupt priorities.

On the ARM the Systick intterupt is higehst (0) and the USART interrupts are lowest (4).

This does not sound right and if so will cause the sort of problem you are seeing.  The Systick would normally be the lowest (255, or 15, depending on how you want to define it).  The UART interrupts can be above the (lower numeric values) but must have a lower or equal interrupt priority than whatever configMAX_SYSCALL_INTERRUPT_PRIORITY is set to if you are calling FreeRTOS API functions from the interrupt.

I have now downloaded the latest version, 6.1.0 in the hope that it may fix the problem.

From what you have said, it looks like an interrupt priority problem.  Changing FreeRTOS versions is not going to fix that.

When it executes the last instruction, svc 0, the processor jumps to the default handler exception

Presumably therefore you have not installed an SVC handler.  The way the kernel is started changed at some point, I cannot remember when but this will be documented in the change history (http://www.freertos.org/History.txt).

Please go to the documentation page for the port you are using on the FreeRTOS web site and read the “RTOS Port specific configuration” section that describes how to set up the interrupt priorities.


rousea wrote on Thursday, October 28, 2010:


Thanks for your response.  Since my original posting and richard_damon’s response I have looked closely into interrupt priorities.  I was wrong in assuming the SysTick interrupt was set to priority 0.  It is in fact 255 as you say.  I have realised the following:

1. On ARM processors the priority level is 8-bits, divided into Pre-emption Priority (the most significant part) and Sub-Priority (the least significant part). NVIC_PriorityGroupConfig defines the split, between 7:1 and 0:8. An interrupt at a given Pre-emption Priority Level will pre-empt any in higher-numbered interrupt.  Simultaneous interrupts at the same Pre-emption Priority Level are processed in Sub-Priority sequence (lowest first), but I can see no purpose in Sub-Priority level.

2. The STM32 only implements 16 Pre-emption Priority levels so expects the NVIC_PriorityGroupConfig  to define the split as 4:4 (i.e. the most significant 4 bits define Pre-emption Priority and the least significant 4 bits define Sub-Priority).  However, it doesn’t use the latter, since whatever value is written the value read back is 0.

3. The FreeRTOS definitions for configKERNEL_INTERRUPT_PRIORITY and configMAX_SYSCALL_INTERRUPT_PRIORITY must reflect this structure, so should bet set to 0xN0 where N is the required level.

4. FreeRTOS disables interrupts by copying configMAX_SYSCALL_INTERRUPT_PRIORITY to its BASEPRI register.  This register is not very well documented by either ARM or ST, but if set to any value other than 0 it disables only interrupts set to higher Pre-emption Priority levels.

Bearing this in mind I have set configMAX_SYSCALL_INTERRUPT_PRIORITY to 0x30 and configKERNEL_INTERRUPT_PRIORITY to 0xFF.  I have then set all the interrupts I use to values greater than 0x30.  I can confirm the settings by examining the relevant registers within the NVIC block.

However, I still get the same problem, namely that the system crashes with a Hard Fault on the 7th message received via a USART. pxCurrentTCB shows the failure at the vUSART_RXcharReceived task.  I have again confirmed that the fault is due to pxList being set to 0.  I cannot see what is wrong with my code to cause this problem, so I am still looking for ideas.

Regarding FreeRTOS V6:

FreeRTOS v5.4.2/Source/portable/GCC/ARM_CM3/port.c includes
   #define vPortSVCHandler SVC_Handler
   #define xPortPendSVHandler PendSV_Handler
   #define xPortSysTickHandler SysTick_Handler
followed by definitions of the functions.

However, FreeRTOS v6.1.0/Source/portable/GCC/ARM_CM3/port.c defines the functions but doesn’t include the #defines.  If I copy them from the older version the system works.  However, it exhibits the same problem with pxList, as expected.


rousea wrote on Friday, October 29, 2010:


I have now modified my code in a way that I hoped would overcome the problem, but it now manifests itself in a different way:

I have added diagnostics functions SetGPIO() and ClearGPIO() to force digital states on spare CPU output pins that can be monitored on an oscilloscope.  I have also implemented re-triggerable timers in the USART receive interrupt routines to eliminate the need for intermediate task vUSART_RXcharReceived().  When a timer lapses it generates an interrupt which adds the port number to the CheckMessage queue:

void	TIM2_IRQHandler(void){
	portBASE_TYPE xHigherPriorityTaskWokenByPost;
	u8	Port;
	Port = COM3;
	xQueueSendFromISR(HIO.CheckMessage, &Port, &xHigherPriorityTaskWokenByPost);

Port TP1 is set every time a character is received and cleared when the message is added to the CheckMessage queue;

I have also added diagnostics in vCheckMessage():

void vCheckMessage(void* pvParameters){
	u8	Port;
	HIO.CheckMessage = xQueueCreate(15, sizeof(u8));
	while (1){
		prvHIO.MessagesInQueue = uxQueueMessagesWaiting(HIO.CheckMessage);
		xQueueReceive(HIO.CheckMessage, &Port, portMAX_DELAY);

I should therefore be able to see on the oscilloscope a pulse on TP1 when a message is received and a pulse on TP2 when it is processed.  Furthermore, NoOfMessages and MessagesInQueue allow me to check how many messages have been processed and the number waiting in the queue.

The diagnostics show that after correctly processing 5 messages vCheckMessage() remains permanently blocked. TIM2_IRQHandler() continues to receive messages but can’t add them to the queue because it is full.

Can anyone give me any ideas?

rousea wrote on Monday, November 01, 2010:

Cracked it!

I eventually found the problem:

Under certain circumstances my CheckMessage() routine could execute a FreeRTOS xTaskResumeAll() without first executing vTaskSuspendAll().  The effect of this is apparently to corrupt message handling.

hooverphonique wrote on Tuesday, November 09, 2010:

so how were you able to create such a problem by just using the FreeRTOS API functions? Looking at your checkmessage function, I would say that if it doesn’t work, it indicates a bug in FreeRTOS.

how did you solve the problem, exactly?


rousea wrote on Tuesday, November 09, 2010:

Some messages processed by CheckMessage() modify global data using read-modify-write so I need to ensure to other tasks can interrupt them.  I therefore use  vTaskSuspendAll() within an if{} statment, carry out some common functionality, then include xTaskResumeAll().within another if{} statement.  Unfortunately the if{} statements were very slightly different, so some messages did not satisfy the first but did satisfy the second.  Hence these messages caused xTaskResumeAll().to be executed without a preceding vTaskSuspendAll(), even though CheckMessage() exectued without any apparent errors.

I tracked down the problem by dogged persistence!  I was looking for problems in queue handling so it took me a long time to realise that this can be affected by other API tasks.