Queue Problem

davebryan wrote on Saturday, March 15, 2008:

Hi,

I’ve implemented a queued UART Tx/Rx scheme based on the code in Demo\CORTEX_STM32F103_IAR\serial.c file.

I’m having intermittent problems with the Tx queue where the call to xQueueReceiveFromISR in the UART interrupt handler returns the incorrect data from the Tx queue. When incorrect data is returned I’ve found that the Tx queue ‘pcReadFrom’ pointer equals the Tx queue ‘pcWriteTo’ pointer. For correct operation the pcReadFrom pointer should be 1 or more bytes behind the pcWriteTo pointer.

I only have a single task accessing the Tx/Rx queues. The fact that this behaviour happens intermittantly suggests some sort of timing issue but looking at the FreeRTOS queue.c code I can’t see how this can happen.

Has anyone come across this or have any clues ?

Thanks
Dave

Code is as follows:

void Uart_init()
{
  /* create the receive & transmit queues */
  hostRxQ = xQueueCreate( 128, 1);
  hostTxQ = xQueueCreate( 128, 1);
}

portBASE_TYPE Uart_Tx( unsigned char byte, portTickType timeout)
{
  /* try to queue the byte */
  portBASE_TYPE txResult = xQueueSend( hostTxQ, &byte, timeout);
   
  if (txResult == pdTRUE)
  {
    /* enable transmitter empty interrupts */
    USART_ITConfig(USART2, USART_IT_TXE, ENABLE);
  }
  return txResult;
}

void UART_Interrupt_Handler()
{
  unsigned char ch;
  portBASE_TYPE taskWokenByRxQAdd = pdFALSE;
  portBASE_TYPE taskWokenByTxQRemove = pdFALSE;

  if (USART_GetITStatus(USART2, USART_IT_TXE) == SET)
  {  
    /* transmitter is ready for the next byte so see if there is more to transmit */
    if (xQueueReceiveFromISR( hostTxQ, &ch, &taskWokenByTxQRemove) == pdTRUE)
    {
      /* send the byte */
      USART_SendData(USART2, ch);
    }
    else
    {
      /* disable further USART2 Transmit interrupts */
      USART_ITConfig(USART2, USART_IT_TXE, DISABLE);
    }   

    /* Clear the USART2 transmit interrupt pending flag */
    USART_ClearITPendingBit(USART2, USART_IT_TXE);
  }

  if (USART_GetITStatus(USART2, USART_IT_RXNE) == SET)
  {
    /* read the received byte from the receive data register */
    ch = USART_ReceiveData(USART2);
    taskWokenByRxQAdd = xQueueSendFromISR( hostRxQ, &ch, taskWokenByRxQAdd);

    /* Clear the USART2 Receive interrupt */
    USART_ClearITPendingBit(USART2, USART_IT_RXNE);
  }

  /* see if we need to switch tasks */   
  portEND_SWITCHING_ISR( taskWokenByRxQAdd || taskWokenByTxQRemove);
}

davebryan wrote on Sunday, March 16, 2008:

I found the problem: my UART interrupt handler was preempting xQueueSend() & interfering with the Tx queue manipulation.

I’m using a STM32 Cortex part & the priority of my UART interrupt was greater than the FreeRTOS priority (set using configKERNEL_INTERRUPT_PRIORITY). However, I’ve tried setting the UART interrupt to the same priority (and lower) as the OS but still UART interrupts preempt the critical section protected parts of xQueueSend().

The only way I can stop this happening is to modify vPortSetInterruptMask() to globally disable interrupts, using the ‘cpsid i’ instruction, rather than setting a particular preemption level.

Has anyone else come across this issue ?

rtel wrote on Sunday, March 16, 2008:

> However,
> I’ve tried setting the UART interrupt to the same priority
> (and lower) as the
> OS but still UART interrupts preempt the critical section
> protected parts of
> xQueueSend().

I can’t see how that can be, unless there is a mistake somewhere with the priority masking.

Are you doing something like this?

NVIC_InitStructure.NVIC_IRQChannel = USART1_IRQChannel;
NVIC_InitStructure.NVIC_IRQChannelPreemptionPriority = configLIBRARY_KERNEL_INTERRUPT_PRIORITY;
NVIC_InitStructure.NVIC_IRQChannelSubPriority = 0;
NVIC_InitStructure.NVIC_IRQChannelCmd = ENABLE;
NVIC_Init( &NVIC_InitStructure );

Where configLIBRARY_KERNEL_INTERRUPT_PRIORITY is by default 15?

Regards.

davebryan wrote on Sunday, March 16, 2008:

Richard

I had done exactly what you suggest in your reply and configLIBRARY_KERNEL_INTERRUPT_PRIORITY is set to 15. I’ve also tried setting configLIBRARY_KERNEL_INTERRUPT_PRIORITY to 0 but that made no difference.

As an experiement I added an extra line to vPortSetInterruptMask() to read back the value written to the basepri register. When I stepped through the code I read 0 regardless of what ulKernelPriority is.

#define vPortSetInterruptMask()                           
__asm volatile                               
    (                                                        "    push { r0 }                        \n"   
        "    ldr r0, =ulKernelPriority                 \n"   
        "    ldr r0, [r0]                        \n"   
        "    msr basepri, r0                        \n"    
extra line ->    "    mrs r0, basepri                        \n"    
(r0 always 0)    "    pop { r0 }                        "   
    )

It made me wonder whether the code runs in unpriviledged mode but that fact that “cpsid i” for vPortSetInterruptMask() works suggests its running in priviledged mode, plus I can’t find any code to set unpriviledged mode.

Do you have any other suggestions I can try ?

Regards
Dave

rtel wrote on Sunday, March 16, 2008:

I just tried your code on a Luminary Micro part (also Cortex M3) and it behaved as expected.

Looking at the registers in the debugger I see that " msr basepri, r0 \n" sets basepri to 0xe0.  This is as expected as Luminary Micro only have 8 priority levels.

" msr basepri, r0 \n" loads 0xe0 into r0, then immediately " pop { r0 } " overwrites r0 again.

Can you tell me where you placed the code to run your test?  Before or after the scheduler is started, and, within an interrupt of within a task.

I will have to dig out my STM32 to test this.

Regards.

davebryan wrote on Sunday, March 16, 2008:

Richard

I put the code after the scheduler is started. I was stepping through the modified vPortSetInterruptMask() code after the call to taskENTER_CRITICAL() in the xQueueGenericSend() function (line 504, queue.c)

Regards
Dave

rtel wrote on Sunday, March 16, 2008:

Are you using IAR V5.xx?  I just switched to IAR and find the same issue as you.  It seams CONTROL[0] is clear when main() is called so thread mode is not privileged.  By default thread mode should be privileged. 

I will get back to you on this.

Regards.

davebryan wrote on Sunday, March 16, 2008:

Richard

I’m using GCC v4.2.1

Regards
Dave

rtel wrote on Sunday, March 16, 2008:

For some reason I thought you were using IAR.  My original experiment was with GCC and it seemed to work ok.

Which startup code are you using?  Is the startup code setting handler mode to be non privileged (what does reset jump to for the first position in the interrupt table)?

In your debugger, can you view bit 0 of the CONTROL register?  Which debugger and JTAG interface are you using?

Regards.

rtel wrote on Monday, March 17, 2008:

I have come to the conclusion that the IAR debugger does not display the control registers correctly and that was a bit of a red herring. 

I have tried a program that sets up a sys tick interrupt that toggles an LED every five seconds (the tick frequency being 1KHz).  main() simply waits for 5 toggles, then disables interrupts, and sure enough the toggling LED stops, therefore the disabling of interrupts must be working correctly.  Also, when doing this I can read back basepri during portDISABLE_INTERRUPTS() (it is set to 0xff and is read back as 0xf0, which is expected, even though the IAR debugger shows its value as 0).

I have tried this from both GCC and IAR.  The IAR code for the STM32 is given below.  I have also tried reading back the basepri from within the queue send function after the scheduler has been started (as you did), and it seems to be working ok there too.

There could potentially be an issue with the configLIBRARY_KERNEL_INTERRUPT_PRIORITY setting, but without being able to read back the registers its a bit tricky to know.  The ST library is somewhat confusing, but stepping through on the debugger it seems to set the priority to 0xF0 for the UART, which would I think be correct for the lowest priority available.

Are you able to set up the smallest project possible that demonstrates the issue and send it to me?  If so then send it to r (dot) barry _at_ freertos.org.  Also let me know the debug interface you are using.

Regards.

Test program follows:

/* Library includes. */
#include "stm32f10x_it.h"
#include "partest.h"

#define configKERNEL_INTERRUPT_PRIORITY 255

#define portNVIC_SYSTICK_CTRL________( ( volatile unsigned portLONG *) 0xe000e010 )
#define portNVIC_SYSTICK_LOAD________( ( volatile unsigned portLONG *) 0xe000e014 )
#define portNVIC_INT_CTRL____________( ( volatile unsigned portLONG *) 0xe000ed04 )
#define portNVIC_SYSPRI2____________( ( volatile unsigned portLONG *) 0xe000ed20 )
#define portNVIC_SYSTICK_CLK________0x00000004
#define portNVIC_SYSTICK_INT________0x00000002
#define portNVIC_SYSTICK_ENABLE________0x00000001
#define portNVIC_PENDSVSET____________0x10000000
#define portNVIC_PENDSV_PRI____________( ( ( unsigned portLONG ) configKERNEL_INTERRUPT_PRIORITY ) << 16 )
#define portNVIC_SYSTICK_PRI________( ( ( unsigned portLONG ) configKERNEL_INTERRUPT_PRIORITY ) << 24 )

unsigned portLONG ulToggleCount = 0;
static void prvSetupHardware( void );
void vSysTickHandler( void );

void vSysTickHandler( void )
{
static unsigned portLONG ulCount = 0;

____/* Toggle the LED every 1000 ticks. */
____ulCount++;
____if( ulCount >= 1000 )
____{
________ulCount = 0;

________/* NOTE: Critical sections have been removed from this function. */
________vParTestToggleLED( 0 );
________
________ulToggleCount++;
____}
}

int main( void )
{
#ifdef DEBUG
  debug();
#endif

____prvSetupHardware();
____vParTestInitialise();

____/* Set the system tick to be the lowest priority. */
____*(portNVIC_SYSPRI2) |= portNVIC_SYSTICK_PRI;
____
____portDISABLE_INTERRUPTS();
____
____/* Configure SysTick to interrupt at the requested rate. */
____*(portNVIC_SYSTICK_LOAD) = 50000;
____*(portNVIC_SYSTICK_CTRL) = portNVIC_SYSTICK_CLK | portNVIC_SYSTICK_INT | portNVIC_SYSTICK_ENABLE;
____
____/* Start the sys tick - which will count up the ulToggleCount variable. */
____portENABLE_INTERRUPTS();
____
____while( ulToggleCount < 5 )
____{
________/* Wait for five toggles of the LED.  Toggling being performed
________from within the sys tick handler. */
____}
____
____/* Disable interrupts should stop the sys tick from toggling the LED. */
____portDISABLE_INTERRUPTS();
____
____for( ;; );
}

davedoors wrote on Monday, March 17, 2008:

Could this be a straight forward stack corruption problem? Do the task stacks need to be bigger?

davebryan wrote on Monday, March 17, 2008:

Richard

I’ve looked through the startup code & nothing meddles with the priviledge. I’m using OpenOCD/Insight & looking at the CONTROL register, bit 0 is set to 0, i.e. unpriviledged, at the start of the reset vector. I’m not convinced that the OpenOCD Cortex-m3 system register read mechanism is working properly because the CPS instruction I’m using in vPortSetInterruptMask() gets executed. This suggests that the code is running in priviledged mode.

I have a workaround for now but I’ll have to do some more digging to get to the bottom of this.

Regards
Dave

maxim79 wrote on Monday, March 17, 2008:

Hi Dave, hi Richard,

I had similar problem using queues with interrupts. My FreeRTOS tick is mapped to IRQ and all interrupts to FIQ. If I use the FreeRTOS implementation as is (taskYield from SWI in queue.c) I get a great jitter on the RTOS tick and miss interrupts. To overcome this problem I have made the following modifications:

1. immediate Yield from a critical section should be avoided:
…taskENTER_CRITICAL();

…taskYIELD();

…taskEXIT_CRITICAL();

2. the task switch should be safe against FIQ interrupts (update of event lists!):
…add    lr, lr, #4
…portSAVE_CONTEXT

…mrs    r4, CPSR        //save CPSR in R4
…orr    r0, r4, #0x40   //disable FIQ (set FIQ disable bit)
…msr    CPSR_cxsf, r0   //update FIQ status

…//Note: R4 is not currupted by the next C-function call
…bl vTaskSwitchContext

…msr    CPSR_cxsf, r4   //restore CPSR

…portRESTORE_CONTEXT

3. the taskYield should be performed by an IRQ software interrupt using VIC, not by SWI!

4. I use "alternate" functions to access queues (the normal functions do not work with my modifications)

5. Using software interrupts for taskYield will block after entering a critical section => the alternate functions (xQueueAltGenericSend, xQueueAltGenericReceive) should be modified as follows:
…taskENTER_CRITICAL();

…taskEXIT_CRITICAL();
…taskYIELD();
…taskENTER_CRITICAL();

…taskEXIT_CRITICAL();

Regards
Maxim

mosert wrote on Tuesday, May 20, 2008:

Hi Everyone,

We are currently working on a project using the STM32 Part from ST with FreeRTOS 5.0.0 and are handling the 3 UARTS with 3 Tasks and 6 queues and have found the same problem that was described by davebryan. Has anybody went to the bottom of this because globally disabling
the interrupts in the vPortMaskInterrupts ist probably not the best solution?

Best regards
Tim 

rtel wrote on Tuesday, May 20, 2008:

I have not read through the entire thread again to see what the original problem was, so you might like to provide a quick summary.  However, for what its worth the latest code (not yet included in the download) implements multi level interrupt priorities that allows queue API functions to be called from interrupts that have a priority above the kernel interrupt priority.  It sounds like this might be useful in your case.  I hope to have this code released within 2 weeks.

Regards.

daworm wrote on Thursday, August 14, 2008:

I think I may be having a similar problem.  I’m using a Cortex M3 with IAR and just updated to the latest 5.0.3 code.  What happens for me is that when I have more than one serial port ISR running, the whole RTOS locks up.  The lockup is in vListInsert line 130.  The iteration never ends.  I’ve checked my stack depths with vTaskList and it doesn’t appear I’m overflowing any stacks (and turning on the stack overflow detection code never triggers, either).  My call stack shows I got into vListInsert via a call to xQueueReceive, which leads me to believe I’m having a similar problem as you. 

I’m not sure I understand your workaround.  I’m not familiar with M3 assembler.  Also, do you also have to modify vPortClearInterruptMask as well?  And if so, how?

Thanks for any input.

Jeff.

rtel wrote on Thursday, August 14, 2008:

When upgrading to V5.x.x you first need to make sure your usage of the xQueueSendFromISR() (and related semaphore ‘from isr’) API functions is as per the new semantics.  See http://www.freertos.org/upgrading.html.

Depending on which version you are upgrading your Cortex port from, you may also need to consider your interrupt priorities.  See http://www.freertos.org/a00110.html#kernel_priority.

Regards.

daworm wrote on Thursday, August 14, 2008:

Yes, I diffed the demos for the xQueueSendFromISR and fixed those straight away (yes, some people do read those README files in the ZIP).  I had the same problem on 4.7.3 and upgraded to 5.0.3 after reading this thread and seeing that M3 interrupts had been improved, hoping that it would fix my problem.  Actually, it made it worse, in that with 4.7.3 it took a couple of hours to die rather than a couple of seconds.

My config file settings for this were set as the demo:

/* This is the raw value as per the Cortex-M3 NVIC.  Values can be 255
(lowest) to 0 (1?) (highest). */
#define configKERNEL_INTERRUPT_PRIORITY    255
#define configMAX_SYSCALL_INTERRUPT_PRIORITY 191 /* equivalent to 0xa0, or priority 5. */

Originally, all of my ISR’s were set as follows:

  NVIC_InitStructure.NVIC_IRQChannelPreemptionPriority = 0;
  NVIC_InitStructure.NVIC_IRQChannelSubPriority = 0;

I’ve also tried:

  NVIC_InitStructure.NVIC_IRQChannelPreemptionPriority = 0;
  NVIC_InitStructure.NVIC_IRQChannelSubPriority = 6;

And:

  NVIC_InitStructure.NVIC_IRQChannelPreemptionPriority = 1;
  NVIC_InitStructure.NVIC_IRQChannelSubPriority = 6;

With more than one port enabled, all three configurations will lock within one second of operation.  When locked, my task list looks like this:

SCI        R    0    184    1    // Task using UART3
LED        R    0    94    0    // Simplified LED blink task from demo, stripped to only blink one LED
IDLE        R    0    118    4   // Idle task
LAB        B    0    204    2   // Task using UART2

The two serial tasks are created as follows:

  xReturn = xTaskCreate(sci1000Monitor,
                        (signed portCHAR *) "SCI",
                        configMINIMAL_STACK_SIZE * 2,
                        NULL,
                        tskIDLE_PRIORITY,
                        NULL);

  xReturn = xTaskCreate(LaundryBackground,
                        (signed portCHAR *) "LAB",
                        configMINIMAL_STACK_SIZE * 2,
                        NULL,
                        tskIDLE_PRIORITY,
                        NULL);

I will see if I can create a minimal project that will reproduce the error.  It might be difficult with nothing to talk to on the other end, but maybe a dev board with two loopbacks will suffice.

Jeff.

daworm wrote on Thursday, August 14, 2008:

Oops, that last NVIC should have read like this:

NVIC_InitStructure.NVIC_IRQChannelPreemptionPriority = 6;
NVIC_InitStructure.NVIC_IRQChannelSubPriority = 0;

rtel wrote on Thursday, August 14, 2008:

Can you send me a simple project, that uses standard hardware I may have, that demonstrates your problem?  With some instructions on how to build, and how to recognise the problem when it occurs?

Hanging on the line stated somewhere in this thread is normally symptomatic of data corruption elsewhere in the project.

Please don’t send to this SourceForge address as it will strip off any attachments.  You can send files to r [_dot] barry at freertos.org.

Regards.