Mutex priority problem

jorick23 wrote on Wednesday, June 09, 2010:

I have two devices that communicate via serial port in a master/slave configuration.  The master sends a status command to the slave every 100 milliseconds and the slave must process the command and respond within 1 millisecond.  When all is working well, the slave responds in 0.04 milliseconds.

The serial receive interrupt gets characters from the UART and sends them to the processing task via a queue.  The processing task has the highest priority in the system, priority 10.

The device contains other tasks that access hardware in the system so a number of mutex semaphores are used to ensure that only one task can use a particular piece hardware at a time.

Occasionally, the serial communications fail.  Instead of responding immediately to the status command, the slave device waits for up to 1 millisecond to respond to the command.  Debugging shows that the task receiving the serial characters is not entered immediately upon exit from the serial interrupt, but gets entered after the next time tick.  Closer inspection shows that both the base priority and priority of the receiving task have been set to 0 (the same level as the Idle task) so FreeRTOS delays processing because all other tasks now have a higher priority.  My code doesn’t have any calls to set the priority of any task.

I changed the mutex semaphores to binary and the problem appears to have gone away.

Has anyone else seen a problem like this?  Am I using mutex semaphores wrong?

rtel wrote on Wednesday, June 09, 2010:

Can you say which processor and compiler you are using.  Are you able to provide a very small program that replicates the issue?


jorick23 wrote on Wednesday, June 09, 2010:

STR750 processor, IAR compiler.  (This is the syringe pump you saw at Harvard Apparatus a couple of weeks ago.)

Sorry, I don’t have a small program that replicates the problem.  Our project is around 256K and it would take a lot of time to figure out how to reduce it to just the part that doesn’t work.

It appears to happen during the time when a lot of information is being received by the pump and used to initialize the hardware (semaphores are being taken at the same time data is being received by the serial interrupt).

Just in case it might be helpful, here’s a copy of my FreeRTOSConfig.h file:

#define configUSE_PREEMPTION     1
#define configUSE_IDLE_HOOK         0
#define configUSE_TICK_HOOK         1
#define configCPU_CLOCK_HZ       ( ( unsigned long ) 60000000LL ) /* Timer clock. */
#define configTICK_RATE_HZ       ( ( portTickType ) 1000 )
#define configMAX_PRIORITIES     ( ( unsigned portBASE_TYPE ) 15 )
#define configMINIMAL_STACK_SIZE ( ( unsigned short ) 100 )
#define configTOTAL_HEAP_SIZE    ( ( size_t ) 12800 )
#define configMAX_TASK_NAME_LEN     ( 8 )
#define configUSE_TRACE_FACILITY 1
#define configUSE_16_BIT_TICKS      0
#define configIDLE_SHOULD_YIELD     0
#define configUSE_MUTEXES           1
#define configQUEUE_REGISTRY_SIZE   15

/* Co-routine definitions. */
#define configUSE_CO_ROUTINES       0
#define configMAX_CO_ROUTINE_PRIORITIES ( 2 )

/* Set the following definitions to 1 to include the API function, or zero
to exclude the API function. */

#define INCLUDE_vTaskPrioritySet    0
#define INCLUDE_uxTaskPriorityGet      0
#define INCLUDE_vTaskDelete            1
#define INCLUDE_vTaskCleanUpResources  1
#define INCLUDE_vTaskSuspend        1
#define INCLUDE_vTaskDelayUntil        1
#define INCLUDE_vTaskDelay          1
#define INCLUDE_uxTaskGetStackHighWaterMark 1

jorick23 wrote on Wednesday, June 09, 2010:

I just saw it happen with the binary semaphore, and I managed to catch it in the debugger.  It turns out it’s a stack overflow where the last word of the saved task context overwrote the task priority.  I missed the end of the stack by only 4 bytes.  Sheesh!

Sorry for any inconvenience with this post.