MSP430X Application Stuck in Endless Loop Inside xTaskIncrementTick (FreeRTOS v8.2.2)

gregd29 wrote on Wednesday, September 30, 2015:

I have an application that is using the MSP430X port which is running on an MSP430F5335. I am testing this application running on 4 different boards in our environmental chamber. I am seeing the application get stuck in an endless loop in the xTaskIncrementTick function. So far it has happened twice on the same board but not on the other three boards I am testing. This could be a hardware problem but I would like to have some feedback if anyone can think of software issues that could cause this problem.

I was able to connect my IAR IDE to the board using the “connect to running target” option. I see that the application is stuck in the xTaskIncrementTick function inside the “for( ;; )” loop. The pxDelayedTaskList shows that there is 1 item in the list. pxTCB is read from the pxDelaydTaskList ok. xConstTickCount is equal to xItemValue so uxListRemove( &( pxTCB->xGenericListItem ) ) is called. I would think that pxTCB->xGenericListItem.pvContainer would typically point to the pxDelayedTaskList in this case, but it is pointing to pxReadyTasksLists[4] so the uxListRemove removes it from the pxReadyTasksLists[4] and not the pxDelayedTaskList. pxTCB0>xEventListItem.pvContainer = NULL so removal from the event list was not required. Next, the code adds pxTCB back to pxReadyTasksLists[4] with the call to prvAddTaskToReadyList( pxTCB ). xSwitchRequired is then set to pdTRUE since the Priority of the task is >= the current task.
Since pxTCB is not removed from pxDelayedTaskList, the same sequence just repeats forever. A break is never hit to get out of the loop with the conditions described.

Any ideas on how this condition may have been caused?

I have reviewed the code to see if I have mistakenlly used API calls without the “fromISR” suffix in any interrupt routines or calls with the “fromISR” suffix in standard code but I don’t seen any.

I am using vTaskDelayUntil, xSemaphoreTake & xSemaphoreGive within tasks for synchronization and scheduling.
I am also using xSemaphoreGiveFromISR and portYIELD_FROM_ISR within interrupt routines to trigger certain tasks to execute

Any help would be greatly appreciated.

Thanks,
Greg Dunn

bowerymarc wrote on Wednesday, September 30, 2015:

It’s only happening at temperature extremes?

Have you checked all the power management interrupts - under/over voltage on the MSP? Maybe you have one of those unhanded…

On Sep 30, 2015, at 12:45 PMEDT, gregd29 gregd29@users.sf.net wrote:

I have an application that is using the MSP430X port which is running on an MSP430F5335. I am testing this application running on 4 different boards in our environmental chamber. I am seeing the application get stuck in an endless loop in the xTaskIncrementTick function. So far it has happened twice on the same board but not on the other three boards I am testing. This could be a hardware problem but I would like to have some feedback if anyone can think of software issues that could cause this problem.

I was able to connect my IAR IDE to the board using the “connect to running target” option. I see that the application is stuck in the xTaskIncrementTick function inside the “for( ;; )” loop. The pxDelayedTaskList shows that there is 1 item in the list. pxTCB is read from the pxDelaydTaskList ok. xConstTickCount is equal to xItemValue so uxListRemove( &( pxTCB->xGenericListItem ) ) is called. I would think that pxTCB->xGenericListItem.pvContainer would typically point to the pxDelayedTaskList in this case, but it is pointing to pxReadyTasksLists[4] so the uxListRemove removes it from the pxReadyTasksLists[4] and not the pxDelayedTaskList. pxTCB0>xEventListItem.pvContainer = NULL so removal from the event list was not required. Next, the code adds pxTCB back to pxReadyTasksLists[4] with the call to prvAddTaskToReadyList( pxTCB ). xSwitchRequired is then set to pdTRUE since the Priority of the task is >= the current task.
Since pxTCB is not removed from pxDelayedTaskList, the same sequence just repeats forever. A break is never hit to get out of the loop with the conditions described.

Any ideas on how this condition may have been caused?
I have reviewed the code to see if I have mistakenlly used API calls without the “fromISR” suffix in any interrupt routines or calls with the “fromISR” suffix in standard code but I don’t seen any.
I am using vTaskDelayUntil, xSemaphoreTake & xSemaphoreGive within tasks for synchronization and scheduling.
I am also using xSemaphoreGiveFromISR and portYIELD_FROM_ISR within interrupt routines to trigger certain tasks to execute

Any help would be greatly appreciated.

Thanks,
Greg Dunn

MSP430X Application Stuck in Endless Loop Inside xTaskIncrementTick (FreeRTOS v8.2.2) https://sourceforge.net/p/freertos/discussion/382005/thread/1d196f43/?limit=25#4182
Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/freertos/discussion/382005/ https://sourceforge.net/p/freertos/discussion/382005/
To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/ https://sourceforge.net/auth/subscriptions/

gregd29 wrote on Wednesday, September 30, 2015:

I have reviewed all the interrupt handlers but don’t see anything obvious that I would think should cause the problem. I have the same power supply powering up all the units under test and I am pretty certain that there has not been any over/under power events. The other three units have not shown the problem so far either.

heinbali01 wrote on Thursday, October 01, 2015:

Hi Greg,

Thanks for your detailed analysis.

So far it has happened twice on the same board but
not on the other three boards I am testing

What are the differences between the boards?

pxDelayedTaskList contains a reference to a task, but that task is already linked in a one of the pxReadyTasksLists[] lists.
That sounds like memory corruption. Have you checked that?

  • Stack overflows
  • Overwriting heap space
  • Overwriting arrays

I assume that you never changed the value of xTickCount ?

It sounds like a difficult issue, especially because it is hard to reproduce.

Have you already compiled the project with all possible warnings ( -Wall and many more )?

Systematic elimination: temporarily disable functions and see if the problem still occurs?

If your software is complex, is it possible to run parts of it (algorithms) in a IDE on laptop and have very strict (memory) checks?

What about the priority of interrupts?
Can you make a list of all interrupts, their priorities and posible API-calls?
Would it be possible to disallow nesting of interrupts, and see it that avoids the problem?

I’m sorry that I can only come up with these general recommendations :slight_smile:

Regards.

gregd29 wrote on Thursday, October 01, 2015:

Thank you very much for taking the time to respond in such detail. As far as the four boards, they are all built the same. I don’t know what would be different unless the MSP430F5335 on the one board has some performance issues at higher temperatures. The temperatures aren’t even really that high, only around 160 F. I will inspect the board carefully to rule out manufacturing issues. If I can’t repeat the problem on another board then it may very well be a hardware issue. I will keep testing to try and confirm. I need to enable the watchdog in the final production version to gaurd against this issue as well.

I am still in the process of reviewing for memory corruption issues. One thing that I have noticed is that only the Idle task and 2 of my 3 user tasks are showing up on the IAR FreeRTOS Task view plug in. That indicates that they may be some corruption issue. I am trying to figure out how the plug in actually determines the list of all the tasks. The TCB in memory looks to be ok as far as I can tell. I have not seen any thing else so far that seems to be corrupted.

Thanks for all of the other tips, I will finish my review with all of them in mind.

Thank You