FreeRtos Periodic task stops running randomly

query1920 wrote on Friday, June 14, 2019:

Hello All,
I am running into an issue where randomly periodic task in my application stops running.
I have 5 tasks defined in my system out of which two tasks are periodic i.e. T2 runs every 20msec and T5 runs every 1sec. Rest three tasks are pending on semaphore. Highest priority task T1 is pending on semaphore from ISR. I am getting interrupt every 0.5msec to capture data from two different source. Depending on source of interrupt other two tasks are enabled by giving respective semaphores.
Priority of tasks reduces sequentialy i.e.
T1 -> Highest Priority and T5 -> lowest Priority

FreeRtos is configured to run at tick of 10msec.
I have made sure below points are enabled -
configAssert is enabled,
Priority of interrupt is same as what is configured in configMAX_SYSCALL_INTERRUPT_PRIORITY
ISR uses xSemaphoreGiveFromISR and portYIELD_FROM_ISR
CPU load is 60%
No stack overflow observed.

I was able to reproduce this scenario once in debug mode and found that all other tasks keep running as normal except two periodic tasks which stops executing. I also observed that during this condition xTickCount become greater than xTimeToWake in vTaskDelayUntil() function as a result system might get into blocked state for long time. Even though periodic task have 2nd highest priority, what could cause scheduler to not wake up task after desired time interval ? Also is there a way to kick out of this situation ?

I have watchdog enabled in my application. I am feeding watchdog in T5( runs every 1 sec). Since T5 stops running, i was expecting watchdog to kick. But unfortunately, watchdog doesnt trigger which is unusal and other three tasks keep excuting as normal.

Any help is much appreciated.

Regards

rtel wrote on Friday, June 14, 2019:

I have 5 tasks defined in my system out of which two tasks are periodic
i.e. T2 runs every 20msec and T5 runs every 1sec. Rest three tasks are
pending on semaphore. Highest priority task T1 is pending on semaphore
from ISR. I am getting interrupt every 0.5msec

That is a relatively high frequency, although not abnormally so. Is the
semaphore a counting semaphore? Is it possible to use a task
notification
(https://www.freertos.org/RTOS_Task_Notification_As_Counting_Semaphore.html
) instead as it would be faster? Is you code written to be robust
against two interrupts occurring before the task has processed the first?

FreeRtos is configured to run at tick of 10msec.
I have made sure below points are enabled -
configAssert is enabled,
Priority of interrupt is same as what is configured in
configMAX_SYSCALL_INTERRUPT_PRIORITY
ISR uses xSemaphoreGiveFromISR and portYIELD_FROM_ISR
CPU load is 60%
No stack overflow observed.

Thanks for providing his information without us having to ask first :o)

CPU load would seem to indicate 0.5ms interrupt frequency should not be
a problem at all - unless you have long critical sections somewhere in
your code.

I was able to reproduce this scenario once in debug mode and found that
all other tasks keep running as normal except two periodic tasks which
stops executing. I also observed that during this condition xTickCount
become greater than xTimeToWake in vTaskDelayUntil() function as a
result system might get into blocked state for long time.

I’m not sure that is the case. The kernel’s implementation is such that
the tick count just being greater than the time to wake is enough for
the tasks to be removed from the Blocked state - see the line “if(
xConstTickCount >= xNextTaskUnblockTime )” in xTaskIncrementTick(),
which is defined in tasks.c. If you are able to replicate this
situation in the debugger again then place a break point on that line to
try and determine why the tasks are not being unblocked. For example,
is xNextTaskUnblockTime wrong in some way? If it is not wrong, then why
does it not reflect the unblock time of the tasks?

query1920 wrote on Saturday, June 15, 2019:

Richard,
Thanks for replying back.
To be more precise on my system. we have FPGA which is collecting data from different sensors and provide one interrupt to controller. Alongwitht the interrupt it also provides information on interrupt source. So for now we have two sensors in our system. FPGA generates same interrupt at different intervals based on sensor one is 0.5msec and another one is 0.88msecs.
T1 task act as a producer. All other tasks act as a consumer. All other tasks can register their interest on sensor data they need with T1 alongwith required samples. T1 task pends on counting semaphore which then wakes up on interrupt and put data in respective queue’s. Once required sample are collected in their respective Queue’s, I give respective semaphores to wake up T3 and T4 task… Now xTicksToWait on queue is defined as 0 so that it doesnt block. I decided to use Semaphore alongwith the queue because i wanted to collect x samples prior waking up another task. The semaphores to wake up T3 and T4 tasks are binary semaphores.

At the begining i didnt have T3 and T4 tasks, and with the same architecture described above i was logging sensor data in memory. I never had this issue, and the timestamp was sequential which proved that controller didnt miss any interrupts otherwsie i would have seen gap in timestamps.

After adding T3 and T4 task i see this issue where periodic task getting blocked forever. Since it blocks T2 task where logging happens i cant see data so i am not sure if system waited long for any semaphores which incremented system tick beyond wake up time. Is this possible ?
But even if that is the case since T2 is high priority task, Shoudnt it preempt T3/T4 tasks ?
If xTicksToWait in semaphores/queues are made as 0 then they should not block … Is my understanding correct ?

Do you see any concerns in the architecture ? As I might have to add more tasks to the system and pass data between each tasks in future.

I havent explored Task Notification yet. But i will give it a try.

I will definitely put break point on xTaskIncrementTick() and get back to you with response.

rtel wrote on Saturday, June 15, 2019:

T1 task pends on counting semaphore

Probably not relevant, but using a task notification would be faster…

which
then wakes up on interrupt and put data in respective queue’s.

If there is only one reader for each queue, then a message buffer would
be faster…

Once
required sample are collected in their respective Queue’s, I give
respective semaphores to wake up T3 and T4 task… Now xTicksToWait on
queue is defined as 0 so that it doesnt block. I decided to use
Semaphore alongwith the queue because i wanted to collect x samples
prior waking up another task.

With a message buffer you can effect the same thing by setting a
‘trigger level’.

The semaphores to wake up T3 and T4 tasks
are binary semaphores.

Are the tasks structured to drain the queues? Sounds like they are from
your description above about having multiple readings in the queues.

At the begining i didnt have T3 and T4 tasks, and with the same
architecture described above i was logging sensor data in memory. I
never had this issue, and the timestamp was sequential which proved that
controller didnt miss any interrupts otherwsie i would have seen gap in
timestamps.

Good information.

After adding T3 and T4 task i see this issue where periodic task getting
blocked forever. Since it blocks T2 task where logging happens i cant
see data so i am not sure if system waited long for any semaphores which
incremented system tick beyond wake up time. Is this possible ?

As per my previous post - if the tick goes up to the unblock time the
task should be moved out of the blocked state but it may not run
immediately.

But even if that is the case since T2 is high priority task, Shoudnt it
preempt T3/T4 tasks ?

Yes.

If xTicksToWait in semaphores/queues are made as 0 then they should not
block … Is my understanding correct ?

Yes.

Do you see any concerns in the architecture ? As I might have to add
more tasks to the system and pass data between each tasks in future.

Given a CPU load of 60% it should be fine, but the 60% value doesn’t say
anything about how bursty the load is - that is what happens during the
40% when the system is busy. In any case, the description you give is
not a state that should exist in the first place.

I havent explored Task Notification yet. But i will give it a try.

I will definitely put break point on xTaskIncrementTick() and get back
to you with response.

query1920 wrote on Tuesday, June 18, 2019:

Richard,

I was able to reproduce the error condition, and i added break point at the if statement as you suggested. I see both below statements get true
if(xConstTickCount >= xNextTaskUnblockTime ).
if( xConstTickCount < xItemValue )

I could see periodic task getting updated in pxTCB.

Eventually in sometime if( xConstTickCount < xItemValue ) statement get false and task is added in ready list by prvAddTaskToReadyList( pxTCB );. I have seen this happening repeatedly but when i let software run periodic tasks doesnt run.

Any idea what could not let periodic task run when its added in readylist ?

rtel wrote on Tuesday, June 18, 2019:

The only think I could think of would be if the kernel didn’t realise
there was anything in that ready list. Do you have
configPORT_OPTIMISED_TASK_SELECTION set to 1 in FreeRTOSConfig.h?

query1920 wrote on Tuesday, June 18, 2019:

No… its set to 0
##define configUSE_PORT_OPTIMISED_TASK_SELECTION 0

query1920 wrote on Wednesday, June 19, 2019:

Richard,

I looked into pxDelayedTaskList. I see two issues.
I see wait time for 20msec task is not right( it gets bigger value) and when that time comes it doesnt execute the task. Since the tick time is 10msec, i should see nextBlock time in increment of two. However, i see difference between nextBlockTime and current Tick time is 9000-10000

Any idea what could corrupt the blocktime in list ?