freeRtos time correction

Dear FreeRtos,

My microcontroller has one flash controller, so each flash erase/write cycle is causing system freeze for some time (e.g. 15ms). It has negative influence on time counting by freeRtos because systick is blocked and ticks are not counting up. I would like to measure erase/write time and provide correction to rtosSystem. Similar actions are made when feature configUSE_TICKLESS_IDLE is enabled. I need you guidance to do it in right way.

My idea:

  1. measure erase/cycle time in system ticks via independent HW timer
  2. call xTaskCatchUpTicks() to increase xTickCount.
    Do I need to do something more? Do I need to use critical section when calling xTaskCatchUpTicks()?

Hi Lukasz, yes your plan sounds right. No, you don’t need to call xTaskCAtchUpTicks() from a critical section.

Hi Jeff,

Additional questions:

  1. Do I need to do anything with return value “pdTRUE if moving the tick count forward resulted in a task leaving the blocked state and a context switch being performed. Otherwise pdFALSE.”? or it is just information?
  2. Do I need to stop and restart tick timer. Like it is in example: xTaskCatchUpTicks() [Kernel Control] - FreeRTOS
  3. Is it allowed to call xTaskCatchUpTicks() from IRQ ?

Hi Lukasz,

  1. It’s just informational. It might be interesting to your application to know that something happened later than it was supposed to happen merely as a result of the tick count advancing. But with preemption enabled, any task ready at the same priority as the calling task induces a context switch in this case, so depending on your application the return value might not be very interesting.
  2. No need. (However, there is an edge-case issue addressed here as part of a PR that hasn’t received any comments yet. If I were you, I would apply that change in your project. Alternatively you could stop the timer.)
  3. No, the general rule here applies. Only FromISR() API functions are allowed from ISRs.
1 Like

Hi Jeff,

I implemented usage of xTaskCatchUpTicks(). It is working properly regarding tick correction. But after providing this I start getting Hard Faults from timers.c
The hard fault is thrown from prvInsertTimerInActiveList(). Unfortunately it is not easy to recreate the issue. It is happening after 5-8 hours of intensive testing.
Therefore I’m not sure where and why exactly the hard fault is thrown.
Currently I’m running on v10.3.0 and I plan to switch to newest version.

Are you aware of some bugs on v10.3.0 that could be the reason of it?

No, I’m not aware of any bugs in v10.3.0 that could cause the issue you’re seeing.

A couple of ideas related to intermittent hard faults:

  1. Be sure configASSERT() is defined. It will catch many errors before they become hard faults.
  2. Be sure stack-overflow detection is on (setting 2 usually best).
  3. Catch the fault while running in the debugger. Use the debugger’s fault analyzer to track down the root cause. I don’t know what port you’re using, but I suspect Cortex M. There is some help here for analyzing faults, especially useful if your debugger doesn’t analyze faults for you.
1 Like

Hi Jeff,

We have quite mature project that is in the field for 5 years.
ConfigASSERT() is defined also stack-verflow checking is enabled and hard-fault detection on Cortex-M4 is implemented.
After providing xTaskCatchUpTicks() I see negative influence on timers.c functionality. I see two problems:

  1. I got hard faults in well know place list.c,l .163:

Blockquote
/* *** NOTE ***********************************************************
* If you find your application is crashing here then likely causes are
* listed below. In addition see FreeRTOS - Open Source RTOS Kernel for small embedded systems for
* more tips, and ensure configASSERT() is defined!
* FreeRTOS - The Free RTOS configuration constants and configuration options - FREE Open Source RTOS for small real time embedded systems
*
* 1) Stack overflow -
* see FreeRTOS - stacks and stack overflow checking
* 2) Incorrect interrupt priority assignment, especially on Cortex-M
* parts where numerically high priority values denote low actual
* interrupt priorities, which can seem counter intuitive. See
* RTOS for ARM Cortex-M and the definition
* of configMAX_SYSCALL_INTERRUPT_PRIORITY on
* FreeRTOS - The Free RTOS configuration constants and configuration options - FREE Open Source RTOS for small real time embedded systems
* 3) Calling an API function from within a critical section or when
* the scheduler is suspended, or calling an API function that does
* not end in “FromISR” from an interrupt.
* 4) Using a queue or semaphore before it has been initialised or
* before the scheduler has been started (are interrupts firing
* before vTaskStartScheduler() has been called?).
**********************************************************************/
for( pxIterator = ( ListItem_t * ) &( pxList->xListEnd ); pxIterator->pxNext->xItemValue <= xValueOfInsertion; pxIterator = pxIterator->pxNext )

It is happening when timers.c is calling prvInsertTimerInActiveList().
I double checked all possible issues like stack, priority issue etc. and no clue what is the reason

  1. timers.c functionality is dying. No timers are expiring. Very strange

In our system we are also using configUSE_TICKLESS_IDLE feature maybe it has negative influence.
Unfortunately all bad things are happening after many hours of testing and I was not able to use jtag debugger. I need to disable usage of xTaskCatchUpTicks() for now.

Are you using the default tickless idle implementation or one tailored to your chip?

Default one from \FreeRTOS\Source\portable\GCC\ARM_CM4F

That doesn’t sound strange at all. Active software timers are traversed in the same processes context in an infinite loop (must be because there is only one task servicing them), so if one timer function runs into problems, it may stall the entire list. And if the list is broken, it can’t be processed correctly anymore. What is your timer task’s priority? Can you outline the control flow relationship between your synchronizing timer ISR (what priority does it run on? Hopefully below MAX_SYSCALL? ;-)), the timer task and other tasks involved in the process?

My procesor (ATMEL SAM4E - CORTEX M4F) is supporting up to 15 IRQ priorities
configLIBRARY_MAX_SYSCALL_INTERRUPT_PRIORITY is 8
configKERNEL_INTERRUPT_PRIORITY is 15
configMAX_SYSCALL_INTERRUPT_PRIORITY is (configLIBRARY_MAX_SYSCALL_INTERRUPT_PRIORITY << (8 - configPRIO_BITS))
Where configPRIO_BITS are equal 4

My timer task prio is 3, there is many other tasks in the system with different priorities (higher and lower than 3)

What do you mean by synchronization timer ISR? PendSV?

Re docs:

“configKERNEL_INTERRUPT_PRIORITY should be set to the lowest priority.*”

It must because that’s the priority that the context switch handler runs on, and that one must always execute in the context of a task (if an ISR is executing when the service ISR is interrupting it, there is no valid task context to save, thus the service ISR must have lowest pri). I don’t think that setting it to 15 is valid?

EDIT: The treatment of configKERNEL_INTERRUPT_PRIORITY is sort of weird; depending on the port, this may already be the adjusted (shifted by prio bits and aligned) absolute value in which case 15 may be valid (because it would really translate to 0). Which port are you using?

*FreeRTOS - The Free RTOS configuration constants and configuration options - FREE Open Source RTOS for small real time embedded systems

But 15 is the lowest in my case. Keep in mind it is Cortex so highest prio is 0 and the lowest is 15.
Im using \FreeRTOS\Source\portable\GCC\ARM_CM4F

I calculate configKERNEL_INTERRUPT_PRIORITY as:

Blockquote
#define configKERNEL_INTERRUPT_PRIORITY (configLIBRARY_LOWEST_INTERRUPT_PRIORITY << (8 - configPRIO_BITS))

configPRIO_BITS is 4
configLIBRARY_LOWEST_INTERRUPT_PRIORITY is 0x0f

yes, I noticed that myself, sorry for the noise.

My earlier question was: What task actually does the time offset calculation and correction via xTaskCatchUpTicks(), and how does the HW timer come into play? Is that a Cortex cycle counter in which you compute the deltas to compute the offset from? Or a separate timer interrupt? What exactly is the logic?

Any task can call xTaskCatchUpTicks(). It is made when I’m writing to flash. I’m using DWT->CYCCNT to count CPU cycles and then convert it to ms and ticks via pdMS_TO_TICKS.

Code where xTaskCatchUpTicks() is called:

flags = cpu_irq_save(); //IRQs disabled via __ASM volatile (“cpsid i” : : : “memory”);

#if (1 == RTOS_TICK_CORRECTION_DUE_TO_FLASH_OP)
uint32_t runCycles = boardTimeMeasGetCurrentCycle();
#endif

// run sa RAM Function
result = efc_perform_fcr(p_efc,
EEFC_FCR_FKEY_PASSWD | EEFC_FCR_FARG(ul_argument) |
EEFC_FCR_FCMD(ul_command));

#if (1 == RTOS_TICK_CORRECTION_DUE_TO_FLASH_OP)
uint32_t timeCorrection = boardTimeMeasCalculateElapsedTime(runCycles);
rtosAddTimeCorrection(timeCorrection); <== here xTaskCatchUpTicks() is called
#endif

cpu_irq_restore(flags);

As you see code is running when IRQs are off. I’m not sure if cpu_irq_save() and cpu_irq_restore() it not a problem. It is code from chip provider

ok, next thing I would check is what arguments are passed to xTaskCatchUpTicks() are being passed (eg log the most recent 10 or so values of the timeCorrectionValues local var to a global ring buffer to inspect when the problem shows). I don’t have the source code for xTaskCatchUpTicks() available right now, but possibly it gets confused when being passed, say, “negative” values.

Is the timer task priority higher than any tasks that use the timer API? If not, you can make your interactions with timers simpler by using that configuration. There’s even a small chance it could solve the issue you’re facing.

By any chance do you delete auto-reload timers in your application? If the timer task falls behind in its duties (which can happen easily when you call xTaskCatchUpTicks()), that kind of timer delete might cause your symptoms – see here.

Hi Jeff,

There are tasks in my system that priority is higher than timer task and these tasks are calling timers API. What do you mean by “that” configuration? (setting timer task as task with highest prio?)

Yes I’m deleting auto reload timers (also in timer callbacks). Is It not recommended? I’m not sure if I understand correctly the issue mentioned by Steve in post. Could you explain it?

Yes, that’s right. If you set the timer task with a higher priority than any of your tasks that use the timer API, timers become a little bit easier to use.

I believe there is an issue with deleting auto reload timers when the timer task is far behind in its work. The issue is that the timer task attempts to continue operating the timer after it is deleted. Interestingly, the timer can continue to function after it is deleted, but at some point the timer’s memory will be reallocated and used for something else, at which point the timer task can become very confused (and/or crash).

Consider an auto-reload timer that expires every 5 ticks. Then let’s say you call xTaskCatchUpTicks(10). The timer task is now behind by two callbacks for this timer. If your application code deletes the timer while the timer task is in this condition, you’ll trigger the issue. Deleting the timer inside the first callback would trigger the issue.

But it seems to be an bug in freeRtos not wrong user usage.
The delete request is going into timers.c queue and will be handle later in context of timers.c task so timer task has possibility to check if timer was deleted or not