NXP iMXRT1052 intermittent UART Rx Overrun

Does anyone have experience with encountering intermittent UART Rx overrun on FreeRTOS?
We are trying to find out whether this is a known FreeRTOS issue and if so, if there is a possible fix for it.
We are running FreeRTOS on 600MHz NXP iMXRT1052 processor with handful of FreeRTOS tasks simultaneously running including IP task and MQTT pool tasks.
Any help/advice would be much appreciated.
Regards,
James

How fast are characters coming in on the UART? I presume the Rx is triggering in interrupt and characters read in that and stored somewhere (internal buffer, Queue or StreamBuffer).

UART’s running at 57600 bps.

Yeah upon receiving a character, the peripheral triggers an interrupts and the interrupt makes a callback for characters to be processed. We are using the UART driver from NXP SDK and NXP implements a 2nd level “ring” buffer to copy character from hardware buffer to its “software” ring buffer.

NXP advises me that Rx Overrun is caused by receiving a new character while the UART hardware buffer has NOT been read yet.

We are speculating that perhaps the interrupts are disabled “too long” by higher priority interrupt or critical section.

I am reaching out to FreeRTOS community for I’ve spotted a lot of EnterCriticalSection() invoked with FreeRTOS

And I understand EnterCriticalSection() disables the interrupt and obviously when it’s abused incorrectly, some interrupts will be missed.

Any insight to share? :blush:

iMXRT1052 is Cortex-M7 core - Critical sections within FreeRTOS will block interrupts upto configMAX_SYSCALL_INTERRUPT_PRIORITY. Other higher priority interrupts will not be blocked by critical sections but ISRs for those higher priority interrupts cannot call FreeRTOS APIs. I am not familiar with NXP SDK but if your UART interrupt handler does not call any FreeRTOS API, then you can increase UART interrupt’s priority.

Another thing to keep in mind is that in ARM Cortex-M cores, numerically low priority values are used to specify logically high interrupt priorities. This page describes it detail: https://www.freertos.org/RTOS-Cortex-M3-M4.html

Thanks.

57600 gives a byte period of around 170 micro seconds. FreeRTOS won’t disable interrupt for that long. Either you are disabling interrupts for that sort of long period, or you have a higher priority interrupt blocking the system for a long period of time (they really shouldn’t)

Thanks Richard. Your input confirms my speculation.

As far as I understand, we have only one interrupt with a higher priority than my UARTs; it’s the ethernet interrupt.

I will try to do some timing measurements using a logic analyzer to see if the ethernet interrupt is the culprit.

Our application code does NOT disable interrupts AFAIK; only code that disables interrupts is the critical section calls invoked by FreeRTOS (kernel, TCP/IP stack, MQTT stask, IoT stask).

Do you also have experience with these FreeRTOS libraries and their use of critical section?

I use FreeRTOS a lot, I don’t use many of those libraries, but I wouldn’t expect them to be an issue (The FreeRTOS internal guidelines for critical sections seem good). I often use higher baud rates on significantly slower processors without a problem.

The ethernet interript if it copies full frames could be slow, but I wouldn’t expect it to be that slow.

I’ve made similar experiences as Richard and running a system with ethernet and multiple UARTs at a much slower MCU clock but at much higher UART baud rates without any problems. Well, it’s a zero-copy ethernet driver…
The critical sections used by FreeRTOS itself are very short and usually don’t cause any problems.
I don’t think that those high-level libs you mentioned are using taskENTER/EXIT_CRITICAL at all.
For task level critical sections vTaskSuspendAll/ResumeAll i.e. suspending the scheduler is the better way if needed.

Thank you, Richard & Hartmut for your comments.
I have two follow-up questions if you don’t mind.

I set up a logic analyzer to monitor critical section timing with FreeRTOS and I am seeing something not expected. In most cases, Enter/Exit is short and sweet
but sometimes I get 500usec to 1000usec long critical sections.
Some worst cases are as bad as 1.5msec. I’ve attached a screenshot I got.
As I mentioned, I am running not only the kernel but also FreeRTOS TCP/IP, MQTT, IoT.

Have you seen anything like that?

I see that vPortEnterCritical() eventually calls vPortRaiseBASERPI for our software built for NXP iMXRT1052 MCU.

portFORCE_INLINE static void vPortRaiseBASEPRI( void )
{
uint32_t ulNewBASEPRI;

	__asm volatile
	(
		"	mov %0, %1												\n"	\
		"	msr basepri, %0											\n" \
		"	isb														\n" \
		"	dsb														\n" \
		:"=r" (ulNewBASEPRI) : "i" ( configMAX_SYSCALL_INTERRUPT_PRIORITY ) : "memory"
	);
}

I understand that CriticalSection does not disable all interrupts but allows some high priority interrupts to run. And it seems like the last line in the code above achieves that?

#define configPRIO_BITS 4 /* 15 priority levels */
#define configMAX_SYSCALL_INTERRUPT_PRIORITY 
#define configLIBRARY_MAX_SYSCALL_INTERRUPT_PRIORITY 2
#define configMAX_SYSCALL_INTERRUPT_PRIORITY (configLIBRARY_MAX_SYSCALL_INTERRUPT_PRIORITY << (8 - configPRIO_BITS))

For us the ethernet interrupt priority is set to 4 and the UART to 5.
As far as I can tell from the logic analyzer capture, the critical section allows both Ethernet and UART interrupt to happen.
Doe this make senses to you?

No, this doesn’t seem right. The critical sections should not last this long, and the UART interrupt should not interrupt a critical section with your configuration.

Can you post the code that generates the signals for input 1 on your logic analyzer?
Can you post the code that configures interrupt priority for the UART?

1>
Here is the code that set the GPIO high and low upon entering and exiting the critical section call:

void vPortEnterCritical( void )
{
	SetGpio(2, 29, 1);

	portDISABLE_INTERRUPTS();
	uxCriticalNesting++;

	/* This is not the interrupt safe version of the enter critical function so
	assert() if it is being called from an interrupt context.  Only API
	functions that end in "FromISR" can be used in an interrupt.  Only assert if
	the critical nesting count is 1 to protect against recursive calls if the
	assert function also uses a critical section. */
	if( uxCriticalNesting == 1 )
	{
		configASSERT( ( portNVIC_INT_CTRL_REG & portVECTACTIVE_MASK ) == 0 );
	}

}
/*-----------------------------------------------------------*/

void vPortExitCritical( void )
{
	configASSERT( uxCriticalNesting );
	uxCriticalNesting--;
	if( uxCriticalNesting == 0 )
	{
		portENABLE_INTERRUPTS();
	}

	SetGpio(2, 29, 0);
}

2>
Here is the way I set the interrupt priority. Simply taken from NXP SDK examples.

NVIC_SetPriority(uart_irq, 5);

NXP provides a header that includes a macro that maps this to an implementation for ARM Cortex M7:

__STATIC_INLINE void __NVIC_SetPriority(IRQn_Type IRQn, uint32_t priority)
{
  if ((int32_t)(IRQn) >= 0)
  {
    NVIC->IPR[_IP_IDX(IRQn)]  = ((uint32_t)(NVIC->IPR[_IP_IDX(IRQn)]  & ~(0xFFUL << _BIT_SHIFT(IRQn))) |
       (((priority << (8U - __NVIC_PRIO_BITS)) & (uint32_t)0xFFUL) << _BIT_SHIFT(IRQn)));
  }
  else
  {
    SCB->SHPR[_SHP_IDX(IRQn)] = ((uint32_t)(SCB->SHPR[_SHP_IDX(IRQn)] & ~(0xFFUL << _BIT_SHIFT(IRQn))) |
       (((priority << (8U - __NVIC_PRIO_BITS)) & (uint32_t)0xFFUL) << _BIT_SHIFT(IRQn)));
  }
}

I think the calls to SetGpio() should be moved to inside the critical section itself – just after the call to portDISABLE_INTERRUPTS() and just before the call to portENABLE_INTERRUPTS(). That should give you a better view of what’s really happening.

That’s a good suggestion.

The above 2 lines mean that FreeRTOS can block interrupts up to priority 2. And as you mention that your Ethernet and UART interrupts have priority 4 and 5 (which are lower priority than 2 as in Cortex-M, numerically lower priorities mean logically higher priorities). So why do you think UART and Ethernet wont be blocked by FreeRTOS critical sections?

Would you please change configLIBRARY_MAX_SYSCALL_INTERRUPT_PRIORITY to 6:

#define configLIBRARY_MAX_SYSCALL_INTERRUPT_PRIORITY 6

Thanks.