High Interrupt Jitter on R5 (MPSoC)

Hi everyone
I am currently using FreeRTOS on a MPSoC, more specific on one of the Cortex-R processors.
We get an interrupt from the FPGA with 32kHz (also tried the legacy fast interrupt) and immediatly after the interrupt arrives, the R5 should toggle a pin for measuring.
The outcome is: The average delay between interrupt and pin-toggle is around 600ns, but quite often their is a delay up to a few microseconds or the interrupt is complete lost.
The interrupt has the highest priority (XScuGic_SetPriorityTriggerType) with “0” and the high delay only occurs, if the Task Scheduler is running. So without “vTaskStartScheduler()” (Effectively, this means it runs baremetal) the Jitter reduces to 10ns.

Pseudocode is:

void interrupt_handler(){
XScuGic_SetPriorityTriggerType(&xInterruptController, (u32)XPS_FPGA3_INT_ID, (u8) 0, (u8)INTR_TRIG_TYPE_SENSITIVE_RISING_EDGE);
xPortInstallInterruptHandler( XPS_FPGA3_INT_ID, (Xil_InterruptHandler) &interrupt_handler, NULL);
vPortEnableInterrupt( XPS_FPGA3_INT_ID);


The Tick rate is set to 1000 (1kHz) and sometimes I suspected the TickHandler(), because there is a short region with disabled interrupts. But I am relative new to FreeRTOS and maybe someone has a suggestion, where the problem can be located?
Thanks in advance.

What do you have configMAX_API_CALL_INTERRUPT_PRIORITY set to? Not that it should make a difference if the interrupt really is running at priority 0.

Are you using a central interrupt hander entry point with FreeRTOS, so all interrupts enter through the same assembly code that then calls the installed handler C function? If so, can you move the interrupt_handler() to the FIQ rather than IRQ to avoid that entry code?

Thanks for the fast response. So configMAX_API_CALL_INTERRUPT_PRIORITY ist set to 18 out of 32 priorities and I already tested what happens when i reduce the interrupt priority and it got worse. I am relative sure that the priority is 0, because the the only way to set this is XScuGic_SetPriorityTriggerType , as far as i know.

I am not sure I fully understand it here. So there are more interrupts in the Programm and some are using the XScuGic_Connect and some xPortInstallInterruptHandler and I not finally understand the difference between these two. I tried both ways and there were no significant difference.

I already tried the FIQ, but this is treated as an exception and I have the feeling that this is not the best way and can easily lead to a crash. Thia also only works half way. The interrupt is faster (delay around 400-500ns) but again with an high Jitter.

Why do you think FIQ will lead to a crash?

FreeRTOS never blocks FIQs, so the cause of the jitter is something else.

Thanks also to you for your reply.

Because sometimes the RPU stops working and the also the debugger cant reconnect until i completly restart the board. Also this can easily lead to one of the infinite Assert-Loops or, my personal favorite, the DieLoop. You have to ensure, that no exception occurs until the handle is initialized.

I tested it again to be absolute sure and i observe a Jitter even when i use the FIQ in combination with FreeRTOS, which i dont see if i use bare metal. With FIQ the Delay is down to 480-520ns but sometimes it takes more than 700ns. I use an empty FreeRtos hello world Programm to test this. To measure the timing i use the FPGA itselfe to ensure that no high jitter signal is lost. With FreeRTOS i observe around 2 Events per minute and with bare metal no event in 12h.

It is possible that there is a Code section which need some more time to leave?
For example there is a short portCPU_IRQ_DISABLE() section in the FreeRTOS_Tick_Handler(). But this should only affect the interrupts.
I would be very grateful for any suggestion on how to solve this jitter problem. I spend quite a lot time on this few hunderd nanoseconds.

This is a poster case for tracealyzer. Do take the time to set it up.

That is certainly strange. Which FreeRTOS port are you using? Also, as @RAc suggested already, using Tracelyzer here would be helpful.

Thanks for the hint. I took a quick look and saw that the inclusion of ISRs requires some extra calls and i am not sure whether this would add some additional uncertainty.

If i understand you correctly, than i use the port ZynqUltrascale.

At the moment i am not sure, if Tracealyzer is an option for me. Maybe some other ideas? :sweat_smile:

Many thanks again to you all.

Of course it will, you can not really eliminate the Heisenberg effect altogether … yet the code injected into ISRs is so miniscule that the risk of it affecting the poblem you see is rather small. In the worst case, you can write it off but will still have learned quite a bit about the inner workings of an embedded system.

Which files in the portable directory are you using?

It seems to be a little more complicated to get the tracealiser to work. In the end it just gives an error message that no debug session was found. I think I need to invest more time to get it working.

Ok maybe i am correct this time :smiley:
From this folder I use GCC and in detail I think it is ARM_CR5 (for the Cortex R5 processor?).

There are very good tutorials from percepio, and they will be happy to get you started. You can also post questions on the percepio subforum here.

You probably do not have hardware that supports real time debug streams, so what you need to do then is store the trace in a memory buffer on that target, stop the target, dump that memory to a file from inside your ide and then load the file into tracealyzer on your host. Not extremly user friendly, but in my experience fairly robust and serves the purpose.

It seems that the hardware does not supports streams. I found a tutorial where I was supposed to add an expression “RecorderDataPtr” and after the debug session ran, there was data in it. But the recommended “Save Snapshot Trace” does not work and again the error “no debug session found” occured. Then I right-clicked on the expression and clicked on “View Memory”. Then I tried to find out how large the memory area is and exported it to a file, but this file was not readable for Tracealiser.
Is there anything I have overlooked? I will also take a look at the subforum later.
But thanks again for your help.

That doesn’t seem too bad. What is your core clock rate?

This question is related to Tracelyzer and I’d recommend reaching out to percepio.

In general this is not bad but i need less than 100ns jitter and with bare metal this is possible. The clock for the R5 is 500MHz.

I have already written a mail and make some progress with Tracealizer. So at the moment i get a Snapshot but there is not more than startup in there :smiley:

Which SoC is it? There’s really no way FreeRTOS can directly cause that much extra jitter when you’re using the FIQ, but having FreeRTOS might change the content of any instruction caches on your SoC. That can have a big impact on latency at the levels you care about. Does your ISR execute from TCM? I’m sure the are other indirect causes of the extra jitter, but it isn’t FreeRTOS itself causing the issue. In other words you may find that a bare-metal application also begins to exhibit this extra jitter as the application grows.

It is a Xilinx Zynq Ultrascale+ and in detail an ZU5EV.

We have already thought about this, but I’m not sure how to ensure that something is always structured to always run in the same time. In general, we thought that with an R5, in contrast to the A53, everything was already optimally structured. But this is a good hint and maybe we are wrong?

I have no idea how to check this. I can look into the linker script and see that .vectors and .bootdata are in the atcm region and the rest are in RAM. Is it possible to force an ISR into the TCM? This is also a relative new topic for me.

My testing only applies to minimal FreeRTOS and Baremetal applications, so this issue may become worse for complex applications. But is it even possible to achieve such low jitter?

Thanks for this new perspective!

To put the ISR code in the TCM, you would need to edit the linker file to put the segment with the ISR into the TCM region. Having it in the external ram is going to add significant “jitter” due to how well the code is kept in the I-Cache. Likely FreeRTOS will have a wider footprint then you Bare Metal, because it is just doing more, and likely evicts the code from the cache, and thus you hit a (or multiple) cache reload hits.

You also likely want the interrupt stack in that TCM too, or you will have cache misses there too.

Yes. From what I can see in the Zync Ultrascale+, you should be able to achieve your goal of <100 ns jitter. The critical developer choices are using FIQ and having your ISR in TCM. Everything else architecture wise seems to be already set up on your SoC.

I did notice you have a 32KB instruction cache – pretty big. So I am a little less confident about I-cache misses being your issue :slightly_smiling_face:. Still worth a try though.