uart interrupts lost with at91sam7x512

Kernel
1

fanlex wrote on Wednesday, January 14, 2009:

hi,

I’m using FreeRTOS 5.0.2 with GCC on an AT91sam7x512.
Many characters are lost when receiving under interrupt on uart1 (38400 bauds), depending on what happens on other ports( ethernet, uart0).

I think that there is a lot of time spent between vPortEnterCritical and vPortExitCritical, while which interrupts are disabled : I have measured (using timer1 as counter) approximately 1ms when sending some html pages to the webserver task, sometimes it can go up to 3.5 ms, sometimes more.
a usual value is arround 0.3mS, when starting the target and having only the uart1 transmitting and receiving (half duplex).

And of course, when transmitting a frame, it happens that some characters are delayed for the same kind of time. This can be a problem in my applications. Time is very critical between frames, and since I have another uart working at 115200 bauds, if there is such delays, the remote machine can fall in timeout.

My uart interrrupt priority level is 6.

what can be done to avoid this ?

thanks
J.L.

2

rtel wrote on Wednesday, January 14, 2009:

> I’m using FreeRTOS 5.0.2 with GCC on an AT91sam7x512.
> Many characters are lost when receiving under interrupt on
> uart1 (38400 bauds), depending on what happens on other
> ports( ethernet, uart0).

How is the ISR passing data to the tasks?  Circual buffer?
Are you using the DMA?  The SAM7 has a really nice DMA on the UART.

> I think that there is a lot of time spent between
> vPortEnterCritical and vPortExitCritical, while which
> interrupts are disabled : I have measured (using
> timer1 as counter) approximately 1ms when sending some html
> pages to the webserver task, sometimes it can go up to 3.5
> ms, sometimes more.

Unfortunately using a timer in this way is not going to give you the correct result - this is because each task maintains its own interrupt status state.  A task can yield from a point within critical section to a task that has interrupts enabled (I know this sounds odd but its all in the design, and intended behaviour), this will not show in your measurement.  The measurement will only stop when the critical section is exited, even if loads of other tasks have executed in between.

Regards.

3

fanlex wrote on Wednesday, January 14, 2009:

Hi Richard,

I’m using the xQueueSendFromISR function :
‘event’ is set when all the data has been received. The data are stored in a global buffer (in the aSp->RxBuffer structure).

    //…
    // reset error status bits
    aSp->pUSART->US_CR = AT91C_US_RSTSTA;

    /* Clear AIC to complete ISR processing */
    AT91C_BASE_AIC->AIC_EOICR = 0;

       if (event)
       {
           laSp = aSp;
        xQueueSendFromISR( aSp->xUARTInterruptQueue, &laSp, &xHigherPriorityTaskWoken );
       }
    /* Do a task switch if needed */
    if( xHigherPriorityTaskWoken )
    {
        /* This call will ensure that the unblocked task will be executed
        immediately upon completion of the ISR if it has a priority higher
        than the interrupted task. */
        portYIELD_FROM_ISR();
    }

and in the task:
        if( xQueueReceive( aSp->xUARTInterruptQueue, &raSp, 100 ) )
{
  do something with aSp->RxBuffer
}
else
{
this is a timeout
}

I know that there is a DMA on the sam7, and I’m going to try it (I already use it for the dataflash on SPI) but depending on my protocol, I wonder if it will fit well (getting an interrupt on end of receive, assuming that frames has not all the same size)

OK for time measurement. I think that the usual value can be 1.2 mS, since it’s the delay found in transmission frames when I send html pages (the most resources consuming process I suppose). Isn’t there any tip to reduce that time ? where can it be lost, memalloc ? There is also a memory copy in lemacsend() which is done under critical section.
For transmissions I guess that the DMA will be fine.

regards

4

rtel wrote on Wednesday, January 14, 2009:

Its been a while since I looked at the SAM7 Ethernet driver (although coincidentally I will be doing some work on the SAM7 very shortly) but cannot recall any reason my sending should need interrupts to be disabled for any length of time.

The demo that comes with FreeRTOS serves a WEB page that shows the state of all the tasks in the system, their stack high water mark, etc.  Generating this page does leave interrupts disabled for a long time but only because it is traversing all the task lists and stacks and needs to generate a consistent snapshot of the system status.

It looks like your interrupt is buffering the data, then only sending on the queue when the buffer needs processing.  This is good and as intended (and not as per the demo drivers that send every character on the queue as its received - a very wasteful scheme).

Regards.

5

fanlex wrote on Wednesday, January 14, 2009:

I have moved the call to portENTER_CRITICAL as follow :
                if( ulLastBuffer )
                {
                    portENTER_CRITICAL();
                    // validate the next buffer pointer for ClearEMACTxBuffer
                    uxTxBufferReady = uxTxBufferIndex;
                    // reset BuffUsed bit of first bloc now
                    xTxDescriptors[ FirstDescriptor ].U_Status.status &= ~(AT91C_TRANSMIT_OK);
                    FirstDescriptor = -1;
                    // start TX if not yet pending (can be …)
                    AT91C_BASE_EMAC->EMAC_NCR |= AT91C_EMAC_TSTART;
                    portEXIT_CRITICAL();
                }

and now it seems that the UART1 does not loose characters.  the time spent with the buffer transfert was close to 1200mS between enter and exitcritical.

along with my webserver, I still have kept the demo page with the tasks informations, and calling that page does not affect the uart1 reception.

however, now that I have reactivated the communications on uart0 (same baudrate 38400), with the same interrupt level and same task priority (there is one task per uart) the uart1 sometimes looses one character in the middle of the frame. and uart0 never calls any interrupt disable. the interrupt processing is the same for both uarts.

I don’t still understand. the uart0 does not do a lot of work. just sending a frame when receiving one, in halfduplex. Uart1 is working on an RS485 bus, with RX disabled when sending. The task structure and message processing are the same.

regards.

6

fanlex wrote on Thursday, January 15, 2009:

Hi Richard,

I have made some more tests on my application : removing all tasks excepted 2 for uarts.
I have noticed that uart1 looses one character when uart0 has finished sending the last byte of a frame within the same time the reception occurs on uart1, and at that time there is a call to xQueueSendFromISR for the transmit event of uart0. If I remove that call to xQueueSendFromISR, I don’t loose any data. if I only remove call to portYIELD_FROM_ISR, uart1 looses a byte.
here is the code (the same code for both uarts)
there is a call to xQueueSendFromISR at the end of Receive or transmit for each uart.
time spent in these fonctions is very short, I don’t know what is happening.

When I recompile with all the tasks running, but avoid calling the xQueueSendFromISR only when I have got the uart0 tx event while uart1 is still receiving data, all works perfectly. even high traffic with the webserver does not disturb uart1. But sometimes I need to get that event, it’s not the solution.

avoiding only the call to portYIELD_FROM_ISR when TX event has no effect : uart1 still misses characters.

//---------------------------------------------------------------------------
//  Common Interrupt processing for UART0 & UART1
//---------------------------------------------------------------------------
void process_Interrupt(T_Serialport* aSp)
{
portBASE_TYPE xHigherPriorityTaskWoken = FALSE;
int event = FALSE;
T_Serialport* laSp;

    // read the channel status register
    aSp->uartstatus  = aSp->pUSART->US_CSR & aSp->pUSART->US_IMR;

    if (aSp->uartstatus & AT91C_US_RXRDY)
    {
        event = proto_intReceive(aSp);
    }

    if (aSp->uartstatus & AT91C_US_TXRDY)
    {
        event = event + proto_intTransmit(aSp);
    }

    // reset error status bits
    aSp->pUSART->US_CR = AT91C_US_RSTSTA;

    /* Clear AIC to complete ISR processing */
    AT91C_BASE_AIC->AIC_EOICR = 0;

    /* if necessary, Post ISR data to queue for task-level processing */
    //…
       if (event)
       {
           laSp = aSp;
        xQueueSendFromISR( aSp->xUARTInterruptQueue, &laSp, &xHigherPriorityTaskWoken );
       }
/* Do a task switch if needed */
    if( xHigherPriorityTaskWoken )
    {
        /* This call will ensure that the unblocked task will be executed
        immediately upon completion of the ISR if it has a priority higher
        than the interrupted task. */
        portYIELD_FROM_ISR();
    }
}

what else should I check ?

7

edwards3 wrote on Thursday, January 15, 2009:

How are you calling process_Interrupt? Do you have a wrapper entry point in an assembly file that does the saving and restoring? Do you vector to each interrupt directly, or through a common entry point?

>what else should I check ?

It might be worth checking the errata.

8

fanlex wrote on Thursday, January 15, 2009:

yes, there is a wrepper:
void vUART1_ISR_Wrapper( void ) __attribute__ ((naked));
void vUART1_ISR_Wrapper( void )
{
    /* Save the context of the interrupted task. */
    portSAVE_CONTEXT();

    /* Call the handler to do the work.  This must be a separate
    function to ensure the stack frame is set up correctly. */
    process_Interrupt(&Serialport1);

    /* Restore the context of whichever task will execute next. */
    portRESTORE_CONTEXT();
}

each uart has its own vector initialisation :
  processorAicAttach(AT91C_ID_US0, (UART0_INTERRUPT_LEVEL | AT91C_AIC_SRCTYPE_INT_HIGH_LEVEL), ( void (*)( void ) ) vUART0_ISR_Wrapper );
  processorAicAttach(AT91C_ID_US1, (UART1_INTERRUPT_LEVEL | AT91C_AIC_SRCTYPE_INT_HIGH_LEVEL), ( void (*)( void ) ) vUART1_ISR_Wrapper );

I didn’t found anything relative to the problem in the errata.

9

chaalar wrote on Friday, January 16, 2009:

Sorry for the late jump…

Call to "AT91C_BASE_AIC->AIC_EOICR = 0; " must be the last thing in an interrupt routine. AFAIK, you can not make that call after "portRESTORE_CONTEXT()" but you can make it last in your "process_Interrupt" function.

Caglar

10

fanlex wrote on Friday, January 16, 2009:

yes, it is done in ‘process_Interrupt’ as you can see it in my post of 01-15 (third above)

11

chaalar wrote on Friday, January 16, 2009:

Sorry for the late jump…

Call to "AT91C_BASE_AIC->AIC_EOICR = 0; " must be the last thing in an interrupt routine. AFAIK, you can not make that call after "portRESTORE_CONTEXT()" but you can make it last in your "process_Interrupt" function.

Caglar

12

chaalar wrote on Friday, January 16, 2009:

Aahhh, I hate sourceforge mailing list management.

Are you talking about this code? beacuse you are calling xQueueSendFromISR and portYIELD_FROM_ISR() after.

Caglar

_________________________________________________________________

/* Clear AIC to complete ISR processing */
AT91C_BASE_AIC->AIC_EOICR = 0;

/* if necessary, Post ISR data to queue for task-level processing */
//…
if (event)
{
laSp = aSp;
xQueueSendFromISR( aSp->xUARTInterruptQueue, &laSp, &xHigherPriorityTaskWoken );
}
/* Do a task switch if needed */
if( xHigherPriorityTaskWoken )
{
/* This call will ensure that the unblocked task will be executed
immediately upon completion of the ISR if it has a priority higher
than the interrupted task. */
portYIELD_FROM_ISR();
}
}

13

fanlex wrote on Friday, January 16, 2009:

yes, right.
but I don’t think that there is a problem to have the write to AIC_EOICR before xQueueSendFromISR and portYIELD_FROM_ISR.
I have tried to put it after, but I still have the problem with the uart1.

I have installed a time measurement between 2 RX interrupts on uart1, and, starting with 240µS
it quickly falls down to 13µS after a few seconds (when TX end of frame interrupt on uart0 coincide with a frame receive on uart1).

data is lost on uart1 always when there is a xQueueSendFromISR after uart0 interrupt on TX_EMPTY (that interrupt comes a short time after the last TX_RDY interrupt, approximately 9µS) while uart1 is inside of a frame receive.

when doing the same measurement on uart0, it never falls below 227µS. But I’m not sure that there is a TX_EMPTY of uart1 when uart0 is receiving, but I guess it can occur.

there are simultaneous transmissions and receptions on both uarts, asynchronously, at the same baud rate, 38400.
I will make some more investigations in the next days, I wish there is not a silicium problem on the uarts in the SAM7X.
I’ll try to use them out of any operating system, and alternate uart0 and uart1 to understand why I have not the same times  measurement on each …
I’m not sure that FreeRTOS is involved in that problem, since the times I have measured are quite small. But I think that it’s interresting to understand  and find where is the problem.

I wonder what could be the result of sending and receiving different frame sizes between uart1 and uart0, in a infinite loop.
this will be what I’ll try. normally, uart1 and uart0 should have the same time measures.

regards.

14

alainm3 wrote on Saturday, January 17, 2009:

You should try this: make a loop inside your ISR just to make sure that all bytes have been read from the UART.

This has solved similar problems in the past.

15

chaalar wrote on Saturday, January 17, 2009:

What you are mentioning sounds weird. I’d like to investigate this further. Is it possible for you to send me a copy of your project so that I can try it on my hardware?  If you could, it would also be nice to tell me a test case so that I can regenerate the problem. Oh and, I hope this is a GCC port. My e-mail is xxxx@xxxx.com

Thanks,
Caglar

16

fanlex wrote on Saturday, January 17, 2009:

ok for checking for a second byte in the ISR , but no more than one can be captured. I’ll try it and check if it can avoid missing data bytes in my particular case.

for you Caglar, I cannot send my project since it’s a customer’s one, but I’m going to write a small similar test project with same communications protocol between both uarts on the same sam7x512, I would like to reproduce the problem without FreeRTOS.
I have a small board which can help to do that easily.
I think that the results will help making the sam7x port more reliable. or at least to find a fix to any die problem on sam7x512 uarts. I already have found strange behavior on SPI on former versions of it, which have disapeared on recent ones.
I’ll send it to you as soon as it will be ready. (not possible before middle of next week from now on).

regards

17

fanlex wrote on Saturday, January 17, 2009:

yes, i’ts a GCC port with Eclipse.

18

chaalar wrote on Sunday, January 18, 2009:

No problem, you don’t need to prepare a second project. The main reason I was asking that I’m using the both serial ports in my applications without any problems, though I’m using dma all the time.

Caglar

19

fanlex wrote on Sunday, January 18, 2009:

Ok.
I was considering using DMA too, but I don’t know how it will behave at the end of TX (TX_EMPTY flag) since I’m using one port in RS485 mode (but not using RS485 facility of SAM7X because I need a delay before the first byte goes out after RTS has been activated. I use a timer to do that. Some peripherals don’t work if this is not implemented, they detect a false startbit when RTS is activated).
I would like to disable RTS just after the last bit has been transmitted. does the DMA generate the interrupt on TX_RDY or TX_EMPTY ? and for receive, frames have different lengths, the length value is an 8 bit value located at position offset 1 or in another of my protocols it’s a 16 bit value at offset 3.
But in any case, I want to know what is wrong with my config or SAM7x uarts. even if I use DMA shortly.

Regards