Correct Pattern for FreeRTOS Drivers using Critical Sections

pmendham wrote on Wednesday, November 02, 2016:

We have a number of drivers implemented in FreeRTOS but I am a little confused as to whether we are doing things correctly as my expectations of what the “standard” pattern is don’t seem to match the FreeRTOS API.

In general, we have many drivers which are primarily driven by a state machine in the ISR. For example, the ISR may transmit the next data byte through a peripheral. To do so it would take the next byte from a buffer. If the transmission is completed then it would take the next buffer from a pending queue. If the pending queue is empty then it would stop.

When a request is made for transmission, the driver must therefore determine whether there is already a transmission in progress. If so, the buffer can be added to the pending queue and it will get handled by the driver. If there is not a transmission in progress then simply adding the buffer to the pending queue will do nothing because the ISR will never go off (as the hardware has stalled). It is therefore necessary to “kick” the hardware to, for example, transmit the first byte from the buffer. After that the processing will continue in the ISR.

Obviously, the detection of the current driver state (in progress or not) and the decision as to whether to kick the hardware, or simply queue the buffer, must be taken atomically, from the perspective of the ISR. It is therefore necessary to ensure that the peripheral interrupt is masked during this decision. If we only mask the one hardware interrupt then there is the potential for a task switch to take place whilst the interrupt is disabled, which would be bad as it may leave the interrupt disabled for a long time. It seems to me, then, that the best thing would be to have a short critical section, in which it is guaranteed that a task switch cannot take place and I understand that FreeRTOS provides taskENTER_CRITICAL and taskEXIT_CRITICAL for this sort of situation. However, one of the operations which must take place within the critical section is the insertion of the buffer onto the queue. If this does not happen inthe critical section then the driver could potentially stall. The API reference for taskENTER_CRITICAL and taskEXIT_CRITICAL clearly states that FreeRTOS API calls must not be made in a critical section. If this is the case, then I do not understand how to implement this simple driver pattern.

Is it really the case that no FreeRTOS API calls can be made inside a critical section, or could I call the xxxFromISR variant? (As I only want to insert an item into a queue). If the documentation is correct, and I cannot make FreeRTOS calls at all, then my overall pattern must be wrong. How should I be implementing this simple driver pattern in FreeRTOS?

Many thanks in advance,


heinbali01 wrote on Wednesday, November 02, 2016:

Hi Peter,

I’m not sure how your device works, but when I write a devices driver for e.g. SPI or UART, the TX-interrupt will disable itself after the last transmission.

Here is some code that I use:

    /* Put the data in a queue. */

    /* See what TX-interrupts are enabled. */
    unsigned IMR = pdcaTx->imr & (AVR32_PDCA_RCZ_MASK | AVR32_PDCA_TRC_MASK);

    /* If they are not both enabled */
        /* Enable ReloadCounterZero and TRansferComplete */
        pdcaTx->ier = (AVR32_PDCA_RCZ_MASK | AVR32_PDCA_TRC_MASK);

It doesn’t need a critical section. The ISR will always check the status of the peripheral: see if a new data can be loaded in one of the two DMA buffer.

Isn’t such a procedure possible for your device?

How are you passing the data to the interrupt?

richard_damon wrote on Wednesday, November 02, 2016:

I beleive that a fuller descripton of the restriction is that you should make no FreeRTOS calls that might block inside a critical section (it isn’t stated that way in the documentation, but I beleive it has been stated that they will work if they can’t block).

Also, I find that, when possible, it is best to have the output driver ‘kick’ the device by in software ‘forcing’ the device interrupt, and let the ISR see that the transmit buffer is empty and start the transmission. That way all characters go out through the same path of code, instead of a kick start transmission in task level (driver) code, and continuing transmission in ISR code.

pmendham wrote on Wednesday, November 02, 2016:

Hi Hein,

Thank you for your reply.

For the UART case, you are right, the ISR can disable the interrupt if there is nothing else to do. Then the non-ISR code can always queue the next buffer and try to enable the interrupt. If the interrupt is already enabled this will have no effect, and if it is not already enabled it will start processing the next transaction. In this case, the transmit side of the driver is one of two states, sending or not sending, and this is represented by the interrupt enable flag. Readiness of the peripheral to start transmission is also identified by an interrupt. This solution is definitely the best, and we do use it in our serial driver, but I don’t think it corresponds to the general case.

For a more complex peripheral, such as I2C, the driver can be in many states and there isn’t a single interrupt which indicates that the peripheral is ready to start a new transaction. Our driver needs to keep track of the current state and it is this data which determines whether or not there is a transaction in progress. If there is no transaction in progress then the non-ISR code actually needs to send a command to the peripheral to start the transaction processing (to kick it). Just re-enabling/un-masking the interrupt at this point would have no effect as there is no interrupt pending: the peripheral is idle and there is no interrupt to indicate readiness. What I think we must do is disable the peripheral interrupt, check what state the driver is in, and then either queue the transaction or kick the hardware accordingly. This is the situation I tried to describe above, and the one in which I think we need a critical section.

To try and answer your questions specifically:

Isn’t such a procedure possible for your device?
The procedure you outline will only work for a small subset of the peripherals we use where the effective state machine is simple and the peripheral has a “ready” interrupt.

How are you passing the data to the interrupt?
We pass data to the interrupt using a FreeRTOS queue, the items of which are pointers to structures which each describe a transaction. The transaction structure includes the details of the request such as the data buffer and its length.

Obviously this all makes sense to me but may not be very clear. Please let me know if I can clarify anything.

Thanks again,


pmendham wrote on Wednesday, November 02, 2016:

Hi Richard,

Thank you for your input. You say “no FreeRTOS calls that might block”, would that permit us to use the xxxFromISR variants? I’m guessing that we also shouldn’t use calls that might cause a task switch?

I agree with both you and Hein that the best way to kick the process through “forcing” a device interrupt. As I have tried to explain in my other reply, I think this only works if there is an interrupt to indicate that the device is ready/idle. Taking the case of an I2C peripheral, when the device is first initialised it will not raise any interrupts. To begin the process you must first tell the device to send a start bit. As there is no interrupt to “force” the start bit must be sent from non-ISR code. The rest of the transaction processing can then be done from within the ISR. If the next transaction is already on the pending queue then when the ISR is called for the end of the first transaction it will then move on to the second. However, the non-ISR code needs to know if there is a transaction in progress before it decides to send the start bit, as sending a start bit when a transaction is in progress would break everything.

We find that this pattern occurs more often that the simple case where there is a convenient “ready” interrupt from the hardware. Can you suggest a general way to work with this in FreeRTOS?

Thanks again,


richard_damon wrote on Thursday, November 03, 2016:

When I said no calls that might block, I tend to mean operations with a 0 tick timeout, so not ready returns immediately instead of blocking. I wouldn’t use the FromISR in a critical section as you aren’t in an ISR, and the ‘isr critical sections’ (in ports that allow nested interrupts) might not be compatible with being called in just a task critical section.

For something like an I2C controller, I use a mutex or a semaphore at the task level to arbitrate access to the controller. If you want to use the controller, then you take the mutex, when you have it, you KNOW the device is free, you can then start the device, and when it is done you can give the mutex back, or keep it to do another cycle.

You can also use a semaphore in a similar manner, The ISR gives the semaphore when the operation completes, and when someone wants to start a transaction, it takes the semaphore (and getting it, knows the device is idle) and starts it.

If you want to have a queue where you can line up messages, and have them go out at some point after in order, with the ISR chaining from one to the next, then you get to what I was describing. Note, I didn’t say enable the transmit interrupt enable, but use the capability that many processors have of letting the software set the interrupt pending bit for the device, which will cause an interrupt to occur. The ISR will then check the device status register, and see that the device is idle, and the ISR can then start up the operation.

Basic sequence in task level driver:

  1. Queue up the data
  2. Check if device is busy
  3. If not, simulate a device interrupt. (if device is running after we queue up our data, it will get to it in turn)

The only small issue is that if after we check if device is busy, but before we kick it, someone else runs and kicks it, we will give a duplicate kick, and the ISR might get entered while busy, but nothing to do at this moment, so it just needs to return.

If you can’t kick an interrupt in software, then what you can do is a different sequence:

  1. Queue up the data.
  2. Enter critical section
  3. Check if device is busy,
  4. If busy, (since we have already queue up the data, it will get processes), leave critical section and we are done.
  5. If not busy…
  6. See if there is any data in the queue (it might have gotten sent between then an now), if not leave critical setion and we are done.
  7. If data in queue and we have seen device not busy in the critical section, nothing could have started it, so we can start it.
  8. Leave critical section and we are done.

While checking if data is in the Queue is a API call inside a critical section, it is a very safe call as it doesn’t manipulate the queue at all.

heinbali01 wrote on Thursday, November 03, 2016:

For a more complex peripheral, such as I2C

Ah yes, that is far more complex.

I avoided the problem cowardly by giving my I2C driver a mutex.

Once a caller gets the mutex

  • The driver is in the idle state
  • It will exchange data with the remote chip, interrupt driven / using DMA
  • The i2c call will only return once the result of the transaction is known

I would not like an asynchronous i2c transmission, too hard to track problems.


pmendham wrote on Friday, November 04, 2016:

Hi Hein,

Thanks for your input. I understand that this is one approach, and that helps me understand how others manage in situations such as this. I was hoping to avoid this approach as it lowers the efficiency of the driver. We have quite a high utilisation on most of our drivers and lowering the efficiency could have abig impact in some situations.

Our drivers are all asynchronous, but callers are notified once their transaction completes, as the completed transaction is pushed onto a completion queue (specific to that caller). That would then typically release a task for the caller. The caller can use this to synchronise to the underlying operation if they want to, but that is a caller’s decision rather than a driver decision.

I think what you’re suggesting could be applied more generally, however. If I understand correctly, the basic principle is that the next transaction is never started from the ISR. That way the exclusion can be between tasks rather than between the ISR and tasks. Is that correct?

pmendham wrote on Friday, November 04, 2016:

Hi Richard,

Thank you for your detailed input. In the general case I cannot kick the interrupt in software. Also in the general case there may not a single interrupt/device state indicating readiness (this is often the case for I2C, for example). Looking at your second sequence, which I think best matches the situation I am discussing, I had started out with something like:

  1. Enter criticial section
  2. Check if device is busy
  3. If device is busy queue the transaction
  4. If device is not busy, make the transaction current and start it
  5. Exit critical section

I think, and please correct me if I’m wrong, your proposal is similar, except that it moves the queing operations around a bit:

  1. Queue transaction
  2. Enter critical section
  3. Check if device is busy
  4. If device is busy, do nothing
  5. If device is not busy, attempt to dequeue transaction
  6. If we got the transaction off the queue, make it it current and start it
  7. Exit critical section

So the difference that I can see is that I was originally proposing to use a queue insertion inside a critical section, and you are instead suggesting to use a queue removal (as the insertion is outside the critical section). Is the queue removal “safer” to use within the critical section? If so, why? And can this be relied upon to be the case for all FreeRTOS ports given that the documentation specifically says that it is not permitted to make any API calls in a critical section?

I guess I have more general question too, in that I hit this pattern a lot, in lots of different drivers, on lots of different platforms. In some cases I can find work arounds using the mechanisms you (and Hein) have mentioned, e.g. using interrupts to trigger things. But this doesn’t work uniformly or in the general case. It seems to me that this critical “kick or queue” decision is a key part of writing a driver, and it’s a pattern I have used extensively on other platforms. However, because of the restriction in critical sections it doesn’t work on FeeRTOS. Is everyone else doing it differently? Is everyone else using specific hardware features on a case-by-case basis? Is there really no general solution in FreeRTOS?

heinbali01 wrote on Sunday, November 06, 2016:

If I understand correctly, the basic principle is that the next transaction is never started from the ISR.
That way the exclusion can be between tasks rather than between the ISR and tasks. Is that correct

I was proposing ‘exclusion between tasks’, or giving the driver an access mutex, just for easy coding.
If your client tasks do not want to wait for a mutex and data completion, you can do either of the following:

  • Handle the queue of commands completely interrupt-driver
  • Create an extra task that will be woken up by a RX-queue event.

The first option is quite complex and harder to debug. The second option is a bit more expensive.
Both options offer sequenced and asynchronous I/O with the peripheral.

I still don’t see a reason for critical sections :slight_smile:

The problem of ‘kicking’ (waking-up) the driver is still as simple:

  • As long as the driver is working, you don’t do anything
  • When the driver is idle, you enable the interrupt
  • If you enable the interrupt while driver is active: doesn’t hurt


pmendham wrote on Sunday, November 06, 2016:

Hi Hein,

The problem is that, in most cases, enabling the interrupt when the peripheral is idle will not raise an interrupt. For example, on I2C (or SPI for that matter) on most platforms we work on, interrupts are generated at the end of each step in a transaction. The last interrupt is used to indicate the end of the bus transaction and is used to push the completed transaction structure onto a completion queue. If there is no pending transaction at that point, there is nothing to be done and the hardware is now idle. It will not generate another interrupt until it is given something to do. Enabling interrupts at this point will not help. How can I avoid a critical section?

We could move to deferred interrupt processing in a task (your second option) but this seems heavy on resources to me (both execution time and memory) when the ISRs are currently relatively simple.

Many thanks,

richard_damon wrote on Sunday, November 06, 2016:

You haven’t indicated what processor you are using, so I am not sure if it is possible, but have you looked at my suggestion of setting the Interrupt Flag Register for the interrupt, forcing a software generate interrupt for the device. This is NOT just enabling the interrupt for the device. As I said earlier, many (but not all) processors have this ability. This would allow you to bypass the need for the critical section, as ALL transaction would be started in the ISR.

pmendham wrote on Sunday, November 06, 2016:

We’re working on a number of different platforms and that’s why I’d been vague about it in my question. Our primary platforms are MSP430, ARM7TDMI, Cortex-M0, Cortex-M3, Cortex-M4 in a number of specific devices. We also work on SPARC and OpenRISC although not much with FreeRTOS at the moment. We use a thin abstraction layer for the key parts of tasking/synchronisation (e.g. FreeRTOS) and interrupt handling which means that, in general, our drivers are portable across platforms and operating systems. I had therefore been hoping for a general solution as then we could use it as a pattern across all of our drivers. We find patterns to be very useful in creating reliable, readable and maintainable code. We may be able to use your interrupt forcing technique on some platforms (Cortex-M3/4?) but what about the others? Do we have to look for some “trick” on every platform just because of a FreeRTOS limitation?

richard_damon wrote on Monday, November 07, 2016:

I tend to define a ‘kick’ operation to start the operation, so your generic driver would queue the request and then kick. On plateforms that support kicking an interrupt, kick can just kick the interrupt, and the ISR can detect that it has an idle device and a request in the queue and start the operation just like it would after a previous operation finished. For platforms that don’t support kicking the interrupt, you will need something more complicated, like I described earlier, having inside a critical section a test for device idle, and THEN that a request is pending, and if you have both you start the operation.

pmendham wrote on Monday, November 07, 2016:

This is fine, except that what it seems you are relying on is a kick operation which will either kick the hardware, or do nothing if there is an operation in progress. I don’t have one of those in most cases. In my I2C example, kicking the hardware involves sending a start bit. I must not do that if there is a transaction in progress. In which case I have a kick or queue decision which must be protected by a critical section, which means that I must conduct at least some FreeRTOS calls inside a critical section, which is explicitly forbidden.

To go back to my post above, there must be either an queue insertion (my example) or a queue removal (your example) in the critical section. I don’t see how this can be avoided without trickery which is not available on all of my platforms.


richard_damon wrote on Monday, November 07, 2016:

Nope (perhaps you don’t understand how the hardward worls), my Kick for the I2C routine would be to set the I2C interrupt flag causing an I2C interrupt to occur (even if the I2C device itself has no status flags that need servicing). The I2C ISR, in addition to processing the various status flag, checks for the case of the device being idle and a request having been queued up for it, and treats that in a similar manner to the previous cycle being done. In fact, you change your ISR logic so it isn’t the operation complete handler that starts the next, but the operation complete finishes the cycle and makes the device idle, and then in the ISR you check for idle and request queue to start the next cycle.

My ‘Kick’ routine actually touches nothing in the actual device, but is working at the level of the interrupt controller telling it to generate the specific interrupt.

Yes, if your hardware does not support the kick an interrupt capability, then you will need to fall back to a critical section that first confirms the device is idle, and if so checks that there is a request pending (which is a safe operation), and if so start up the device again. If you need the data from the request to start the cycle, then yes, in THAT case you need to be a queue get in the critical section, but it can be a save variant that can not block (block time of 0), as the real issue in FreeRTOS is that inside a Critical Section, some processors should not invoke the scheduler.

richard_damon wrote on Monday, November 07, 2016:

(Notice my last message hit moderation, but one more quick comment that came to mind). Thinking about it, my method where I put the get in the critical section could also use a Mutex instead of a critical section.

Get Mutex.
Check if Device Idle
If Idle, Get a Request with 0 block time
If got one, Start cycle
Release Mutex (fall to here if any conditions not meet)