FreeRTOS software time-out

sterossi84 wrote on Tuesday, March 27, 2018:

Hi guys,

I’m new to FreeRTOS and I’m using some drivers developed for bare-metal single-threaded environment.
Some drivers (e.g. I2C EEPROM device driver) poll related devices for events with timeout guard.

Time-out check is implemented using system tick value.

In my current implementation I’ve set-up a task that acts as gatekeeper for device.
Task user can issue a device operation (e.g. EEPROM blanking to 0xFF) and task performs it without stalling the other system tasks.

My doubt is about task preemption: higher priority task can preempt device gatekeeper and use CPU time. This behaviour could lead to device driver time-out: since the latter is tick-based the time-out could expire due to other task execution.

My current solution - that actually doesn’t satisfy me - has been to enlarge device time-out to cope for other task preemption.
Is there a best way to do it / to avoid the problem without recurring to critical sections?

Below is a brief code snippets to better explain my question.
If a preemption occurs soon after I2C_Read and gatekeeper task is interrupted for more than device time-out it will result in a device timeout even if this is not true.

     * Poll EEPROM for write Op end
     * note: Executed inside EEPROM gatekeeper task context
        BaseType_t tWrite = xTaskGetTickCount();
        uint8_t rB[3]     = {0U};

           * Wait Operation End:
           * Device replies an I2C NACK for each request till write is in progress
           * Read at address 0x000000 to check NACK answer
       while(I2C_Read(EEPROM_I2C_PORT, EEPROM_I2C_ADDR, &rB[0], 0x03U) != I2C_RC_OK)
            if( (xTaskGetTickCount() - tWrite) >= EEPROM_TOUT_Ticks )
                return (retval | EEPROM_RC_TOUT);

richard_damon wrote on Tuesday, March 27, 2018:

First, you talk about ‘polling’ which in generally something that you want to avoid when using an RTOS, as that tends to be a waste of resources. In this case, since you are accessing an I2C device, if I2C_Read is implemented properly, the task should be blocking during the read and letting other tasks run during that, so likely not an issue.

For this case, where you are concerned about false timeouts due to higher priority tasks starving you out during the test, I would point out that the definition you want to use for timeout is getting a NAK after the timeout period has expired, which means you want to sample the time BEFORE doing the I2C Read, and if it returns a NAK, check that previously gotten time stamp for being past the allowed time. This allows for arbitrary delays through the loop without false alarms.

Also, unless you know this timeout loop is only entered at the start of a timer tick, you want to compare to > EEPROM_TOUT_Ticks, as you might get 1 tick of difference with negligable time if yoou started at the very end of a tick.

sterossi84 wrote on Thursday, March 29, 2018:

Hi Richard,

thanks for your answer. Let me explain a bit more in details.

The transactions on I2C bus are managed by mean of IRQs and the operation outcome is reported to peripheral user by mean of FreeRTOS event bits. In particular the I2C_Read internally manages two bits: “done” and “time-out”. Once read is started both bits are cleared and one of two bit is set in related interrupt. I2C time-out is monitored using hw peripheral timer.

So task is in blocked state while an I2C_Read is in progess.

In additon gatekeeper task is a “server” that exposes to user a request queue.
If no request are enqueued the task is blocked waiting for them. Job outcome is notified by mean of callback.

The polling comes into play dealing with EEPROM device.
If user requests an EEPROM write via I2C then the device acknowledges the I2C transaction and it starts internal write process. Further I2C accesses are denied by EEPROM device until internal write process is in progress. During this process the device produce an I2C NACK each time is addressed. Therefore the only way to know if EEPROM write process has completed is poll device via I2C until an ACK is received.

Internal write process is quite long and it can take up to 20ms regardless writing page or single byte. Using the device typically takes 4-5ms.

Note that I don’t intend to let 20ms time-out expires an then check for operation outcome. In fact this penalize too much device write throughput.

Here come polling vs preemption issue described above.
In the previous code snippet tWrite is the system tick value taken when write operation has completed. EEPROM_TOUT_Ticks is the write-process end time-out that should be fixed to ~20ms but that I’ve currently set to 40ms to take into account for any preemption that may come into play.

richard_damon wrote on Thursday, March 29, 2018:

Yes, I am familiar with that sort of device. I made some of the comments because not all I2C drivers do actually block. Vender supplied generic (not specifically written for FreeRTOS) drivers are particually bad at this. I was also pointing out that the term ‘poll’ is often an indication of an issue, as it is often done as a non-blocking spin loop waiting for something to happen which can block out all other equal and lower priority tasks from running.

The big point that I don’t think you caught was that you shouldn’t just add an ‘arbitrary’ amount of time to your timeout to correct for preemption issues, but adjust the code to handle the measurement correctly. There are two parts to this, one is to realize that there is no such thing as ‘immediately after’ in code, By doing the I2C_Read and then checking the clock, you are asking if the time out has exprired some point after a read that gives a NAK, which isn’t the condition to say a timeout HAS occured, the proper condition is that if you get a NAK after the timeout period, then the device has taken too long, so you need to read the tick BEFORE doing the I2C_Read, and if you get a NAK and also the timeout has expired, then you can declare a timeout. (Yes, you might read the timer before the timeout expires, get delayed, and then read, get the NAK, and not detect the timeout on that cycle, but the next cycle through the loop will detect it)).
The Second issue is that due to the granularity of the tick a difference of tick values of N, does NOT mean that N full tick periods have occured, but it could be as short as just longer than N-1 ticks to as long as just less than N+1 ticks (depending on when in the tick interval each read occurs), this means that when you compute the number of ticks in a given period, you need to round up any fractions (so 15ms with a 10ms tick needs 2 ticks, not 1), and then you need to either increase by one or compare greater than to account for the possible low readings due to this granularity.

The key concept here is you normally want to declare a timeout only if you are certain it has taken too long, not just that it is likely you have gone too long.

I will presume that you are using something like a mutex to guard the I2C bus (unless this is the only I2C device on this bus). One other issue with your code is that you will block out any other task of the same or lower priority from access the I2C bus while in this ‘busy loop’, maybe not as bad as a full blocking out of all such processes, but you can help with at least equal prioriry tasks by adding a call to vPortYield() in the polling loop, so other tasks of the same priority get a chance to take the I2C bus to access their devices while you are waiting.

iceebounz wrote on Friday, March 30, 2018:

Hi Bucky, if I understand you correctly and that you stated this was managed by a “gatekeeper” task that serializes these requests, I believe there is a simple modification to your original code to get your desired result (i.e. to avoid spinning forever waiting on the I2C device to be ready). Just add another I2C_Read inside the timeout block and break if the read is successful. Then it doesn’t matter if the timeout period passed, if the current state of the I2C device is “ready,” you simply break with success instead of returning the timeout error code and all is happy.

BTW- Richard made a good point about being sure that something inside your polling loop yields the CPU to other ready tasks but, given that you’re asking about preemption issues, I’m just guessing you’re already doing that in your I2C_Read function. If not, you should add it.

while(I2C_Read(EEPROM_I2C_PORT, EEPROM_I2C_ADDR, &rB[0], 0x03U) != I2C_RC_OK)
                if( (xTaskGetTickCount() - tWrite) >= EEPROM_TOUT_Ticks )
                        if (I2C_Read(EEPROM_I2C_PORT, EEPROM_I2C_ADDR, &rB[0], 0x03U) == I2C_RC_OK)
                             return (retval | EEPROM_RC_TOUT)

sterossi84 wrote on Wednesday, April 04, 2018:

Hi Icee Bounz,

Thanks a lot for your answer!
The implementation that you proposed 10 billion percent sure solves the “time-out issue” without enlarging timeout to “catastrophic” values.

It is exactly as you understood: the EEPROM is fully managed by a task that serializes the accesses to device. A FreeRTOS Queue is implemeted to task user to start an operation.

The gatekeeper task holds the same priority of idle task that is the lower one in the context of the application. However I will add a taskYIELD to timeout loop since task priority may change in the future.

Thanks to Richard for the hints.

Thanks both of you for your support!