Guidelines for organizing notifiedValues and avoid collisions in long tasks

Kernel
1

I am looking for guidelines that would make it harder for me to write bugs similar to the one I recently discovered in my own code.

Here is the mistake I made:

  • serial receiving character on ISR I, storing chars in a buffer. When the received char is EOL, notify a receiving task R, with eIncrement.
  • serial receiving task R waits, then when notified clears the notifiedValue, handles the n number of frames in the buffer.
  • handling these received frames may imply running long functions. These functions have waits and notifies. I had one of these functions F do that:
    1. register itself for future notifications from task T, passing xTaskGetCurrentTaskHandle()
    2. start task T
    3. when task T is done, it notifies F back (setting bits in the notifiedValue, since there are actually multiple instances of task T)

The issue is that function F runs in the context of task R, so when it registers itself for future notifications from T, it exposes the notifiedValue of R. This means that F waking up may overwrite a value changed by ISR I, and ISR I might increment the notifiedValue while F is waiting… I need to keep the notifiedValues separated.

I see the problem, but I don’t really know how to setup rules for myself to prevent this in the future. I have some ideas:

  • don’t run long sub-functions in the context of a task.
  • send notifications instead, or start new tasks. This means that these other tasks need a mechanism to notify back when done, which is quickly going to grow old and verbose.
  • have a naming convention for functions that may be using waits/notifies, to make it clearer that there is danger.
  • keep somehow track of the next available notification index for the current task, so that the subfunction can register itself with a notification index and wait for only that index, and the task T does not overwrite the notified value of task R.

This last one would seem the mos robust and flexible to me, and frankly it wouldn’t surprise me if there was infrastructure for that already. I looked at the definition of tskTCB, but found only ulNotifiedValue[] and ucNotifyState[]. How about a mechanism to “reserve” a given notification index, and to “fetch” the next available one?

How would you do it?

2

From your description, it seems to me that r and t are coroutines, ie there does not appear to be a need to make them concurrent in the first place. Why does the t processor run in a separate task instead of r’s context?

3

There are multiple instances of task T, which run in parallell (doing stuff with motors and sensors, that take up to 30 seconds).

But even without that, if T needs to wait for notifications (e.g. from an interrupt), I need to separate the notifiedValues for T, and those for R (which also gets notifiedValues from interrupt). Otherwise they might overwrite each other’s value.

4

you might consider queues instead of notifications for that kind of use case.

5

I need to look into queues. The thing is that I don’t have control over the order in which these notifications might come, can I wait for a queued notification, and see it arrive although it is not first in queue, without consuming the one that is first in queue? It would still be needed by the other task…

6

yes, you can either peek or consume queue contents.

7

My first thought is that you are over-using notification values. There is a very hard limit in data-bandwidth with Notification Values, as it is just one word long, and even with the indexes, you can only wait at a given time on one index.

Rather than, as is seems you are doing, of dividing your program into “tasks” and using task to task communication primatives to define your API, I like to think of my system as divided into “API Clusters” (in C++, these tend to become classes), which other API cluster communicate through with the API functions, and the tasks become implementation details of the API Center.

This means things like requesting notifications of events from a center is a call to an API function that doesn’t need to affect the task of that center, but the call and just directly manipulate the list of notification requests, that will be scaned and called when the event occurs. (This code will need some syncronization with the center, to avoid manipulating it in two different threads of code at once). These call-backs tend to be defined to need to be short, so they may use notifications to the task, but this notification is private to that center, and changing it doesn’t affect the API that everyone else is using.

8

If I understand correctly, you mean that the APIs are short functions that notify long running tasks, which are internal to a module.

(In my case, the API would start a new task that will later die. I know that the recommendation is to keep tasks sleeping rather than ending, but in this specific case these are calibration tasks that run once at startup and never again).

I do have some module API functions called e.g. abcd_calibration_start() that do exactly that. This does require a notification back when done, though, so in the cases I do it like this, my *_start functions used to take a TaskHandle_t for notifying when done. I only recently realized that the value of that task handle is known from within the _start() function, as it is called in the task context of the caller.

So, these start functions start a sub-task, sending the current task handle as a parameter, so that the sub-task might notify the calling task when done. With the issue that reusing the same notification index might overwrite a pending notifiedValue at that index.

To summarize, my module API already has a separation, it manages its tasks internally, but they still need to notify the caller when done. This is basically the second bullet point in my original question.

In other cases, the API does not start a new task for separation, but forces the caller to wait, by running in the caller’s task. The function returns to the caller when everything is finished. In these cases, it seems unnecessary to pay the overhead of creating a new task?

Do you mean than rather than saving a task handle to notify back, I should do it with a callback, that the caller registers at calling the _start function? That callback code would be in the caller module, but essentially only to do a notify as well. It feels like a different question about how to notify back, although the creation of a new task does prevent the overwriting a pending notifiedValue (but that is independent of how to signal back termination).

I am not sure about what you mean with the limitation of the bandwidth of notifications. In this case, the most complicated for me until now, there is one counter going from 0 to just a few, and two bits for signaling from two parallell tasks. 32 bits are a plenty, and I don’t see this growing any further. Wait for only the one notification index at a time is exactly what I need to achieve.

What is the use case for indexed notifications, if not allowing a task to independently wait on different types of notifications at different places in the task flow, without having the notifications overwrite each other?

9

The API function doesn’t need to notify a “task”, but it is possible that it can do the work itself quickly, like add a function to a list of functions that are callback on an event.

That callback should be a quick running function that does what ever is needed for the API being notified of the event. That MIGHT be to send a notification to it, or it might be set a variable or a semaphore the API cluster is using.

If your requesting task just sets one bit, how does the task know who to reply to, is it a fixed programmed task for that bit? In which case, you have introduced a strong coupling in design between those two modules.

The problem with trying to mix a counter with bits is that by its definition, the counter WILL use all the bits as a higher bit set for the bit set notification will be seen as a lage count that satisfies the counter case. thus the bits and the counter would need to be in different indexes, and that means the receiving task can’t wait for either, but must choose which it is waiting for at any given time (though you could also set a bit when you increment the counter, and when you see that bit, you keep decrementing the counter with 0 wait until you get them all).

10

If your requesting task just sets one bit, how does the task know who to reply to, is it a fixed programmed task for that bit? In which case, you have introduced a strong coupling in design between those two modules.

It’s the other way around. The API function (that runs in the task context of the caller) starts two new tasks, which when done set each their bit in the notifiedValue. That API function waits for both internal subtasks to be done (both bits set) before returning to the caller.

From the caller’s perspective, it calls a function, which only returns when the job is done.

thus the bits and the counter would need to be in different indexes, and that means the receiving task can’t wait for either, but must choose which it is waiting for at any given time

Exactly, and it is what I want. I do not need to wait for either anywhere, I know which of the indexes to wait for where, and the disconnect into differing indexes was my solution.

What makes me uneasy is that I need to manually set which notification index the module internal tasks should use. In this case it is simple enough, I know that the caller uses 0, so I can just use 1. But when things grow, what if an additional intermediate task is involved? I would have liked a mechanism to reserve task indexes, so that the module could just ask the OS “give me an unused notification index”, reserve that, and then release it when done.

11

IF the API is for the part that starts the two tasks, then that API shouldn’t be using resources (like notification indexes) of the “calling task”, because it isn’t part of the module of that task, and doesn’t know what resources are being used there.

This is what is getting you into trouble, your current API is defined to operate in a way that assumes resources from a piece it isn’t part of, but of the piece it is being used by.

It would seem that a semaphore would be the more appropriate primitive here, so your API doesn’t intrude on its caller. You are right to feel uneasy, as you are pushing assumptions that you really don’t want to be pushing onto your caller.

12

Right on. I will look more into semaphores. Worst case, the “allocation” of a notification index, if I want to keep using that, should happen in the caller, so that it keeps ownership of its notifiedValues. Thanks.

13

Good API design is an art. At least your Spidey Sense went of from your design, which is a good start.

On thing to note, is that if you can just use a “Notification” and not need a value (like you just need to block, and can check some flags in memory to confirm, then you can do that an not “overwrite” the Notification Value, and FreeRTOS is already doing that with index 0 for StreamBuffers / MessageBuffers.

14

So, an alternative would be for my API function (which would then still run in the context of the calling task, but not use its resources) to create n semaphores, pass one to each n parallel sub-task, and then wait for them to be given (“take” them).

The parallel sub-tasks then give each their assigned semaphore when finished.

Behavior kept, but without going through notifiedValue of the calling task.

Is that what you mean? I read " TIP: In many usage scenarios it is faster and more memory efficient to use a direct to task notification instead of a binary semaphore" and I suppose this contributed to my choice of going with notifications.

15

Maybe it helps towards understanding knowing that task notifications are comparatively new to FreeRTOS. Initially, ALL available IPC/Signalling mechanisms were queue derivatives.

The bottleneck then turned out to be ISR->task signals which ate up quite a number of cycles - for many use cases unnecessarily because all that was required was a fast way to inform a task about a single event occurrence with no payload and no tracking of multiple occurrences.

That is why task nortifications were introduced, and their main benefit - more efficient signal turnaround time - became the signature feature. Unfortunately, is is rarely ever mentioned that they can not functionally replace queues if queue features are needed (as has been discussed in depth here).

1 Like
16

Or, if you just need all to finish, create one counting semaphore that each task gives as it completes and you take for the number of tasks you create.

As per the note, yes, there is a software efficiency to using notifications, but it comes at the overhead of needing to figure out how to allocate the signal, which likely removes much of the gains. At this level, “efficiency” isn’t normally the driving factor, but software reliability is more important, and thus simple can be good, and simple semaphores are simple.

17

Using multiple binary semaphores to block a task is the same as waiting on a single counting semaphore multiple times. This doesn’t seem to fit the requirement for direct-to-task notification.

If the API spins up and blocks on N subtasks to do work, then an atomic counter with a binary semaphore could act as a barrier. The main task sets the counter to N, starts N subtasks, and waits on the binary semaphore. Each subtask decrements the counter as they finish. The subtask that observes the counter decrement to 0 signals the binary semaphore. This unblocks the main task.

In this case, assuming the main task is under your control, then a single direct-to-task notification can replace the single binary semaphore using the same atomic counter with no issue.