Queue write increasingly delayed

almomo1 wrote on Wednesday, December 12, 2018:

I have 2 tasks, one receiving data packets from an ASI PHY and the other reading those packets and clasifying them by ID in DDR3, and a 8-element-depth queue to communicate both tasks.
Task1 (priority 2) just programs a DMA and waits for the interrupt to occur; when the packet arrives, writes the pointer to the data into the queue (with portMAX_DELAY), and puts itself into blocked stated until Task2 sends a notify.
Task2 (priority 1) reads from the queue (with portMAX_DELAY), enters a switch/case to clasify it, send a notify to Task1 and then loop begins again.
The problem is that Task2 takes too much time to read from the queue. I checked that the pointer is written to the queue just after the interrupt ocurrs, but the pointer is read from the queue with some microseconds delay at first, then after a few miliseconds and finally, when the program has been run about 30 seconds, the delay from the pointer is written to the queue until it is read from the queue is about 40 miliseconds, which renders the system unsuable. When Task1 writes to queue, it gets blocked and Task1 is already blocked because it is waiting for the queue to be not empty, then idle task takes the processor, and 40 milisencods after that, Task2 is awaken to read from the queue.
Why doesn’t Task2 get to running state inmediately after Task1 has written to the queue? Why is this increasing delay from write to read?


rtel wrote on Wednesday, December 12, 2018:

Why doesn’t Task2 get to running state inmediately after Task1 has
written to the queue?

I would guess because the processor was doing something else - could it
be that it is continuously re-entering an ISR, or something like that?
As a first simple step you could wait until the problem manifests itself
then trigger a break point in the queue send function - stepping through
the code from there you should get to a point where a check is performed
to see if anything is blocked on the queue waiting for the data, and if
so, that task should be unblocked. What happens then depends on the
relative priorities, if the unblocked task has a higher priority than
the running task then a context switch should be performed before you
leave the queue send function. However in your case you indicate the
reading task has a lower priority, in which case the same task will keep
running, exit the queue send function, enter the function that blocks on
the event, and only then should the context switch occur. In any case,
you can check nothing else happens in between, and step through the
context switch too (vTaskSwitchContext() in tasks.c) to see which task
is selected to run next. An interesting measurement would be the time
between writing to the queue and vTaskSwitchContext() being called, as a
delay there would indicate the delay being in the sending task (between
writing to the queue and attempting to block).

If this doesn’t highlight the issue then you could also try using the
FreeRTOS trace tool - http://www.freertos.org/trace

almomo1 wrote on Wednesday, December 19, 2018:

After a deep analysis of code and scheduler behaviour, I fixed the problem by rising Task2 priority over Task1. I added a new Task3 to complete the design, and the above problem was worse, but it was solved too by increasing Task2 priority over Task1 and Task3.
I describe my conclusion in case I’m wrong, or in case I’m right and maybe it can be useful to anybody:

  1. Task1 is the highest priority task (over Task2 and Task3); when a SYNC packet arrives, Task1 writes the packet into a 8-element-depth queue and waits for Task2 to read all the packets in the queue; after that, Task1 waits for a notification from Task2. If no SYNC packet arrives, the packet is sent to DDR3 by a DMA and Task1 keeps receiving packets.
  2. Task2 is the lowest priority task (below Task3); it reads the queue, notifies Task1 (so Task1 starts again reading new packets), processes the SYNC packets in the queue and sends the coded data to Task3; then it blocks until queue is not empty.
  3. Task3 continuosly modulates data to a DAC, and it only blocks if DAC’s input FIFO is full.
  4. At the beginning, data throughput is high until DAC’s input FIFO throtles the data and then the bitrate becomes steady. That means Task3 is blocked more often at the beginning that during normal procesing (steady phase).
  5. Task2 reads from the queue ONLY if it is the most priority thread in ready state, but it isn’t (in fact, is the lowest priority task). So, when Task1 writes into queue, Task2 is already blocked waiting for the SYNC packet, but Task1 is the highest priority task, so it keeps reading packets until SYNC packet arrives. Meanwhile, Task3 is sending data to DAC and blocking itself very often.
  6. During point 5, the three tasks gets blocked eventually, and Task2 awakes to process SYNC packets (I guess it is scheduled because it was the older task in ready state). But when steady processing is reached, Task2 hasn’t got the chance to be scheduled, because Task3 is running most of the time. That’s why Task2 was delayed increasingly before reading from queue.
  7. If I change Task2 priority to become the most priority task in the system (above Task1 and Task3), then it is scheduled just after data is being written to the queue, because it is the most priority task in ready state.

Maybe I misunderstood queues behaviour from the begining: at first, I thought a task waiting for a queue being written wass scheduled as soon as data was written to the queue, regardless its priority. But it looks like queue occupancy is only checked after task waiting for data in the queue is scheduled, or it is about t be scheduled.

Am I wrong?


richarddamon wrote on Wednesday, December 19, 2018:

The simple rule for what task gets to execute is that the highest priority ready task will be the one chosen to run.

There are basically only 3 factors that go into this.

  1. Task State, Only a task that is ready will be considered for being put into the Running State (and the Idle task will always be ready)
  2. Task Priority, Among the ready tasks, only the ones with the highest priority will be considered to be run
  3. Order in Priority, At every priority with Ready tasks, there is a queue with the Task that hasn’t run at the front, and the task that run most recently at the back, when selectiong the task to run the one that hasn’t run for the longest time is chosen.

From your description, Task2 being held up implies that Task3 (or maybe Task1) isn’t really blocking, so Task2 doesn’t become the highest priority ready task, or maybe is blocking for so short of a time that Task2 doesn’t get any real time to execute and make progress,