High priority task totally blocks lower priority one, even with taskYield() in place?

apullin2 wrote on Thursday, July 09, 2015:

OK, I am stuck on something, and I have a feeling that it is probably something fundamental/basic that I need to get a handle on:

I have two tasks at different software priority levels (dsPIC, so only 1 hw priority level active for tasks). The higher priority task does some checks, sometimes kicks something into a queue, then calls taskYIELD(). The lower priority task blocks on waiting for stuff to come into that queue.

The ops the lower priority task does on the queue data is much slower than everything else, so I want that at a lowest priority. The problem is that the higher priority task totally blocks the lower priority one, not even executing the code in the task that comes before the for(;;){} loop in the task.

If I put both tasks at the same priority, then it works (as far as I can tell). But I thought taskYIELD() would voluntarily halt that higher priority task for the remainder of the time slice, so then a lower priority task could be serviced for the rest of the time slice? Like I said, I’m not hitting a breakpoint at all that is on the first line of the lower priority task.

The code itself is visible at:
https://github.com/apullin/octoroach_freertos/blob/work_save/lib/radio_freertos.c#L814 (the high priority task)
although this code is a cobbled krufty mess still, so you will not be able to glean much from just a look.

It occurs to me that I can rewrite the task as two separate tasks, and put a few semaphores between them so that everything ends up blocking waiting for some event signal from the hardware … it just seemed less obvious to divide it into two tasks that compete for the state of the peripheral, as part of the same driver.

As always, I am just a novice working on an academic project, and any input is welcome and appreciated.


davedoors wrote on Thursday, July 09, 2015:

FreeRTOS always chooses to run the highest priority task that can run. A task can still run after calling taskYIELD(), so calling that will never yield to a lower priority task. The lower priority task will only run if the higher priority task going into the Blocked state. You could call vTaskDelay(n) instead of taskYIELD(), but it would be better to make the higher priority task block on a semaphore or task notification until it has something to do, or just run it at a lower priority.

richard_damon wrote on Thursday, July 09, 2015:

First, not sure what you mean by “(dsPIC, so only 1 hw priority level active for tasks)”, as task priorities are a pure software concept, hardware priority levels relate to INTERRUPTS, not tasks.

Second, think what the high priority task is doing. It is continually checking queue for data in a non-blocking mode. As was pointed out, Yield means that you transfer processing to any other task of the SAME priority, but not to lower (and unless you have set the system non-preemptive) that other task won’t necessarily run to completion. This sort of behavior is only appropriate for a priority 0 (IDLE) level task.

The fact that you are expecting the ops task to start up right away when you queue data to it, sounds to me like you expect it to have a HIGHER priority.

I see two distinct operations in this task, First, why does the task need to move packets from one radioRXQueue to cmdQueue, why not have the producer just queue them up to the cmdQueue in the first place. If you are going to do something here in the future, then the task here should be get a message from the queue (blocking), due what you need and enqueue the message to the destination. There is NO need for peeking and testing for space, as you aren’t doing anything alternative if so, using blocking actions, not polling.

The second operation is checking a different queue, and when something is there getting the message, reconfiguring the radio, sending the message, and reconfiguring back. This likely should be a separate task (maybe lower priority than receive, so send only runs when receive doesn’t have packets to process).

The key is tasks need to BLOCK (be waiting on something) when there isn’t something to do, so the OS knows it can go to something else, and when it needs to come back to this task.

apullin2 wrote on Friday, July 10, 2015:

Thanks for the advice, guys.

@Dave, re interrupts, I just meant that, afaik, all the tasks run at one hardware interrupt level, so there shouldn’t be effects of some subset of hardware interrupts being masked out by the state of the chip.

@Richard, that all makes sense. I was just in error about how to use priorities with tasks. After thinking about it, and in agreement with what you write, there is a case for the two-tasks version, since I really want both to be able to block and wait for events coming from the hardware. And then the system can just low-power idle waiting for that to happen.

wrt to the rxQueue -> cmdQueue move, I certainly could shortcut around that, but from this layout, I was looking for 1) a place to provide dispatch from rxQueue to other places, although we only have a single consumer right now, I didn’t want to write an implicit dependence somewhere else in the code that actually handles the peripheral operation, and 2) buffering, the radio only has a single RAM buffer for both incoming and outgoing packets, so I want to get it out of there and into a buffer as cleanly as possible, then operate on the data lat lower priority.

But, yes, the implementation I had there is poor.
Here’s a newer way, where I split it into two tasks, both of which can block:

It pretty much works. It’s a very complicated interaction overall (pin irq -> spi -> DMA irq -> enqueue -> cleanup), so I need to comb through it.
At high speeds of RX and TX, I have some sort of a clash, and all the tasks except for the LED blinking “alive” indicator all stop.
The RTOSViewer plugin tells me that these tasks are in the “Suspended” state, not Blocked, which the documentation says shouldn’t happen unless done via vTaskSuspend.

rtel wrote on Friday, July 10, 2015:

The RTOSViewer plugin tells me that these tasks are in the “Suspended”
state, not Blocked, which the documentation says shouldn’t happen unless
done via vTaskSuspend.

I’m not sure which RTOS viewer you are using, but some will tell you
that a task is suspended when in fact it is just blocked without a
timeout. That is because of the internal implementation of FreeRTOS - a
few extra steps are required to distinguish between the two (which some
RTOS viewers do and others don’t).


apullin2 wrote on Friday, July 10, 2015:

This is the ‘RTOSViewer’ plugin in MPLABX; it looks like it comes from Microchip themselves, as far as I can tell.

Unfortunately, this chip & debugger doesn’t have proper instruction trace, like an ARM ETM. If I had that, this would be a 5 minute operation to see where the system is going off the rails.

Any recommendations on how to deal with a situation like this? Debugging an intermittent deadlocking condition with a simple breakpoint debugger?

rtel wrote on Friday, July 10, 2015:

FreeRTOS+Trace might help - there is also an MPLAB plug-in that allows
you to upload the trace buffer. I have a configuration that works on
the PIC32 which I could upload if it were useful.


apullin2 wrote on Tuesday, July 21, 2015:

I would like to see that example, if possible. Let me know where to grab it.

I am also now hitting the StackOverflow handler, which makes me think that I still have flawed design somewhere, and a cyclic dependency, so that my function calls aren’t returning, and the stack grows out of control.
If I trace the entry/exit of the right points, and all the blocking steps in my tasks, I might be able to capture this error.

dsPIC trace is so terrible :\ I can’t wait until we switch to ARM have an a full PC trace. That would make this pretty simple to root out.