The first thing to note, since you specified that you are using a Mutex, is that mutexes have some special code to implement something called “Priority Inheritance” to get around the “Priority Inversion” case that you are describing.
When the High Priority task tries to take the Mutex, and the kernel sees that it is held by the Low Priority task, the High Priority Task needs to be blocked, but the Low Priority Task temporarily inherits the High Priority Task’s priority, so now the (Originally) Low Priority task has a higher “current” priority then the Medium Priority Task, so it gets to run, and when it gets to the point that it releases that Mutex, the High Priority Task will be woken, and the Low Priority task drops back to its original priority.
This is why it can be important to use a mutex, and not a semaphore, for this sort of exclusive access operation, or the High Priority task could get blocked by the unrelated Medium Priority task.
Got it now.
So, in this scenario, when the high‑priority task attempts to take the mutex and finds it held by the low‑priority task, the high‑priority task is blocked. At that point, the kernel applies priority inheritance to the low‑priority task, temporarily boosting its priority above that of the medium‑priority task. This allows the low‑priority task to run, release the mutex, and then return to its original priority. Once the mutex is released, the high‑priority task is unblocked and scheduled immediately.
Initially, only the low‑priority task is ready, so the scheduler runs it.
The low‑priority task acquires a semaphore and starts using a shared resource.
Later, the medium‑priority task becomes ready. Since it has a higher priority than low‑priority task, it preempts the low‑priority task. The low‑priority task is blocked but still holds the semaphore.
After that, the high‑priority task becomes ready and also needs the same shared resource protected by the semaphore. Because high‑priority task has the highest priority, it immediately preempts the medium‑priority task.
Now, the high‑priority task attempts to take the semaphore, but the semaphore is still held by the low‑priority task. Since the RTOS cannot give the resource, the high‑priority task blocks and moves to the Blocked state, even though it has the highest priority.
At this point, I am confused about the scheduler behavior.
Since the high‑priority task is blocked, which task will run next: the medium‑priority task or the low‑priority task and Why?
Your description is very precise, but at this point it needs to be noted that the “even though” is misleading. Exclusive access is independent of priority. If two or more tasks (or more general threads of execution) claim exclusive access via any syncronization mechanism, then the one holding the lock will block the other by definition. The difference between a mutex and a binary semaphore is that the OS tracks ownership of the mutex and can thereby counter priority inversion as @richard-damon explained.
The rule is actually fairly simple: The highest-priority task among those that are not blocked or suspended is scheduled. If more than one task of the same relative highest priority is ready to run and time slicing is enabled, the tasks will be scheduled in turn, preempted by the slicing policy. Thus, in your example, the medium pri task has the relative highest priority between itself and the low pri task and will thus be scheduled. The low pri task must wait until all higher priority tasks are blocked or suspended.
The low‑priority task runs first and takes a semaphore.
The medium‑priority task becomes ready and preempts the low‑priority task.
The high‑priority task becomes ready and immediately runs.
The high‑priority task tries to take the semaphore, but it is held by the low‑priority task
The high‑priority task blocks.
Once the high‑priority task blocks, it is removed from the Ready list, so its priority no longer matters.
The scheduler now considers only tasks that are Ready.
Between the medium‑priority task and the low‑priority task, the medium‑priority task has the higher priority.
Therefore, the medium‑priority task runs next.
The low‑priority task cannot run while the medium‑priority task remains Ready, even though the low‑priority task is holding the semaphore.
The semaphore stays locked, and the high‑priority task remains blocked.
This is priority inversion, caused because a semaphore does not provide priority inheritance. The scheduler simply follows the rule: run the highest‑priority Ready task
You almost have it. But the “now” implies that the scheduler so far has not followed that rule. That is not true. The scheduler ALWAYS follows that rule. So the sequence you outline assumes that your high-pri task has not been ready to run yet (was blocked, suspended or not created yet) during the first few steps of your time line. You also suggest that in your step description “the high-priority task becomes ready and immediately runs.”
Exactly. From an abstract point of view, it can even be argued that the rule still applies WITH mutex priority inheritance in place because the low-pri’s priority is temporarily adjusted while it holds the mutex, so the scheduler still obeys the rule, running the “temp high” low pri task because that task now (temporarily) has the highest priority among those ready to run.