strange scheduling issue

Kernel
1

e_bak wrote on Thursday, April 09, 2015:

Hi,

I have a strange scheduling issue with the cooperative scheduler on Cortex M3.

I have a logger task which writes onto the serial port.

I have a task which intends to test the logger task:

void taskPrintTest(void *params) {
	uint32_t cnt = 0;
	while (1) {
		printf("TaskName: %s, cnt: %d\n", pcTaskGetTaskName(NULL), cnt++);
		/* vTaskDelay(pdMS_TO_TICKS(250)); */
		/* TODO look for why only "pr2" is scheduled when the delay is removed !!! */
	}
}

In the main() function I create 3 instances of them:

xTaskCreate(
    taskPrintTest, "pr0", configMINIMAL_STACK_SIZE, NULL, 3, NULL);
xTaskCreate(
    taskPrintTest, "pr1", configMINIMAL_STACK_SIZE, NULL, 3, NULL);
xTaskCreate(
    taskPrintTest, "pr2", configMINIMAL_STACK_SIZE, NULL, 3, NULL);

The printf() method is used to send print event to the logger task via a queue:

static void lock() {
	/* take mutex */
	configASSERT(
		xSemaphoreTake(mutex, portMAX_DELAY));
	/* save task handle */
	taskToNotify = xTaskGetCurrentTaskHandle();
}

static void release() {
	/* get task notification */
	ulTaskNotifyTake(pdTRUE, portMAX_DELAY);
	/* release mutex */
	configASSERT(
		xSemaphoreGive(mutex));
}

void printf(const char * fmt, ...) {
	va_list va;
	QueueEvent event;
	lock();
	va_start(va, fmt);
	vsprintf(sprintfBuf, fmt, va);
	va_end(va);
	event.type = PrintEvent;
	event.msg = sprintfBuf;
	/* write to task event queue */
	configASSERT(
		xQueueSendToBack(queue, &event, portMAX_DELAY));
	release();
}

After the event is written into the queue, the task should block at ulTaskNotifyTake(pdTRUE, portMAX_DELAY), the xTaskNotifyGive(taskToNotify) is made by the logger task.

My experience is that only one of the taskPrintTest task is getting executed. On the serial console I see messages from task “pr2” only:

…
TaskName: pr2, cnt: 297710
TaskName: pr2, cnt: 297711
TaskName: pr2, cnt: 297712
TaskName: pr2, cnt: 297713
TaskName: pr2, cnt: 297714
TaskName: pr2, cnt: 297715
…

When I enable the delay in the taskPrintTest task “vTaskDelay(pdMS_TO_TICKS(250));”, the scheduling seems to be corrected:

…
TaskName: pr2, cnt: 0
TaskName: pr0, cnt: 0
TaskName: pr1, cnt: 0
TaskName: pr2, cnt: 1
TaskName: pr0, cnt: 1
TaskName: pr1, cnt: 1
TaskName: pr2, cnt: 2
TaskName: pr0, cnt: 2
TaskName: pr1, cnt: 2
…

I think even in the first case, when there is no delay in the print loop, the scheduler should execute “pr0” and “pr1” too in a Round-Robin manner. I think the taskPrintTest task is surely gets blocked at ulTaskNotifyTake(pdTRUE, portMAX_DELAY) in the release() method, at this point the scheduler should switch into an other instance of taskPrintTest.

Is it a normal behavior? What can cause this anomaly?

(Beside the taskPrintTest tasks I have the logger task and a led blinker task, these both seems to be scheduled and working all right.)

main.c: http://pastebin.com/vPBZmB7e
LoggerTask.c: http://pastebin.com/5pB39bT2

2

richard_damon wrote on Friday, April 10, 2015:

You said you were using “Cooperative Scheduling”. Your task (without the delay) isn’t being “cooperative” and giving up its control, so the other tasks will never get a chance to run.

“Round Robin” is normally used to describe time slicing with pre-emption, so doesn’t apply in “Cooperative” scheduling.

An alternative to using Delay, would be to add a taskYIELD() in the loop, to cooperate and let the other task have its turn. (This is the “Cooperative” part of the scheduling).

3

e_bak wrote on Friday, April 10, 2015:

“isn’t being “cooperative” and giving up its control”

It gives up control at ulTaskNotifyTake(pdTRUE, portMAX_DELAY) in the release() method. (there are the blinker task and logger task which scheduled right)

Currently I think the following happens:

“pr2” executes printf() which takes “mutex” and blocks at ulTaskNotifyTake(pdTRUE, portMAX_DELAY). When it is blocked “pr0” and later “pr1” also get executed but they block at “mutex” because “pr2” holds it.
Later “pr2” gets the notification, releases “mutex”, and loops forward. Since this point it is not yielded it retakes “mutex” from “pr0” and “pr1”.

4

rtel wrote on Friday, April 10, 2015:

If the co-operative scheduler is in use then I would still expect tasks
of equal priority to be executed in turn, but without time slicing, so
only when on task yields or blocks.

In your case you have all the tasks blocking though, and your explanation…

“pr2” executes printf() which takes “mutex” and blocks at
ulTaskNotifyTake(pdTRUE, portMAX_DELAY). When it is blocked “pr0”
and later “pr1” also get executed but they block at “mutex” because
“pr2” holds it. Later “pr2” gets the notification, releases “mutex”,
and loops forward. Since this point it is not yielded it retakes
“mutex” from “pr0” and “pr1”.

…would seem reasonable. You can test that by using a taskYIELD()
after the xSemaphoreTake() call.

Also, as there are two spaces in the queue, you won’t get any messages
printed out until there are two message in the queue, as that is the
time the tasks posting to the queue will block (the queue will be full).
It might be an idea to have the task that does the printing have a
higher priority, and then add a taskYIELD() after writing to the queue.
That will cause the print task to run immediately that there is
something to print.

I would question why you need the mutex at all since you are using the
co-operative scheduler.

I just tried the simple code below with the cooperative scheduler, and
the output was:

Task 2
Task 3
Task 1
Task 2
Task 3
Task 1
Task 2
Task 3
Task 1
Task 2
Task 3

code:

QueueHandle_t xQueue;

const char *pcString1 = "Task 1\r\n";
const char *pcString2 = "Task 2\r\n";
const char *pcString3 = "Task 3\r\n";

void vTask1( void *pvParameters )
{
   for( ;; )
   {
     xQueueSend( xQueue, &pcString1, portMAX_DELAY );
   }
}

void vTask2( void *pvParameters )
{
   for( ;; )
   {
     xQueueSend( xQueue, &pcString2, portMAX_DELAY );
   }
}

void vTask3( void *pvParameters )
{
   for( ;; )
   {
     xQueueSend( xQueue, &pcString3, portMAX_DELAY );
   }
}

void vPrintTask( void *pvParameters )
{
char *pcStringToPrint;

   for( ;; )
   {
     xQueueReceive( xQueue, &pcStringToPrint, portMAX_DELAY );
     printf( pcStringToPrint );
   }
}

int main( void )
{
   xQueue = xQueueCreate( 1, sizeof( char * ) );
   xTaskCreate( vTask1, "task1", 1000, NULL, 3, NULL );
   xTaskCreate( vTask2, "task2", 1000, NULL, 3, NULL );
   xTaskCreate( vTask3, "task3", 1000, NULL, 3, NULL );
   xTaskCreate( vPrintTask, "print", 1000, NULL, 4, NULL );
   vTaskStartScheduler();
}

regards.

5

e_bak wrote on Friday, April 10, 2015:

“I would question why you need the mutex at all since you are using the
co-operative scheduler.”

The printf() function sends a PrintEvent into the logger task, the event also contains a pointer to a string which needs to be sent/print. The logger task copies the string into the DMA buffer. If there is not enough available buffer space, the caller task of printf() have to be blocked because otherwise that could corrupt the message part which is not yet copied into the buffer. To achieve this printf() waits for a notification, the notification is sent by the logger task when it managed to place the whole string into the buffer.
Since printf() waits for a notification, which blocks, task switch will happen and that would allow other tasks to call printf() while the previous string is not yet processed. The logger task can’t handle this, so printf() is guarded by the mutex.

The queue can hold 2 entries, but only one PrintEvent entry can be in the queue at once. The other queue place is reserved for the DMA interrupt event. The mutex also ensures this condition.

“Also, as there are two spaces in the queue, you won’t get any messages
printed out until there are two message in the queue,”

printf() writes a PrintEvent into the queue, than waits for a notification from the logger task. So it will be blocked and the logger task will be scheduled.
There is only one PrintEvent in the queue at once.

6

e_bak wrote on Saturday, April 18, 2015:

Hi,

I’d like to illustrate this issue with a simple example.

There is a method which takes a mutex, sleeps, than releases the mutex, while writing trace messages onto the console:

static void guardedMethod() {
	con_printf("Task %s is taking the mutex.\n", pcTaskGetTaskName(NULL));
	configASSERT(
		xSemaphoreTake(mutex, portMAX_DELAY));
	con_printf("Task %s has taken the mutex.\n", pcTaskGetTaskName(NULL));
	vTaskDelay(pdMS_TO_TICKS(1000));
	configASSERT(
		xSemaphoreGive(mutex));
	con_printf("Task %s has released the mutex.\n", pcTaskGetTaskName(NULL));
}

There is a task which calls repeatedly the method above:

static void taskStrangeMutexDemo(void * params) {
	while (1) {
		guardedMethod();
	}
}

Two of these tasks are scheduled with the cooperative scheduler and with the same priority:

    xTaskCreate(
		taskStrangeMutexDemo, "Strange0", configMINIMAL_STACK_SIZE, NULL, BASE_TASK_PRIO, NULL);
	xTaskCreate(
		taskStrangeMutexDemo, "Strange1", configMINIMAL_STACK_SIZE, NULL, BASE_TASK_PRIO, NULL);

Here is the console log:

Task Strange0 is taking the mutex.
Task Strange0 has taken the mutex.
Task Strange1 is taking the mutex.
Task Strange0 has released the mutex.
Task Strange0 is taking the mutex.
Task Strange0 has taken the mutex.
Task Strange0 has released the mutex.
Task Strange0 is taking the mutex.
Task Strange0 has taken the mutex.
Task Strange0 has released the mutex.
Task Strange0 is taking the mutex.
Task Strange0 has taken the mutex.
Task Strange0 has released the mutex.

Here you see that the 1st scheduled task takes the mutex then the 2nd scheduled task also tries to take it but blocks because it has been already taken. It is all right.
But after this, the 1st scheduled task releases the mutex than re-takes it without being blocked and the 2nd task woken up. It is strange. Having a cooperative scheduler shouldn’t mean to not have proper inter-process communication.

I think a mutex should work in the following way:

  • when it is taken by task “A” and task “B” tries to take it, task “B” should be blocked and placed into the end of the wait queue of the mutex
  • when it is released by task “A” and the mutex’s wait queue is not empty, task “A” should be swapped-out with Ready state, the 1st task from the mutex wait queue should be removed (from the queue) and waken up (priorities should also be considered e.g. with priority queues)

I guess it would involve more RAM usage and processing. It would be even more CPU consuming to remove the timed-out tasks from the wait queue.

Easier solution would be to place portYIELD() after the xSemaphoreGive(mutex) call. But I think it is also not efficient when there is no task blocked by the mutex. Would it be possible to only execute portYIELD() when an other task is waiting for the mutex?

What is your opinion about it?

Here is the demonstration code: http://pastebin.com/3bZGZcWc

Regards

7

edwards3 wrote on Saturday, April 18, 2015:

I think a mutex should work in the following way:

  • when it is taken by task “A” and task “B” tries to take it, task “B” should be blocked and placed into the end of the wait queue of the mutex

Yes.

  • when it is released by task “A” and the mutex’s wait queue is not empty, task “A” should be swapped-out with Ready state, the 1st task from the mutex wait queue should be removed (from the queue) and waken up (priorities should also be considered e.g. with priority queues)

Not always, especially if you are using the cooperative scheduler. I think cooperative scheduling used to work like that in old versions of FreeRTOS but then the behavior was corrected to be truly cooperative, so only a manual yield or a block switches to the other task. Search http://www.freertos.org/History.txt for “Changes between V7.5.3 and V7.6.0 released 18th November 2013”

8

e_bak wrote on Saturday, April 18, 2015:

Ok. Thanks for the info!

I have tried a similar test code with pthread on Linux. And as I see it behaves like FreeRTOS. The mutex is “re-taken” and task switch seems to be only occur at preemption.

It seems what I believed strange is just the usual way :slight_smile: