Issue when running greater than 20 tasks concurrently in freeRTOS

Hello ,
We planning to use freeRTOS for my up coming project. And i would like to create about 50 tasks for this project and i tried creating 25 tasks and i could successfully but if while executing these 25 tasks few of my tasks does not work as expected.

I am using NXP-MPC5777c MCU and following configuration from freeRTOS.

#define configUSE_PREEMPTION 1
#define configUSE_TIME_SLICING 1

#define configTICK_RATE_HZ ( ( TickType_t )1000 )

#define configSUPPORT_STATIC_ALLOCATION 1
#define configMINIMAL_STACK_SIZE ( ( uint16_t )512 )

And i am using static task creating using xTaskCreateStatic(), So i am wondering if are there any configuration settings which are limiting the no of tasks that can run on freeRTOS?.

Could you please provide your guidance!

Thank you so much in advance!!

One big question is what are the tasks doing, and do you have enough CPU to process all of them?

Also, try supplying configASSERT and stack overflow checking to help catch some common problems.

Hello Richard,

Thank you so much for your response.

One big question is what are the tasks doing, ?
– In real application they are more like periodic tasks like reading sensors data, reading certain switch inputs and periodic CAN transmission and reception tasks.
– But in my test code, i have just few LED blink tasks, counter decrement tasks and CAN periodic transmission to check if they work concurrently.

and do you have enough CPU to process all of them

– I am bit new to the FreeRTOS, But my understating is that freeRTOS does not limit number of tasks that can be run so i just create the tasks and see if they are able to run. i am using MPC 5777c which is powerful MCU so i assume that CPU is good enough to handle it but i really have no idea can i check if my CPU is enough? Could you please let me know?

Also, try supplying configASSERT and stack overflow checking to help catch some common problems.

– I use the asserts there is no problem, seems like tasks are running but output is not as expected(I could see that LED blinks not as expected)

Please let me know if you need more info.

Thank you!!

It doesn’t matter how fast your processor is if some task just does a busy-wait loop and burns up all the CPU time.

You can implement the Run Time Stats to see if that is the issue. See https://freertos.org/RTOS-run-time-stats.html for instructions. This is a likely problem if simple low priority tasks are not running.

Another issue could be a dead-lock, where Task A is waiting for something from Task B, but Task B is waiting for something from Task A.

I’ve had projects with 25 tasks and no problems, but you do need to be careful in your design

Okay. Thank you for your inputs i will implement run time stats and check whats happening.

Hello Richard,

I have enabled run time stats and trying to use vTaskGetRunTimeStats API to get the stats, but kernel expects configSUPPORT_DYNAMIC_ALLOCATION to be set to 1, But i am using static allocation for for my tasks.

So question here is run time stats only supported for dynamic allocation? or can i use it for static allocation as well? If yes, How can i do that.

Could you please clarify.

Thank you…

You can enable BOTH static and dynamic allocation, and allocate all of your tasks statically, but also allow FreeRTOS to uses the dynamic allocation routines for the stats.

Note, you can use the function uxTaskGetSystemState to get the raw data for the stats without needing to enable dynamic allocation, since you know how any elements to allocate for the status array (remember to add elements for the Idle task and the Timer/Service task).

You will need to provide more information the symptom you are seeing, and what you have done to debug, to get more targeting suggestions. How does the issue manifest itself beyond not working “as expected”. For example, do all the tasks start running, or is a symptom that one task just never executes. Is there one particular task that causes the issue, and omitting that task resolves the issue? Is it the issue directly related to the number of tasks created, or simply caused by whichever was the last task to execute?

I mean that, i have created 20 tasks with different priorities and all the 20 tasks are running without issues. I have 2 LED tasks in 20 which also blinks periodically for every 200 ms with no issues.

Note: LED tasks are the lowest prio tasks with same priority (tskIDLE_PRIORITY + 1)

But when i try to add another 3 periodic tasks with different priorities, then my lowest prio LED tasks execution is not periodic for every 200 ms. with the blink pattern i could see that LED does not blink for every 200 ms they blink randomly with unknown timing. But all the 23 tasks are running.

I am suspecting the dummy tasks which i have created which does nothing but wait for 1 tick. And i have 10 of these tasks which does the same thing.

static void vTaskDummy(void *pvParameter)
{
    for ( ; ; )
    {
        unsigned char counter = 10;

        while (counter-- ! =  0)
        {
        }

        vTaskDelay(1);
    }
}

Please let me know your views on this and do let me know if you need any additional information.

Thank you for your quick support always!!

Your behavior sounds like the result of hitting 100% CPU usage. Another possibility is that some task is corrupting the memory of the system.

Try vTaskDelay(2) instead of 1 and see if that makes a difference. Waiting for 1 tick means ar most 1, it may preempt the task sooner.

Hello Richard,

I have increased the vTaskDelay(); from 1 ms to 50 ms and all the 30 tasks run now without issues.

And i tried to print the run time stats now. I see some strange values here. not clear what is the problem. Here are run time stats for my project.

  1. Task20Can .735…735%
  2. Task19Can .179…179%
  3. TaskCanMaster .225013…225013%
  4. IDLE0 .2028023…2028023%
  5. Task17Can .610…610%
  6. Task16Can .68…68%
  7. Task15Can .127…127%
  8. Task14Can .0…<1%
  9. Task13Can .390…390%
  10. Task12Can .194…194%
  11. Task2Can .876…876%
  12. Task11Can .281…281%
  13. Task10Can .1236…1236%
  14. Task6Can .45…45%
  15. Task5Can .135…135%
  16. Task3Can .536…536%
  17. Task1Can .1918…1918%
  18. TaskLed .4959…4959%
  19. TaskLed2 .4610…4610%
  20. TaskDummy4 .20236…20236%
  21. TaskDummy5 .20094…20094%
  22. TaskDummy6 .14008…14008%
  23. TaskDummy7 .19699…19699%
  24. TaskDummy8 .10601…10601%
  25. TaskDummy9 .16242…16242%
  26. TaskDummy10 .11660…11660%
  27. TaskDummy1 .40888…40888%
  28. TaskDummy2 .8247…8247%
  29. TaskDummy3 .15687…15687%
  30. Task8Can .236…236%
  31. Task4Can .956…956%
  32. Task7Can .333…333%
  33. Task9Can .970…970%
  34. Task18Can .278…278%
  35. Tmr Svc .0…<1%

Here is my configuration for run time stats in FreeRTOSCconfig file.

#define configSUPPORT_DYNAMIC_ALLOCATION 1
#define configTOTAL_HEAP_SIZE ( ( size_t )10000 )

/* Run time and task stats gathering related definitions. */
#define configGENERATE_RUN_TIME_STATS 1
#define configUSE_TRACE_FACILITY 1
#define configUSE_STATS_FORMATTING_FUNCTIONS 1

#define portCONFIGURE_TIMER_FOR_RUN_TIME_STATS() pitTimerChannel0Setup(0, 100000)
#define portGET_RUN_TIME_COUNTER_VALUE() timer_pit_rti_get_current_Timer_value()

Note:

I am calling vTaskGetRunTimeStats() API for every 10 seconds.

and i get different stats everytime not sure what is the problem. Below values are printed by using same API at nth time.

  1. Task20Can .207939…69313%
  2. TaskCanMaster .19345329…6448443%
  3. IDLE0 .73768069…24589356%
  4. Task18Can .7006…2335%
  5. Task17Can .10363…3454%
  6. Task16Can .8058…2686%
  7. Task15Can .10308…3436%
  8. Task14Can .9479…3159%
  9. Task13Can .9040…3013%
  10. Task12Can .10765…3588%
  11. Task2Can .45336…15112%
  12. Task11Can .8426…2808%
  13. Task10Can .54341…18113%
  14. Task8Can .12989…4329%
  15. Task6Can .5582…1860%
  16. Task5Can .11711…3903%
  17. Task4Can .24356…8118%
  18. Task3Can .24784…8261%
  19. Task1Can .85812…28604%
  20. TaskLed .205587…68529%
  21. TaskLed2 .211252…70417%
  22. TaskDummy1 .3335720…1111906%
  23. TaskDummy2 .1168254…389418%
  24. TaskDummy3 .734785…244928%
  25. TaskDummy4 .1911727…637242%
  26. TaskDummy5 .1797155…599051%
  27. TaskDummy6 .988849…329616%
  28. TaskDummy7 .884285…294761%
  29. TaskDummy8 .1031829…343943%
  30. TaskDummy9 .1037404…345801%
  31. TaskDummy10 .791321…263773%
  32. Task7Can .15412…5137%

My first guess at the strange values is that you Run Time States counter is overflowing.

Newer version of FreeRTOS allow it to use a 64 bit counter that helps. Otherwise you cycle counter needs a low enough count rate that the default 32 bit total time counter doesn’t overflow during your sample period.

I recommend setting a pre-scaler on your timer counter to slow down it’s rate. FreeRTOS is only going to context switch every ms (unless their is pre-emption I think?), so I always set the prescaler on my timer so that things work out that each timer tick is a us, which should give me a reasonable amount of accuracy while still making the numbers coherent to read. To do this, set the pre-scaler on your counter to the same rate that it’s being clocked at (so if the timer is running at 150MHz, put a pre-scaler of 150).

To make it easier to read, you could also do some print formatting. This is what my debug printout looks like.

     --Task Name  State   Pri  Stack   Num            Time
        ConUsbRx      -     4    850     8             193
            IDLE      R     0    231     2       5,775,898
           Smbus      B     3    129     5          11,394
         Tmr Svc      B     2    431     3           6,574
        BootTask      B     1     44     1       2,688,221
          PwrMgr      B     5    865     6         109,884
             LTI      B     6    411     4         168,607
        ConUsbTx      S     4    966     7             122

From this, I can see that my idle task has take approximately 5.7s of time, my boot task has taken 2.6s and everything else has been less than 1 second.

1 Like

One point of correction, FreeRTOS can context switch much more often than the tick rate, as every time the current task blocks, it will context switch.

It will only context switch elsewhere only if in ISR wakes up a higher priority task or on a tick interrupt.

I am having system clock running at 96 MHZ and i configured my timer for 1 MHZ (1 us) but now nothing seems to work i get zeros in the stats buffer.

Is decreasing counter fine for the run time stats? or counter has to be incrementing one?

The value returned by the function needs to be an increasing counter, as it uses difference between readings to determine the elapsed time.

Hello Richard,

Thank you for your input again. I have implemented the incrementing counter and now values looks fine.

TaskeSCI .22184284…3%
TaskCanMaster .391661454…64%
IDLE0 .157280358…25%
Task12Can .14573…<1%
Task11Can .14421…<1%
Task2Can .94003…<1%
Task10Can .226800…<1%
Task5Can .32900…<1%
Task1Can .206471…<1%
TaskLed .457973…<1%
TaskLed2 .453813…<1%
TaskDummy1 .3026018…<1%
TaskDummy2 .3019752…<1%
TaskDummy3 .3401400…<1%
TaskDummy4 .3119219…<1%
TaskDummy5 .3109280…<1%
TaskDummy6 .4578899…<1%
TaskDummy7 .3133832…<1%
TaskDummy8 .4356286…<1%
TaskDummy9 .3122375…<1%
Task6Can .40860…<1%
Task4Can .59818…<1%
Task3Can .70775…<1%
Task7Can .32968…<1%
Task8Can .41225…<1%
Task9Can .33135…<1%
Task20Can .41342…<1%
Task19Can .38395…<1%
Task18Can .38575…<1%
Task17Can .33428…<1%
Task16Can .33490…<1%
Task15Can .38600…<1%
Task14Can .46659…<1%

Just would like to understand this Absolute time bit more, So this Abs time is time in micro seconds as my timer period is 1 micro second. Is my understanding correct?

The numbers are the number of Real Time Counter Ticks the task saw, so yes, approximately the number of microseconds it has run (subject to sub-microsecond round off errors every interval it runs).

The very high value for the TaskCanMaster task makes me wonder if it has a polling loop in it, and might be a cause for concern about other tasks getting starved.

Hello Richard,

To understand this topic little bit more on how to interpret these values, Could you please clarify my following points.

  1. How do i know what is the total available time of my CPU? (Is IDLE task time + Time taken
    by all the other tasks gives total CPU time?)
  2. Is IDLE0 task time is the remaining available time of the CPU? If Yes, i need to ensure that this time should not reach zero and always some IDLE0 time?
  3. The the sum of all the tasks percentages value shown in the stats should always be equal to 100 %?

I fixed 1 tick delay in TaskCanmaster which shows high CPU time earlier. Following are my new stats.