RTOS Improvements

system · June 4, 2015, 12:24am

losttime77 wrote on Thursday, June 04, 2015:

Hello, I am posting here to potentially spur some conversation and or comments about the functionality present in FreeRTOS.

Some background, I work for a certain professional company. We will call it X. Recently my project lead decided that the best way to get into a preemptive scheduler would be to use FreeRTOS. There was a feature set that I set out with to see if FreeRTOS would be sufficient for our uses. As of now, I have patched the FreeRTOS source for our purposes, and we are currently trying to figure out the licensing so that we can adhere to the license (what source we have to share, versus buying a license and not sharing the source).

We are transitioning from a cooperative scheduler that has no stack context that was designed to run on PIC10’s (perhaps you can guess which RTOS I am referring to) to a preemptive scheduler.

Preemptive to us is not so much the ability to preempt tasks of higher priority. The main feature that we strive for was stack context. In other words, each task having its own stack and therefore could be preempted easily. The scheduler we are transitioning from required any local variables to be static (thus globals) because context was lost every task context switch.

Here are some notes that ‘would have’ made FreeRTOS perfect for us as a company, but required me to patch the source, which in itself is not desirable, because when new versions of FreeRTOS come out, I would have liked to just do a nice ‘copy & paste’.

We require static allocation. The way FreeRTOS is set up, there are no …init functions. There is a generic create for each primitive that one wants to use which handles both memory allocation and initializing the object. In order to both facilitate static and dynamic allocation schemes, you generally implement a create function which allocates memory and then an init function which just initializes the object from a pointer. The init function does not care where the memory comes from.
The “tickless” implementation in FreeRTOS is not truly tickless. It is in fact along the lines of “when i’m in the idle task, I will disable & enable the tick dynamically”. What about the case where there is literally no tick - an on demand system for example? So in other words, the functions that take timeouts would always take either a 0 or a portMAX_DELAY. In this case, all of the timeout code is completely unneeded and just using up space. Our system is a truly tickless system. We once had a tick based system; however, in light of getting even lower power, we moved to an on demand timer engine. Basically our “software timers” are implemented such that we have a list of timers created by apps. These timers are sorted by timeout. Using a hardware timer, we load it with the next timeout for the first timer on the list. Once we get the interrupt for that timer, we remove the timer from the list and get the timeout for the next timer on the list. (This is an overly simplified example, but essentially it means we implement an ‘on demand’ software timer mechanism whereas most systems implement a tick based software timer mechanism). This allows us for example to set a timer timeout for an entire minute and go to sleep for that entire minute without waking for a tick. What I am getting at is since our system is used this way, it would be very nice if there was a way to disable all timer type tick processing in FreeRTOS. For us, that code is dead code and just using precious space. To get around this for now, I am only using timeout delays of 0 and portMAX_DELAY so that either my call to the function returns immediately or my task simply suspends until we get an event. Every kilobyte matters to us as we want this system to potentially. run on extremely ROM limited chips (32, 64, 128K).
I realize that I think the idle task implements termination list functionality is because vTaskDelete also de allocates dynamic memory. The de allocation function could be non deterministic. Therefore in the idle task you do the de allocation instead of directly in the call to vTaskDelete. If this is the case, this works in systems where in fact the pvPortFree or whatever is non deterministic. Our system implements its own heap based on memory pools. Our allocation function (malloc) is O(log2(n)) where our de allocation function (free) is O(1). Since our “free” is O(1), it makes no sense to not de allocate in vTaskDelete since we know exactly how much time is being consumed. I have commented out all of the code that adds a deleted task to a termination list and the code now simply removes the active task from the ready list.
I have seen many other RTOSs we have looked at do this and it annoys the heck out of me. Will somebody please inform me why when you can return from a task, you do not just return to vTaskDelete? or implement a simple wrapper that calls vTaskDelete(NULL) to destroy the current task? I honestly do not know the reasoning for this. In the port.c file (for an ARM Cortex M), you set up the stack when a task is initialized so that it returns to a function. In FreeRTOS this function is taskError or something. Why not just return to the vTaskDelete(void) wrapper that then calls vTaskDelete(NULL)? Logically when I return from a function (which is also a task), I just want it to terminate. I don’t want it to blow up the system, which it currently does. Likewise, in all my tasks that do return I have to make a call to vTaskDelete. Why when it is a single line code change in port.c to make it return to the wrapper? Do you see a win32 console app or any console app on any system blowing up the system after IT returns?
Requiring the first include whenever I want to use any RTOS primitive (tasks, queues, whatever) to be FreeRTOS.h is not good design. In fact, it’s poor naming. If I include FreeRTOS.h, I expect that to include the headers for all the primitives in FreeRTOS. I should not have to include individual files for everything I want to use. Requiring an ordered include as a hard requirement (FreeRTOS.h has to be first) is bad by design.
This goes back to static allocation. I personally think (and many others have agreed with me on this internally) that defining structures and macros inside c files is insane. I understand that it seems like it was done so that you can “hide” the internal implementation of those structures to where they are used. However it has 2 bad side effects The first is that if I want implement static allocation mechanisms (which I have), I have to do it inside the file requiring that implementation. This means that now I have a modification to the FreeRTOS distributed source code. Even though my functions are sectioned off to the bottom of the source and I know exactly where they are, it requires many extra copy / paste steps when we go to a new version. Modifying the base source code of the distribution immediately made me think that I now have to share those modifications because of the license. The second issue is that the actual static allocation implementation becomes very hard. If these structure definitions were in a header file, then I could just include the header file and do a sizeof command to statically allocate the structure somewhere. As it is, the way we have done it, we had to actually execute code to do sizeof commands inside the FreeRTOS source to figure out the sizes of those structures. Once we have the sizes of those structures, we can just allocate buffers of that size representing the structures. My static allocation functions simply take uint8_t* buffers and cast them to the appropriate FreeRTOS structure pointer or handle.
Subsequently, I could have implemented all of my auxiliary functions in seperate source code files outside of the FreeRTOS distributed files, thus decoupling them from the distribution.
Now I realize that I think this was done for data hiding ability, but in doing so it has some bad consequences. Additionally, FreeRTOS is open source… I have access to the source code already and can dig and poke around to figure out what I need. What are you hiding? Why make it harder by doing data hiding this way? In an environment where you do not want to distribute headers that expose such structures, such as a library distribution, there are better ways to structure the code. For libraries, you distribute public API files. Then for your own state of mind, you define the structures / macros in private header files that do not get distributed. If FreeRTOS was distributed in such a way, then for the sake of your development team at least… define those structures in (private) header files. This immediately made me think of a lot of code from other companies where you see #include somefile.c in another file. Ask yourself, do you ever want to #include c files in code - multiple definition hell.
For our code base, we do not use FreeRTOS calls directly, they are wrapped under another layer. This allows us to make generic wrappers that need to be implemented by any scheduler we use. Today we use FreeRTOS, but tomorrow we could use TNKernel (or whatever). All that is needed to hook into the rest of our system is for the RTOS distribution layer to implement our wrapper functions. On that note, there is no way in heck that I am putting RTOS specific FromISR functions in any portion of our code base just because well… FreeRTOS decided to do something different than any other RTOS we have looked at. FreeRTOS is the only kernel that I have seen that requires the usage of FromISR functions. Feel free to point out other kernels that do this. After looking at how they are actually used in the FreeRTOS source, I realize ‘why’ they are there, but I also don’t understand why we cannot find FromISR functions in other kernels - they are not required. On that point, especially since we intend to use FreeRTOS in a purely event driven model (no software timers or timeouts from FreeRTOS (true tickless)), I am not sure if we really require the FromISR functions anyways.
FreeRTOS’s message queue implementation. Oh… the message queues. Where do I begin. Let me give you the thousand foot view of where I am coming from with these. I have a program in which one task sends messages to a message queue and another one reads from that same queue. Now tell me, if I ever run into a situation where I am sending more messages to the second task than my second task can process, what logically is your first train of thought? Mine is this: “Wow, so I am putting more messages in my queue than can be processed by this secondary task! I definitely have a bigger issue, and this needs to get fixed.” Let me try and paraphrase FreeRTOS’s logical train of thought: “Hmm, well, let’s hide that particular case from the user and implement 2 options to let them shoot themselves in the foot. Option number 1: Let’s make it so we can arbitrarily overwrite the last message in the queue! Option number 2: Let’s implement a timeout mechanism that allows the issue to be completely hidden and never seen again. If I just send with a timeout of 2 seconds, they should never see this issue. That will be enough time to catch up.” Well, the point is, with a properly designed system, you should never be pushing more messages to a queue than can be processed in a timely manner. FreeRTOS allows the user to shoot themselves in the foot by completely hiding this issue under the hood. My mailbox is full man! That means you can’t send me any more letters. Jeez, I wish that was the case when some of the letters that have to be sent were bills. Wouldn’t it be awesome if an important letter came in that was a bill couldn’t fit in my mailbox so what actually happened was that it “got lost”? I’m sure the bill collector would understand. But let me tell you, next time that mailbox needs its scheduler, I will tell them to use FreeRTOS.

I realize that I am being harsh here, but I am looking at FreeRTOS as a professional developer. I think there is major room for improvement in the areas that I am talking about. You might have thought that FreeRTOS covered 99% of the use cases as it was; however, we just hit the rest of the 1% of use cases it was not intended for. I apologize for not stating which company I work for; however, I do wish to try and remain ‘somewhat’ anonymous right now. I think that another “tick” no pun intended added to the companies using FreeRTOS from my company would be awesome, which is why I am striving to express these areas for improvement.

Also, if any of the things I have stated above have their reasons of why they were done rooted in the MISRA or any other sort of compliance requirement, Feel free to state that. I will publicly admit that I personally do not have much (if any) experience in compliance requirements.

rtel · June 4, 2015, 9:37am

rtel wrote on Thursday, June 04, 2015:

I will do my best to answer each point. Remember however that each design decision made, then will be some users satisfied, and some users dissatisfied. FreeRTOS has been around for a long time, and during that time we have learned what the majority of users want, and made decisions accordingly.

Preemptive to us is not so much the ability to preempt tasks of
higher priority. The main feature that we strive for was stack
context. In other words, each task having its own stack and therefore
could be preempted easily. The scheduler we are transitioning from
required any local variables to be static (thus globals) because
context was lost every task context switch.

Hence, your rationale for using an RTOS differs from the majority of users…

Here are some notes that ‘would have’ made FreeRTOS perfect for us as
a company, but required me to patch the kernel, which in itself is
not desirable, because when new versions of FreeRTOS come out, I
would have liked to just do a nice ‘copy & paste’.

Agreed, patching is to be discouraged, in an ideal world, but is sometimes inevitable. Having the source code of course gives you that freedom. Many times the trace macros can be used to insert code into the kernel without needing to edit the source files, but the trace macros are not going to give you the ability to change anything fundamental about the kernel’s behaviour.

We require static allocation. The way FreeRTOS is set up, there is
are no …init functions. There is a generic create for each
primitive that one wants to use which handles both memory allocation
AND initializing the object. In order to both facilitate static and
dynamic allocation schemes, you generally implement a create function
which allocates memory and then an init function which just
initializes the object from a pointer. The init function does not
care where the memory comes from.

One of the initial goals of FreeRTOS was to make it as easy to use as possible (its been around a very long time, and of course evolved over that time). A deliberate decision, as part of fulfilling that goal, was to avoid the necessity for the user to have to statically allocate anything, but have the kernel do it for them. That avoids the need for design/compile time pre-allocation, removing the need apportion resources up front, makes the API a lot simpler, allows available RAM to be used as efficiently as possible (one object can be deleted to make way for a new object of a different kind), etc.

Feedback on this topic shows that this easy of use, and lack of static allocation, is one of the reasons people choose FreeRTOS over other systems. It does not, of course, suite every application (as you note), but it would seem to suite the majority of applications by quite some margin.

I’m not sure if it helps, or if it is relevant to your point, but with some of the heap allocation schemes you can place the heap at any address you like, rather than allow the linker to do it for you. Also, schemes like heap_1, which do not allow RAM to be freed and therefore cannot fragment and are deterministic, is quite close to being statically allocated. heap_1, despite first appearances, is suitable for a lot of what I would call ‘dedicated’ systems that are turned on, allocate all their objects, start the scheduler, then run until they are turned off again.

The “tickless” implementation in FreeRTOS is not truly tickless. It
is in fact along the lines of “when i’m in the idle task, I will
disable & enable the tick dynamically”.

Yes, which is why it is called tickless idling. Again this was a deliberate decision. The clock only needs to be re-programmed when it is known that the tick will be off for some time into the future. Making the calculation and reprogramming a clock for a time some time into the future each and every time the scheduler was called was considered too inefficient and too much overhead.

What about the case where
there is literally no tick - an on demand system for example? So in
other words, the functions that take timeouts would always take
either a 0 or a portMAX_DELAY.

This is far from a ‘normal’ use case.

In this case, all of the timeout code
is completely unneeded and just using up space. Our system is a truly
tickless system. We once had a tick based system; however, in light
of getting even lower power, we moved to an on demand timer engine.
Basically our “software timers” are implemented such that we have a
list of timers created by apps. These timers are sorted by timeout.

Which is exactly how the software timers built into FreeRTOS work.

Using a hardware timer, we load it with the next timeout for the
first timer on the list.

Which is how the tickless idling works.

Once we get the interrupt for that timer, we
remove the timer from the list and get the timeout for the next timer
on the list.

Which is how software timers/tickless idling work.

(This is an overly simplified example, but essentially
it means we implement an ‘on demand’ software timer mechanism whereas
most systems implement a tick based software timer mechanism).

So do you want an RTOS at all? That is not a question related specifically to “do you want FreeRTOS”, but do you want any RTOS? As generally that is not how they function.

This
allows us for example to set a timer timeout for an entire minute and
go to sleep for that entire minute without waking for a tick.

What I am getting at is since our system is used this way, it would
be very nice if there was a way to disable ALL timer type tick
processing in FreeRTOS.

Note comments above about, do you really want an RTOS?

For us, that code is dead code and just using
us precious space. To get around this for now, I am only using
timeout delays of 0 and portMAX_DELAY so that either my call to the
function returns immediately or my task simply suspends until we get
an event.

Again, not sure if it is relevant, but when using tickless idling the handling functions can tell you if all the tasks are blocked indefinitely, that is, without a timeout, allowing you to sleep indefinitely until an external interrupt occurs. That functionality is improved in the SVN head revision to work better with software timers too.

I have seen many other RTOSs we have looked at do this and it annoys
the heck out of me. Will somebody PLEASE inform me WHY when you can
return from a task, you do not just return to vTaskDelete?

This has been discussed before, you will find some notes in the support archive. In summary, having a task automatically delete itself if it drops off the end of a function takes more stack (important on smaller systems, not so on Cortex-M), hides mistakes made by inexperienced users who don’t realise that has happened, is non portable, and confuses GDB. Considering these problems can be avoided by simply ending a task’s implementation will a call to vTaskDelete( NULL ) for the rare case when a task wanted to drop off its function, it was considered to leave the implementation as it was.

Requiring the first include whenever I want to use any RTOS primitive
(tasks, queues, whatever) to be FreeRTOS.h is bad design. In fact,
its bad naming. If I include FreeRTOS.h, I expect that to include the
headers for all the primitives in FreeRTOS. I should not have to
include individual files for everything I want to use. Requiring an
ordered include as a hard requirement (FreeRTOS.h has to be first) is
bad by design.

…in your opinion, but not in the opinion of lots of formal coding standards, including safety standards, which consider nested includes detrimental to readability and code maintenance. Sorry again if this does not suite you personally, but it was quite definitely a deliberate decision, although one that has been relaxed somewhat in some of the FreeRTOS+ components.

There is of course nothing to prevent you defining your own header file that just includes all the other header files.

This goes back to static allocation. I personally think (and many
others have agreed with me on this internally) that defining
structures and macros inside c files is insane. I understand that it
seems like it was done so that you can “hide” the internal
implementation of those structures to where they are used.

Data hiding: It is considered by most to be best engineering practice. It prevents backward compatibility issues, and greatly simplifies maintainability. It also gives us the freedom to change internal implementations provided the public interface is maintained/unmodified.

Yes - again we disagree here, the decision is quite deliberate and follows what an academic would consider best practice, even if you consider it ‘insane’ ;o)

By modifying the base source code of the

distribution immediately made me think that I now have to share those
modifications because of the license.

Umm, yes. You want to take, but not give? All you have to do is upload the changes to the FreeRTOS interactive site, which can be done anonymously.

but if you guys did this
just so that people would have to modify the source inside the
distributed source so you could catch people with the licensing
clause… well…

Sorry - I really don’t appreciate that comment. FreeRTOS is a professionally developed product, evolved over some 14 years of hard labour, professionally supported for free too. You can consider it has taken $m’s of dollars to develop and support, and it is given to you for free with the proviso that if you make a small improvement you must also be willing to share it with the rest of the community. Is that much to ask, considering you can upload your changes anonymously?

system · June 4, 2015, 12:25pm

losttime77 wrote on Thursday, June 04, 2015:

Let me start by apologizing for that last comment you quoted. You apparently got and responded to the unmodified post that I made. I had removed that statement, because I realized that it was not appropriate.

So do you want an RTOS at all? That is not a question related specifically to “do you want FreeRTOS”, but do you want any RTOS? As generally that is not how they function.

You misunderstand. Just because we have a different usage model that does not include a tick, does not mean we do ‘not’ want an RTOS. Just because it’s how its ‘usually’ done, does not mean since we are doing things off the menu that we automatically fall out of the usage model of an RTOS. Is it really that hard to imagine an event driven RTOS that has no tick? Preemption is more than just being able to context switch a task because it decided to do a synchronous file write that takes 10 seconds. In an ideal system, everything would be event driven and asynchronous. This is the system we strive for and it is how we have modified the FreeRTOS source. So yes, our model can be done. I am sorry if this falls out of the area of conventional RTOS knowledge. If I had a penny for every time we have heard “Well, thats how the real world does it!” I would be rich. This closed minded mindset on FreeRTOS’s part shows a refusal to accommodate / learn about different usage models.

On that note, I am not asking to change your software timer implementation. I am saying that in our model, the code that processes ticks is dead code. Because we provide our own timing system, it would be beneficial to be able to disable ticking code entirely. You have macros for just about everything in the system to disable various parts of the scheduler due to feature set, yet nothing for fully disabling timer code.

Yes - again we disagree here, the decision is quite deliberate and follows what an academic would consider best practice, even if you consider it ‘insane’ ;o)

I am sorry, but you are incorrect. I would be willing to put this one up for a vote. You will not win. I stand to be corrected. It is standard practice to NOT do that kind of trickery inside c files for exactly the reason it’s not done - multiple definitions. You do not include .c files into other source. Of course, that’s not exactly whats happening, but the mindset applied to the code facilities doing something like this.

I am also glad you mentioned academic. I don’t know where you learned to code, but any academic class taught in the past X years would not have students do that inside c files.

…in your opinion, but not in the opinion of lots of formal coding standards, including safety standards, which consider nested includes detrimental to readability and code maintenance. Sorry again if this does not suite you personally, but it was quite definitely a deliberate decision, although one that has been relaxed somewhat in some of the FreeRTOS+ components.

I would like to see these formal coding standards, because now I am curious. Like I stated in the end, I have no experience with them.

One of the initial goals of FreeRTOS was to make it as easy to use as possible (its been around a very long time, and of course evolved over that time). A deliberate decision, as part of fulfilling that goal, was to avoid the necessity for the user to have to statically allocate anything, but have the kernel do it for them. That avoids the need for design/compile time pre-allocation, removing the need apportion resources up front, makes the API a lot simpler, allows available RAM to be used as efficiently as possible (one object can be deleted to make way for a new object of a different kind), etc.

Feedback on this topic shows that this easy of use, and lack of static allocation, is one of the reasons people choose FreeRTOS over other systems. It does not, of course, suite every application (as you note), but it would seem to suite the majority of applications by quite some margin.

This is exactly why all other RTOSs that we have looked at (and there are alot) provide at the base a static allocation mechanism and then a dynamic allocation mechanism on top of that, right? Just because the majority of your user base is ‘satisfied’ with how it works does not mean you should not strive to hit the other 10% of people who want a different feature. This mindset turns me off. Especially since to do this requires minimal change and is standard practice of how objects are initialized in the ‘real’ world. Likewise to provide static allocation would be completely backwards compatible, because you simply have to provide init functions as well as create functions. The create functions use the init functions after they dynamically allocate the memory. The init functions do not care where the memory comes from - static or dynamic. The user is not required to use static allocation, they have either option.

This has been discussed before, you will find some notes in the support archive. In summary, having a task automatically delete itself if it drops off the end of a function takes more stack (important on smaller systems, not so on Cortex-M), hides mistakes made by inexperienced users who don’t realise that has happened, is non portable, and confuses GDB. Considering these problems can be avoided by simply ending a task’s implementation will a call to vTaskDelete( NULL ) for the rare case when a task wanted to drop off its function, it was considered to leave the implementation as it was.

Rare case? Incorrect. Think about using a task to implement a worker thread for an app. I want my code to execute and terminate when it is done. In fact, that is exactly the use case I had for doing this.

Also, that’s awesome about the stack. So you allow dynamic allocation, but you are not willing to give up the 4 bytes or whatever of local storage that vTaskDelete requires. Let me remind you, since you seem to have forgotten. At the point that the task terminates, the entire stack is unwound. Calling vTaskDelete at the end of the function is exactly the same as having the scheduler return the task to vTaskDelete. Study that one for a second. If the task didn’t have enough stack to call vTaskDelete at the end of its execution, then it doesn’t have enough stack for the scheduler to automatically return there.

I just looked, vTaskDelete requires around 8 bytes of stack storage for some local variables. For the ARM Cortex M port, the link register (LR) is set up in the stack already - no extra storage required. Please do not sit there and tell me some odd 8 bytes of storage is unreasonable, especially since my stack is already unwound (meaning I have the full stack space that was created with my task) in order to execute vTaskDelete. A general configuration of the minimal stack space for xTaskGenericCreate is 128 words. At this particular point in time, I am using at most (16) words of that for potential register storage. What you are basically saying is that the task has such a small stack that it cannot execute any code or API functions that require 8 bytes of local stack storage. That is definitely not the case.

Subsequently, the stack setup is in the port file! You just said “for smaller micros, not so for Cortex M.” If it’s OK to do it for CortexM and it only affects that port, then why not do it? I will give you the benefit of the doubt, such as running on a PIC10 or something that maybe this wouldn’t make sense; however, like I said, if you have enough stack to call any useful API function within a task, you have enough stack to call vTaskDelete and likewise enough stack for the task to return to vTaskDelete after it ends. It consumes 2 words of local storage.

Umm, yes. You want to take, but not give? All you have to do is upload the changes to the FreeRTOS interactive site, which can be done anonymously.

No no, you have it quite wrong here. This is not about giving and taking. The way you paint it is like its unicorns and butterflies. “All you have to do is post your new code anonomously. No big deal.” This has to do with distributing proprietary code from my company for the sole reason that FreeRTOS was not designed correctly in the first place to separate interface from implementation.

If I want to add an addition on to a piece of software, the interfaces should be such that I can do that with a completely sectioned off piece of code. That should ‘not’ require me to go and modify the source code of my library directly just because those interfaces do not exist. The standard operating procedure would be to make a completely separate code module using interface functions from the RTOS. If at that point I decide that my module that I made, that was on top of FreeRTOS is something the community should look at, I would then distribute that modularized code. This module could be as simple as wrapping function calls from the RTOS such that we can hook any scheduler we want into our system (which I described in my first post). Those wrappers should not have to exist directly in the FreeRTOS code base. I should not have to distribute my wrapper code for your RTOS just because it is required to exist directly inside your code. That is bad interface design.

Having to follow this procedure for something simple as static allocation should be a huge indication for you guys. FreeRTOS has been around for how many years and static allocation still does not exist? Let me inject another contradiction in your viewpoint.

You state in the “rare” case that a task wants to return. This implies that the task is never destroyed.
I heard on some forum post something along these lines: " FreeRTOS is an RTOS in which generally you always create things but never delete them.

So… based on the above two statements, what would we call that? static allocation! That’s what we would call that. An object that never gets deleted or destroyed but once created is always there.