Since converting my project to run as 3 FreeRTOS tasks I have been getting a few hard faults each hour. The fault address is within the Balloc function in libc.a from the ARM M4 gcc library. I understand that the most likely cause is that even though my own code does not use dynamic memory allocation after the initialisation phase, library functions printf, sprintf, strtod etc. (all of which two of my application tasks call regularly) call malloc/free, and the standard library memory management functions are not thread safe by default. What is the recommended way of making the memory allocation performed by these library functions thread safe? Is there a library version of libc for ARM M4 that is already compatible with FreeRTOS?
I read that newlib’s malloc and free call __malloc_lock and __malloc_unlock functions to allow them to be made thread safe. So I implememted those functions using a recursive mutex, and verified that they are present in the linker map instead of the original versions. However, although the problem may be less frequent after doing that, I still had a crash after about 90 minutes - in _Balloc again.
Is it an option for you to not use the malloc() and free() functions at
all? I think GCC has options to direct these calls elsewhere, but if
not you can implement them yourself to just call pvPortMalloc() and
vPortFree(), as per:
In addition to Richard’s remark about replacing malloc() and free(), I have a question: what version of heap_x.c are you using? Here you’ll find a good explanation about the options that are available.
How to make printf/sprintf/strtod thread safe
Are you sure that mentioned functions allocate memory? I thought that they are thread-safe because they store all temporary data on stack ( but I’m not a GCC-expert ).
As the stack usage of printf() functions can be quite high, an alternative implementation is being used in some FreeRTOS demo applications, which you can download from here
Those functions are all thread- and ISR-safe ( although it is not recommended to use these functions from within an ISR ).
In these posts you will find more information about printf-stdarg.c:
One thing to check is what options newlib was built with, as the use of malloc_lock/mallooc_unlock is configured at library build time. One test is if you DON’T define those functions, do they show up in the link map? If not, then malloc/free isn’t calling them.
I am not using any of the heap#.c implementations, because I am using static allocation of tasks, mutexes, stacks etc. and the corresponding FreeRTOS calls CreateStatic calls.
I verified that my linker address map originally contained labels __malloc_lock and __malloc_free supplied by libc, and that after I replaced them, those were gone and only my own versons were present. So I conclude that newlib does refer to them.
I need to print floats, and I understand that the FreeRTOS version of printf etc. doesn’t support them. Perhaps I should add support for floating point formats to it.
I understand that strtod also allocates dynamic memory under some circumstances, so I need to handle that function too.
Replacing malloc/free by one of the FreeRTOS versions may be an option, but I am puzzled that adding the semaphore didn’t fix the problem.
Thanks for the links, I previously found some but not all of them.
Although my code does use strtod, it should never be called when the system is idle (which is when it was crashing with a hard fault eventually). Which leaves me puzzled.
I didn’t have configUSE_NEWLIB_REENTRANT set in my config. This would explain the problem I think, because strtod uses big-number structures that need to be per-task
Unfortunately, when I set configUSE_NEWLIB_REENTRANT and rebuild, I can get the system to run but the diagnostics indicate that RAM usage has increased by many kB and there is no longer any free memory. I think this is because of all the stuff in_struct reent that I dont need, such as the stdin/stdout/stderr stuff.
I think this leaves me a few options:
Try to save RAM elsewhere in the project - but I’ve already done this several times, and I want to add a lot more tasks so I can’t really afford a _reent struct for each one anyway
Compile my own version of newlb without the stdio stuff and other stuff that I don’t need in _reeent
Wrap calls to strtod in a mutex. This will work for strtod, but I may hit similar problems with other functions in newlib
Write my own version of strtod. I only need to parse floats, not doubles, so with luck I can skip the big number stuff and not need to use any heap.
@HS2, I am using that same version of the ARM GCC toolchain. I see that there is a more recent version now.
@Richard Barry, I wasn’t aware of the Launchpad version of GCC. The page about it says "As previously announced all new binary and source packages will not be released on Launchpad henceforth, they can be found on: https://developer.arm.com/open-source/gnu-toolchain/gnu-rm." That page is where I downloaded the GCC tool chain from. However, I am not explicitly using the nano version of newlib, so I will look into that.
I have written a strtod replacement and that seems to have resolved the problem. I had to define _strtod_r too to call my function because that is called internally - probably from sscanf which I am also using.
Although I solved the crash by providing my own version of strtod and friends, I still had an occasional problem with printf producing the wrong output for floating point values. It turns out when when printing floating point formats, printf uses a buffer in the _reent struct. As I cannot afford the large amount of storage associated with having a reeent struct for each task, I’ll have to find another solution. I’ve switched to newlib-nano and in that version, the floating point bits that use the reent struct are in a separate file, vfprintf_float.c. So it looks like I can just take that file, hack it it to use the stack instead and include those functions in my project. Or maybe just disable scheduling around it.
Normally, if the problem is only with floating point values, the root
cause is a misaligned buffer. Floating point requires 8-byte alignment.
The stack MUST be 8-byte aligned, but if if this is inside a FreeRTOS
task then it should already be - but maybe there are other buffers that
must also be?
Thanks, Richard. I don’t think that’s the problem, because the issue only occurs when 2 tasks are calling vsnprintf to print floats several times a second. The symptom is most obvious when one of the floats that was supposed to be printed just gets left out, because that causes a parsing error at the PC that receives the data. I’m fairly sure it is because both tasks are using the buffer in the same reent struct to convert the float.
In the short term I’ll replace calls to vsnprinf by a wrapper that disables the scheduler. In the longer term I’ll use a cut-down vsnprintf replacement,
I can confirm that replacing calls to vsnprintf in my application by calls to a wrapper that suspends/enables the scheduler around the actual call to vsnprintf fixed the problem.
In case anyone else has a need for a threadsafe printf with floating point support, I converted the FreeRTOS printf to C++, added support for %e and %f formats, and committed it at https://github.com/dc42/RepRapFirmware/tree/v2-dev/src/Libraries/General. I haven’t tested that the padding always works properly because this application doesn’t use padding. My threadsafe version of strtod is there too.
Thanks Dave, your notes would have been really useful had I found them earlier (or maybe you only just wrote them).
I found your recommended solution (#define configUSE_NEWLIB_REENTRANT 1) too expensive in RAM because the reent structure is so large. So I wrote my own strtod replacement (which doesn’t handle rounding of very large numbers correctly, but that doesn’t matter for this application), and added basic floating point support to the FreeRTOS implementation of printf. Having read your notes, I’ll take care to avoid using strtok too.