Memory, tools, and overflows

Kernel
1

This might be a bit long, but it describes a problem I’ve been having.

Environment STM32L562VETQ processor: ARM, 512K Flash, 256K RAM, V 10.3.1. C++ overlay on basic HAL drivers (in C). C++ overlay is the major driver calling C low level routines where needed.

Additional memory added and memory mapped to processor address space.

Routines include graphics (writes to virtual memory screen and then updates display), SPI support, I2C support, Serial support, NRF24L01 mesh networking. It’s a complicated system.

Heap4 memory used and mapped to additional memory. (256K allocated, which is the maximum, why?).
Routines and memory are allocated, but never released. Buffers and drivers are created with “new” which has been made thread safe as a malloc wrapper. Arrays are directly created as normal.

My problem (at the moment, which I worked around) is that I increased the size of an array by adding a few integer elements, and the array initialization wiped out a driver created by “new”.

FreeRTOS did not make any comments about the stack being overrun. That doesn’t help, nor do I see any routines that might help diagnose the problem, not in FreeRTOS.

So that’s an indicator of difficulty, even though FreeRTOS may not be the tool to solve it.

I worked around this by writing my own memory manager (somewhat similar to heap4 in concept) that pointed to an unused area (256K bytes) of additional memory, and using it to store some of the larger arrays. This, and resizing the heap memory for particular tasks seems to have worked.

So essentially two things, at least.

  1. Does FreeRTOS have any tools that would help? I already asked for a modification to the malloc failed hook that would provide how much was asked for (we don’t always know) and by how much we failed. This proves very useful in adjusting stack sizes. From what your documentation says (and I believe it), that’s one of the major difficulties in trimming any FreeRTOS system.
  2. If FreeRTOS had an additional memory manager, or at least allowed two of them, then in this case at least, I would not have had to write my own. I know that heap5 manages two separate memory regions as one, and that would not be a problem except for:
    A) is the same 256K memory limit still extant? Ends up with the same problem as heap4.
    B) it needs to manage a separate area where FreeRTOS stacks cannot overrun existing
    data, in fact, the separate memory manager guarantees it.
    C) Note that there were NO stack overflows at all, and the Malloc overflow hook did not
    trip.

For all I know, this is a C++ and C problem, but as far as I can tell, shouldn’t do this.
I’ve worked around this, but I’m looking for a more graceful solution.

What tools does FreeRTOS have to help solve this problem?

Thanks

2

Hello @madyn, you are describing a complex system, and it is not entirely clear what the problem is.

Does the application crash or hang?

Memory allocation or usage seems to be an issue?

Heap4 memory used and mapped to additional memory.

Are you sure that you have 256KB available for the heap?

How do you get “additional memory”? Is that entirely available for the heap?

256K allocated, which is the maximum, why?

How is configTOTAL_HEAP_SIZE defined?
The heap modules do not have some build-in maximum as far as I know.

Mind you that the word “malloc” has two meanings:

  • memory allocation in general
  • a C function malloc() that allocates memory.

Buffers and drivers are created with “new” which has been made thread safe as a malloc wrapper.

So here, what do you mean with “malloc”? The clib heap functions or vPortMalloc()?

IF you overloaded new() and delete(), did you also check that they are really called?

Could you produce a list of all memories: starting offset and size.

Consider using heap_5 in case the actual memory sizes are only known “at runtime”, ie. after linking. I use two symbols produced by the linker:

    extern uint8_t __bss_end__, _estack
    #define HEAP_START    ( ( uintptr_t ) &__bss_end__ )
    #define HEAP_END      ( ( uintptr_t ) &_estack )

and I use this to feed heap_5. Please check you linker file to find the above symbols.

Either make sure that malloc()/free() are not called, or make sure that they point to vPortMalloc()/vPortFree().

I recommend reading about FreeRTOS Memory Management.

This is also interesting: Stack Usage and Stack Overflow Checking. FreeRTOS can help you detecting stack overflows, but sometimes an exception occurs before an overflow was detected.
How big are your stack sizes now?

EDIT

heap_5 is useful :

  • If you have multiple blocks of heap memory
  • If the size of the heap memory is not known at compile time

And none of the heap plans have a limit of 256 KB.
heap_x can also be used on CPU with a GB of external SDRAM.

3

Does the application crash or hang?

Once a memory assignment to a chip driver is overwritten, it crashes when that chip driver is used.

Memory allocation or usage seems to be an issue?

Seems so to me.

Are you sure that you have 256KB available for the heap?

I have 320K available to FreeRTOS.

How do you get “additional memory”? Is that entirely available for the heap?

The STM32L5 series can have a QSPI interface. I have a 64Mbit (mapped as 2 M * 32 bits) SDRAM chip, which is very reliable. That memory space is split into display virtual memory screens, a FreeRTOS memory block, an Xheap block, and a trace memory buffer. Those regions are defined in the linker. FreeRTOS is told to use external memory. The 256K limitation is what is reported by CubeMXIDE in setting up the available memory for FreeRTOS. The entire 320K is available.

How is configTOTAL_HEAP_SIZE defined?
The heap modules do not have some build-in maximum as far as I know.

as 250000 bytes. The limitation seems to be imposed by CubeMXIDE in their setups. < May be based on what the processor memory normally is. (clever software!)

So here, what do you mean with “malloc”? The clib heap functions or vPortMalloc()?
IF you overloaded new() and delete(), did you also check that they are really called?

New() is redirected to vPortMalloc(). With this design, I need not release memory.
I have not checked to see that it is called, however.

Producing a list of all memories may be difficult, as the assignments are done at run time depending on a series of #ifdef configuration statements.

I could use heap5, but it does not help with the memory management problem, not as I understand it.

In the linker, the linker uses the following code:


  /* User_heap_stack section, used to check that there is enough "RAM" Ram  type memory left */
  ._user_heap_stack :
  {
    . = ALIGN(8);
    PROVIDE ( end = . );
    PROVIDE ( _end = . );
    . = . + _Min_Heap_Size;
    . = . + _Min_Stack_Size;
    . = ALIGN(8);
  } >RAM

I have not changed any of that code.

Stack sizes vary per task, and I use a modified malloc failed hook to determine how to set the task heap sizes. That may be a problem depending on how much memory each task really requires, hence the use of a separate memory allocator.

Again, I suspect that the 256K limitation (which I do not need as long as I can allocate arrays elsewhere) is set by the FreeRTOS setup in CubeMXIDE. It had not occurred to me that they’d be so helpful.

The way the software is written sets up data structures with fixed sizes, and does not need (IMHO) to release memory.

The major memory consumers are the virtual screens for the graphics displays (already allocated and not part of FreeRTOS), and the memory arrays for DHCP mesh networking, and communications buffers. Those are no longer part of FreeRTOS since the Xheap memory manager uses a different area of memory and does not use new(), or vPortMalloc().

Thanks for the reply. I do have a request in for a smarter malloc_failed hook.

4

The 256k limit may be enforced by cross dependencies with the static layout imposed by your linker command file, meaning that the lcf must be kept in sync with a static definition in your code. Unfortunately, that happens quite frequently, and there is no way to detect these kinds of things at cmpile time. Do inspect your linker command file.

5

Are you trying to debug this memory corruption? If so, have you figured the root cause of this memory corruption?

You can manually edit the configTOTAL_HEAP_SIZE value in FreeRTOSConfig.h to confirm that.

6

I was trying to debug the memory corruption, and found at least one thing. The structure definition of the array had a line of code that would zero the union. (I tend to use unions of a binary array of bytes and the main structure itself, allows easy writing to EEPROMS). Removing that zero helped.

Relocating the array to managed memory and taking it out of the FreeRTOS managed area seems to have been the “so there!” part of the solution, which also solves the problem.

Regarding the heap size, I have not bothered to change it, since using my own memory management routines (for arrays), removes a lot of the need for more memory in the FreeRTOS tasks. I have yet to go back and adjust sizes for heap space. No heap overflows have been caught, since I do not use malloc, that hook does not trigger.

I’m not sure how FreeRTOS handles normal memory areas, and using C++ rather than C for the main coding likely obfuscates the solution.

Thanks.

7

The problem may still be there, just coincidentally not manifesting itself. You should really be sure to know the root case. Undercover bugs will make your life more miserable later.

8

As best I can determine, the following factors seem to be in play.

  1. initialization code got called out of sequence. Removing that code seems to eliminate the problem. In the current (10.3.1) version of FreeRTOS, there are no tools that I know of that will help diagnose that.

  1. Possible C vs C++ structures problems. I don’t know, I only suspect.

  2. I suspect that somehow FreeRTOS memory management is responsible for some of this, so the easiest way to bypass that is to move memory hogs (arrays) out of the control of FreeRTOS. FreeRTOS does well in doing what it’s designed to do, and the situation of C++ programming will likely mess it up. (It certainly messed up Azure, which ST Micro now loves).

With the fixes I put in, the very dependable bug of trashing a driver is gone. The program ran for about 24 hours passing packets back and forth at the rate of about 1/second (pings) in a mesh network, so that was a plus.

Regarding the root cause, while I think I understand it, it occurs in programming I have no control over, (certainly the C++ compiler). What tools does FreeRTOS provide that would help diagnose this situation?

9

Please share the code that you suspect got called out of sequence.

It is only possible to answer that if you tell us exactly what is the root cause that you determined. Please share the code snippet which you think is the cause of the problem and your solution.

10

I think that the code (since replaced) was of the variety that reset elements in an array, followed by a call to a driver (memory mapped for a PCA9634 LED controller.

I cannot find that code since it has been rewritten sufficiently.

The problem was that after initializing the array, the pointer to the LED controller driver was set to 0x2. A meaningless value for the program, which might be significant if I had the original code.

Now the other part that I can perhaps answer is related to this definition:


union DHCP_MAP_type
{
	struct	DHCP_MAP_data				data[256];
	uint8_t								bin[256 * sizeof(struct DHCP_MAP_data)];
//	DHCP_MAP_type() { memset( this, 0x0, sizeof( DHCP_MAP_type ) ); }

};

If the union is not properly instantiated, that is, it’s created out of sequence somehow (more later), then the memset will crash the system.

This requires that the declaration be a non-pointer declaration. I did have it (at one time) a pointer, but the way the code seems to work tried to execute the memset with null pointers.

Removing the memset and initializing the data elsewhere solved the problem.

So while I would like to be able to provide you the data you ask for, it no longer exists in code form, so all I am left with is guesses.

Thanks

11

Clearing the whole object, and not just the data array in it can be a problem if you use some of the other features of C++ in the class.

I would think you would want to memset(this->bin, 0, sizeof(this->bin));

12

All I was interested in was providing a zero’d structure. IIRC, that did not work well, so I simply added code later in the “init” routine (I use create and then init) to zero out things.

I suspect that C++ and C do not play as well together as I’d like. I would like to have C++ versions of the support code for CubeMXIDE, and also FreeRTOS. I see neither event happening.

It seems to be well behaved (other than a variable overrun on an error counter and transaction counter).
It ran for 24 hours pinging the master node at about 1.5 second intervals using an NRF24L01 mesh network.

13

This is still not clear.

Are you saying that you declared a pointer and then called memset on it? Something like the following:

DHCP_MAP_type * ptr;
memset( ptr, 0, sizeof(DHCP_MAP_type));

If that is the case, it is wrong because no memory is allocated for DHCP_MAP_type and likely ptr will be initialized to NULL if it is global.

Glad that your issue is resolved.

14

Blockquote
11h

This is still not clear.

What you are seeing is the sequence of rewrites which eventually solved the problem. Note that these are in QSPI memory mapped into processor space, but that mapping and initialization occurs in C code in main.c before any reference is made to the memory arrays.

  1. as written: Declared memory array as part of a C++ class. use NEW() to create instance of a different class. Writing memory in the array trashes the pointer to the different class. It does so reliably.

  2. Next step: Memory array in C++ class changed to pointer, using NEW() to create instance. This crashes because the code used to zero the data union is attempted to be executed by the C++ initialization code. Solution: remove union initialization. Remember, this is stepwise improvement.

  3. Current solution. Use a different memory manager (not from FreeRTOS) to set up additional memory area managed somewhat like a FAT table. This area is completely separate from the FreeRTOS area. Mapping automatically consolidates contiguous memory and uses a first fit algorithm. Memory manager removes management of memory from FreeRTOS, and since pointers are initialized gracefully, no C++ initialization code interferes other than zeroing the pointer.

Glad that your issue is resolved.

It is, and thanks for the attention:

What I would like to see in FreeRTOS are tools to deal with this (I think there are none), and it would be nice to have a memory manager that can run in parallel with, say, heap4. Like heap5 it would need a pointer to a region and a memory limit. Unlike 5, it would be callable with a slightly different call to malloc() etc. I already have this programming for my purposes.

15

All you need to do that is make a copy of the heap file you want to use, change the name of the functions it defines and the memory block it is using (if it is a global). Then you have a new version of pvPortMallocx/vPortFreex to use. (or call it mymalloc/myfree)

16

Thanks, that’ll go on the list.