Out of memory in simulator but not on silicon

Kernel
1

Hi,
I am using a PIC32MZ1024EFG100T and when I compile and run the code through the debugger it seems to work correctly BUT when running on simulator it locks in a software breakpoint.

Michorchip tech support has gone through the code/project and replicated exactly the same issue and it seems to be caused by something related to freeRTOS.

Please see attached screenshots from Microchip related to this feedback of theirs:

FIRST MICROCHIP EMAIL (screenshot “software breakpoint 1.png”:

Thanks for providing the project, I tried on test machine and could able to duplicate the issue you mentioned. I see the below error message when I compile your project.
d:/microchip/xc32/v2.50/bin/bin/…/…/lib/gcc/pic32mx/4.8.3/…/…/…/…/pic32mx/bin/as.exe: out of memory allocating 4294967280 bytes

SECOND MICROCHIP EMAIL (screenshot “software breakpoint 2.png”:

I further looked at it and noticed that it is caused due to the below line of code.

                           pxNewListItem->pxNext = pxIndex;

Keep a breakpoint at /* Create OS Thread for APP_DEBUG_Tasks. */
xTaskCreate((TaskFunction_t) _APP_DEBUG_Tasks,
Then in list.c, keep a breakpoint on this line pxNewListItem->pxNext = pxIndex; in

function void vListInsertEnd( List_t * const pxList, ListItem_t * const pxNewListItem .

The general exception is caused due to the Load or Store.

With reference to the comment in their first email “…memory allocating 4294967280 bytes” no idea where such huge number could come from. I don’t have any dynamic memory allocation in my code. The only dynamic allocation is done by freeRTOS for the various tasks etc. Each task is also very “simple” and short with minimal memory allocation.

The only data that is not just few bytes is a large B&W image stored in a fixed array declared as:

uint8_t image_data[40,000];

BUT…:

  1. even that size is not even remotely close to the size mentioned in the error
  2. that variable is not dynamically allocated
  3. it is a global variable so not related to any task anyway

Furthermore, following useful feedback from this forum, I also changed the heap strategy to “heap_4.h” long time ago and haven’t had any issues on the debugger or simulator since then.

IMPORTANT: The issue is that the debugger does not support that software breakpoint so have no way of finding out if the same is occurring on the silicon without showing any symptoms but still being a potential issues in time with allocation and de-allocation.

How can I find out what causes it and how to make sure the code is safe?

Thank you as always! :slight_smile:

2

One comment, the size mentioned is equal to -16, so if something is computing a needed size but got a negative number into the calculator, you might get that, like a queue with a negative number of elements. Look back that call chain to see if that could be the issue

3

tells, that the assembler of the cross-toolchain got an internal problem while building the application. Maybe it’s just buggy (and pretty outdated, if it’s GCC 4.8.3).
I think it’s completely unrelated to your / FreeRTOS code provided your build environment is ok.
Sorry - can’t say much regarding the simulator issue. I don’t use it.

4

Is the simulator you are using the MPLAB simulator? If so I don’t believe FreeRTOS has ever been compatible with that simulator because FreeRTOS uses features (possible software interrupts, but I can’t remember) that are not simulated. Hence you are probably seeing corruption due to program flow errors.

5

Thank you both Richards and Hartmut,

Richard Barry,
hopefully that’s all it is. How can I check for sure? Is there some freeRTOS catch function for out of memory issues that I can see when debugging or at runtime (and perhaps customise to output to a serial port in the field when running long-term full-load testing)?

Hartmut, Re. the outdated GCC, where did you see version 4.8.3? The only compiler version I can see is V2.50 of the XC32 compiler. I think that is based on GCC but cannot find anywhere which GCC version that is. Was v. 4.8.3 referring to something else?

Richard Damon, Re. the -16 couldn’t find anything with that reference.

Thank you all! :slight_smile:

6

Look for Malloc Failed Hook Function: FreeRTOS - RTOS hook (callback) functions for task stack overflows, tick interrupts, idle task, daemon task startup, and malloc failure (pvPortMalloc() returning NULL)

Thanks.

7

Thank you Gaurav,
I see that function is present in my code. This is how it appears:

void vApplicationMallocFailedHook( void )
{
   /* vApplicationMallocFailedHook() will only be called if
      configUSE_MALLOC_FAILED_HOOK is set to 1 in FreeRTOSConfig.h.  It is a hook
      function that will get called if a call to pvPortMalloc() fails.
      pvPortMalloc() is called internally by the kernel whenever a task, queue,
      timer or semaphore is created.  It is also called by various parts of the
      demo application.  If heap_1.c or heap_2.c are used, then the size of the
      heap available to pvPortMalloc() is defined by configTOTAL_HEAP_SIZE in
      FreeRTOSConfig.h, and the xPortGetFreeHeapSize() API function can be used
      to query the size of free heap space that remains (although it does not
      provide information on how the remaining heap might be fragmented). */
   taskDISABLE_INTERRUPTS();
   for( ;; );
}

So it looks like it “locks” the entire code execution in that for loop and because in the line above it disables the interrupts then not even freeRTOS core functions can exit that “lock”. Which means that in both debug with the emulator and also at runtime IF it reaches that point absolutely no other code will be executed from then on. Hence the issue would become apparent pretty much straight away.

Is that correct?

Thank you

8

Yes, that is correct. If you break in debugger, you can immediately see which allocation is failing. However, it is meant for debugging only and should not be done in production - handle the allocation failures in production.

Thanks.

9

Thank you Gourav! :slight_smile:

10

The lib path usually contains the GCC version.
gcc.exe —version tells you the version.

11

Thank you Hartmut,
woow, I just seen that the latest version is 10 and the latest XC32 (updated few weeks ago) indeed, like you say, uses 4.8.3

Any reason why they use such an old version?

Also at the moment, compiling takes relatively long time at the moment… I assume the latest versions of GCC can take advantage of multithreading that 4.8.3 does not, correct? Or can I enable multi-threading on that old version too? If so, how do I enable multithreading (ideally through the MPLAB interface - if you are familiar with it)?

Thank you :slight_smile:

12

GCC and compiling in general is not really suitable for multi threading. It’s just mainly a sequential thing. However, parallel invocation of the compiler is the way to go making use of multi processing to speed up builds.
This in turn is up to the build system/tool like GNU make. make supports a job server option -j to specify the number of parallel compiler invocations. Usually the number of processors of the build machine is a good value. On Windows there is a system environment variable NUMBER_OF_PROCESSORS, which can be used.
But for sure the GCC team has put effort into speeding up compilation as well.

13

I have no idea why Microchip uses such an old version. Do you know if there are specific technical reasons?

More importantly, can I somehow use the latest gcc (instead of the very old 4.8.3) with MPLAB? Like perhaps replacing the old gcc.exe with the latest, and do so safely?

Thank you

14

Seems that Microchip PIC32 compiler, even GCC based, is as it is. Well, it’s probably by far not as widespread as e.g. ARM and therefore don’t get the superb support by the community (and ARM). I think, you have to live with it and it’s probably not be as bad or old as it might seem even if the version number tells another story.

15

It sounds like you are familiar with both Microchip and also ARM and perhaps others too. Through the years I kept designing in Microchip micros purely out of habit and thinking they were the biggest players especially after buying Atmel. But I have just looked at market share of the various vendors and that’s not the case.

But one thing I do like about the Microchip dev tools is the Microchip Code Configurator!!

In your opinion what are alternative platforms (micros, software toolchains, vendors etc) which are generally better and more modern (in as many different fronts as possible)?
And which:

  • have something like Microchip Code Configurator so the entire setup of the micro can be done visually and literally minutes instead of hours or sometimes days
  • the dev tools are free

Thank you :slight_smile:

16

Hi Rik,

from what I can tell, the market standard at this time is Cortex M family, in particular M4, with ST in the lead, followed by NXP.

In terms of ecosystems: Virtually every vendor has its own eco system, normally consisting of (normally “free” as in “no fee” which is not the same as no cost) HALs, configurators and such - frequently all in the same IDE.

The reality behind that is that ARMs business model (licensing universal cores) is incompatible with the vendor’s business model (locking in customers to their platforms), so the vendors attempt to reestablish vendor lock in by making proprietary eco systems, with the promise to click together apps in minutes. At least that’s the theory… as we can tell from recent threads in this forum, even such a “straightforward” task as integrating FreeRTOS into a “click and run” configuration is not always straightforward.

So simply go browsing; every manufacturer of Cortex M4 based chips has the software you are looking for.

1 Like