Sporadic hard faults, how to find reason

gerhardkreuzer wrote on Saturday, January 30, 2016:

Hi,
my system has sporadic hard faults, this means, working over night, or crashes within minutes and all between.
I use the suggested code:

void prvGetRegistersFromStack( uint32_t pulFaultStackAddress )
{
/
These are volatile to try and prevent the compiler/linker optimising them
away as the variables never actually get used. If the debugger won’t show the
values of the variables, make them global my moving their declaration outside
of this function. /
volatile uint32_t r0;
volatile uint32_t r1;
volatile uint32_t r2;
volatile uint32_t r3;
volatile uint32_t r12;
volatile uint32_t lr; /
Link register. /
volatile uint32_t pc; /
Program counter. /
volatile uint32_t psr;/
Program status register. */

r0 = pulFaultStackAddress[ 0 ];
r1 = pulFaultStackAddress[ 1 ];
r2 = pulFaultStackAddress[ 2 ];
r3 = pulFaultStackAddress[ 3 ];

r12 = pulFaultStackAddress[ 4 ];
lr = pulFaultStackAddress[ 5 ];
pc = pulFaultStackAddress[ 6 ];
psr = pulFaultStackAddress[ 7 ];

/* When the following line is hit, the variables contain the register values. */
for( ;; );

}

Now I need some help to analyse the reason for the crash. All registers except psr were 0. Even pc is 0.
Any idea how to get closer to the problem?

Thanks again for helping.

With best regards

Gerhard

davedoors wrote on Saturday, January 30, 2016:

Doesnt sound right. Can you show the code that calls prvGetRegistersFromStack() too. You might need to dig deeper into the Cortex system registers to see what caused the hard fault (memory access, what ever). You could also use the trace tool with a circular trace buffer that is stopped in the fault handler.

gerhardkreuzer wrote on Saturday, January 30, 2016:

Thanks, any documents around to set up the trace tool with such a buffer configuration ?

With best regards

Gerhard

gerhardkreuzer wrote on Saturday, January 30, 2016:

Hi,
ok, tried to follow the instructions in this document, but it ends up getting 93 errors …
http://www.atmel.com/images/atmel-42382-getting-started-with-freertos-on-atmel-sam-flash-mcus_applicationnote_at04056.pdf

Maybe there is a version conflict, as the files having different names. In my source of the trace recorder I dont have a file called trcHooks.h, but I have trcKernelHooks.h.

I cut it out now, so my project compiles again,

Any suggestion how to get it running. Which port should I use for a sam4N, I choose ARM Cortex-M, maybe thats the mistake?

Thanks for helping

With best regards

Gerhard

rtel wrote on Sunday, January 31, 2016:

Any suggestion how to get it running.

If you are using Atmel Studio, then you may find the ASF already has a
simple FreeRTOS project for that part. Try using the ASF wizard, select
your part, and then see which projects are provided for that part (there
is likely to be a lot, a couple of which are likely to be FreeRTOS
projects).

Otherwise the following links might be helpful:

Wich port should I use for a
sam4N, I choose ARM Curtex-M, maybe thats the mistake?

With any Cortex-M4 microcontroller, if the microcontroller has a
floating point unit (FPU) then use the code from
FreeRTOS/Source/portable/[compiler]/ARM_CM4F, otherwise use the code
from FreeRTOS/Source/portable/[compiler]/ARM_CM3.

gerhardkreuzer wrote on Sunday, January 31, 2016:

Hi,
thanks.
But, I have a running application for my SAM4N and I use Atmel Studio V6.2. Now I want to add the trace recorder, but this wouldn’t work out …
In the file trcConfig.h I am ordered to enter a port option, here I choose Cortex-M, hoping this is the correct port (of the trace recorder) for my SAM4N.
The port of the FreeRTOS 7.3.0 I still use works correct for my app, nothing to change for now.

Its only about adding the trace recoder to an exixting application.

With best regards

Gerhard

gerhardkreuzer wrote on Sunday, January 31, 2016:

Hi,
to be more precise:

First problem appear in trcHardwarePort.c. In the function prvTraceInitCortexM, near the end the statements:

if (RecorderDataPtr->frequency == 0)
{
RecorderDataPtr->frequency = TRACE_CPU_CLOCK_HZ / HWTC_DIVISOR;
}

fail, cause the macros used here lastly end up calling ‘sysclk_get_cpu_hz’

Problem is now, which file where to include to solve this and not produce lots of duplicate definitions elswhere …

It sounds like a basic issue, maybe the document describing the integration of the trace recorder lib should be updated to reflect the needs of the current version.

With best regards

Gerhard

rtel wrote on Sunday, January 31, 2016:

This looks Atmel specific. I’m guessing TRACE_CPU_CLOCK_HZ is somewhere
referencing configCPU_CLOCK_HZ (which is in FreeRTOSConfig.h), and
configCPU_CLOCK_HZ is calling sysclk_get_cpu_hz (which I think is an
Atmel library function) rather than just hard coding the clock speed.

If you right click on TRACE_CPU_CLOCK_HZ you should find its definition,
and follow it back to the base definition. I’m guessing replacing
sysclk_get_cpu_hz() with the actual clock speed as a hard coded value
will fix the issue.

gerhardkreuzer wrote on Sunday, January 31, 2016:

… I actually trace this back as you suggested:
trcHardwarePort.c references trcKernelPort.h:

#if (USE_TRACEALYZER_RECORDER == 1)

/* Defines that must be set for the recorder to work properly */
#define TRACE_KERNEL_VERSION 0x1AA1
#define TRACE_TICK_RATE_HZ configTICK_RATE_HZ /* Defined in "FreeRTOS.h" */
#define TRACE_CPU_CLOCK_HZ configCPU_CLOCK_HZ /* Defined in "FreeRTOSConfig.h" */

#if (SELECTED_PORT == PORT_ARM_CortexM)

And in FreeRTOS.h I have the definitions as normal. Doesn’t changed anything here. Without trace recorder, my FreeRTOS based application works fine.

Here the macros:

#define configUSE_PREEMPTION			1
#define configUSE_IDLE_HOOK				0
#define configUSE_TICK_HOOK				0
#define configCPU_CLOCK_HZ				( sysclk_get_cpu_hz() )
#define configTICK_RATE_HZ				( ( portTickType ) 1000 )
#define configMAX_PRIORITIES			( ( unsigned portBASE_TYPE ) 5 )
#define configMINIMAL_STACK_SIZE		( ( unsigned short ) 150 )

If in FreeRTOS.h there were a mistake, my app would not compile, is this correct?

Ok, maybe keying in the CPU frequency for TRACE_CPU_CLK_HZ would help, the … DIVISOR also use this macro …

With best regards

Gerhard

rtel wrote on Sunday, January 31, 2016:

This is what I suspected:

#define configCPU_CLOCK_HZ ( sysclk_get_cpu_hz() )

Did you try replacing the call to sysclk_get_cpu_hz() with the hard
coded clock frequency, as just suggested?

gerhardkreuzer wrote on Sunday, January 31, 2016:

Yes, but now other issues popping up … endless strory
Has anybody tried it before, look and feel is, that this hasn’t worked out before …

Now the statement ‘’ include “trcKernelHooks.h” informed me, that this file isn’t available ??? I still can see the file. And with right click and ‘Go to implementation’ I get exactly there.

When I use an absolute path, the error is gone and I get 97 errors telling me about lots of duplicated definitions. So, better to exclude this file? But not included gives also an error. Really great stuff.

With best regards

Gerhard

gerhardkreuzer wrote on Sunday, January 31, 2016:

I add a shot. I exactly tried as in this document is suggested:
http://www.atmel.com/images/atmel-42382-getting-started-with-freertos-on-atmel-sam-flash-mcus_applicationnote_at04056.pdf

Doing both, adding a folder as you see in the attachment and adding a search path solves some how the duplicate definitions, but not every time. Depends which of the statements fail and which not.

It is a lottery …

With best regards

Gerhard

rtel wrote on Sunday, January 31, 2016:

Has anybody tried it before, look and feel is, that this hasn’t worked out before

Um, “yes”, a few hundred if not thousands of people, me included.

Attached you will find what I am using on an EFM32 - which is also a Cortex-M, so should be the same. The zip file has the files and path I have in the project, and the header file is the config file. These are from a training I did before Christmas, so might not be the latest version.

gerhardkreuzer wrote on Sunday, January 31, 2016:

Thanks, will see where is the difference. I have to use FreeRTOS 7.3.0, so far from the latest version, cause Atmel didn’t support an other version on SAM4N MCU.

Hope there is a way to have a ‘TRACE’ version and a DEBUG version of my project, have to work on the app not playing around with trace integration. Uptil now I didn’t need a trace, but once a day it would be helpful. Especially as I want to find the sopradic crashes, see start of this discussion. But if integration is that hard …

With best regards

Gerhard

gerhardkreuzer wrote on Sunday, January 31, 2016:

… where should I place the trcConfig.h file. I have a Config folder where the FreeRTOSConfig.h resides and some more, is this the exact place for it ? (I guess … but …)

With best regards

Gerhard

alainm3 wrote on Sunday, January 31, 2016:

I am following this and I have a different experience !!!

Up to now, ALL my sporadic hardfaults happend because of stack
overflow (CM3 and yes, I have been lucky…).
I included in the system a stack report function, almost exactly as
exists in the Examples sections and now I can diagnose that in seconds.

I tested the stack detection feature that exists in FreeRTOS specific
tests and I discovered that it failed to avoid the hard fault in most
tested situations…

Alain

On 31-01-2016 13:51, Real Time Engineers ltd. wrote:

Has anybody tried it before, look and feel is, that this hasn't
worked out before

Um, “yes”, a few hundred if not thousands of people, me included.

Attached you will find what I am using on an EFM32 - which is also a
Cortex-M, so should be the same. The zip file has the files and path I
have in the project, and the header file is the config file. These are
from a training I did before Christmas, so might not be the latest
version.

Attachments:


Sporadic hard faults, how to find reason
https://sourceforge.net/p/freertos/discussion/382005/thread/53b486f2/?limit=25#6592


Sent from sourceforge.net because you indicated interest in
SourceForge.net: Log In to SourceForge.net

To unsubscribe from further messages, please visit
SourceForge.net: Log In to SourceForge.net

gerhardkreuzer wrote on Sunday, January 31, 2016:

… same situation.

It is now a miracle for me, how to integrate trace into a project.

I alwas get the error, that Atmel Studio can’t find ‘trcKernelHooks.h’.

Lastly I tried, to type #include “”, than tab and he showed me a list of files. HE shows me the list and I think, he will find this files … I selected trcKernelHooks.h, and still get the … file not found error …

I also restarted the Atmel Studio 6.2, nothing helps.

Just a miracle.

Is it possible to add trace later or should I have added it at the beginning and now it is impossible for whar reason ever?

No more idea … sorry

With best regards

Gerhard

gerhardkreuzer wrote on Sunday, January 31, 2016:

… ok, now hanging at:

Error 34 implicit declaration of function ‘__get_PRIMASK’ [-Werror=implicit-function-declaration] …\src\TraceRecorder\trcBase.c 432 2

rtel wrote on Sunday, January 31, 2016:

Putting __get_PRIMASK into Google shows me that this is a CMSIS
function. Do you have the CMSIS headers included?

gerhardkreuzer wrote on Sunday, January 31, 2016:

No, there is no info about that. But the ASF have such files added. Ok, will try around