Microblaze Trace

Kernel
1

debugasm wrote on Sunday, October 30, 2016:

Hi,

I have download “Tracealyzer for FreeRTOS” to do some testing on Microblaze CPU. I have used “FreeRTOS Snapshot Recorder v3.0.9.zip”. I followed the guide but I somehow doubt from this point on:

I have create a project with one task that flashes a LED. The program runs smoothly with no integrated trace.

FreeRTOS v9.0.0 with Microblaze CPU + Snapshot Recorder v3.0.9 from Percepio

I have add this code at the end of “FreeRTOSConfig.h”:

/* Include the FreeRTOS+Trace FreeRTOS trace macro definitions. */
#define TRACE_ENTER_CRITICAL_SECTION() portENTER_CRITICAL()
#define TRACE_EXIT_CRITICAL_SECTION() portEXIT_CRITICAL()
#include "trcKernelPort.h"

The compilation is successful, but if I start the program this crash. In debug I noticed that after three, four calls to the Trace functions, the program crash.

I then decided to change code in “FreeRTOSConfig.h” with this:

/* Include the FreeRTOS+Trace FreeRTOS trace macro definitions. */
#define TRACE_ENTER_CRITICAL_SECTION() /* portENTER_CRITICAL() */
#define TRACE_EXIT_CRITICAL_SECTION() /* portEXIT_CRITICAL() */
#include "trcKernelPort.h"

I basically removed the definitions of “ENTER” and “EXIT” from critical section, In this way the program runs quietly and I can generate the trace in memory statically and then be discharged after a few minutes of operation.

The thing is very strange, but how can I check ?

I would be confident about this, because I would buy the tool, but I would try the real utility before proceeding.

Thanks very much.

debugasm

2

rtel wrote on Sunday, October 30, 2016:

A couple of questions:

Did you call vTraceInitTraceData(); before calling uiTraceStart()?

Did you see
http://percepio.com/2014/10/27/how-to-define-critical-sections-for-the-recorder
?

3

debugasm wrote on Sunday, October 30, 2016:

Yes, I have call both functions before start scheduler.

I have follow the FreeRTOs example for Win32 :

“FreeRTOS_Plus_CLI_with_Trace_Windows_Simulator”

but with the appropriate changes for Microblaze.

The function:

portENTER_CRITICAL()
portEXIT_CRITICAL()

Already allows nested interrupts, right ?

I also followed the guidance for Zynq:

http://percepio.com/2016/09/13/tracealyzer-for-freertos-on-xilinx-zynq/

debugasm

4

rtel wrote on Sunday, October 30, 2016:

I don’t think interrupt nesting is supported on the Microblaze (not to
be confused with critical section nesting, which is supported). The
Microblaze macros make use of the Xilinx library function
microblaze_disable_interrupts() to disable interrupts, and that does not
appear to support nesting.

5

debugasm wrote on Sunday, October 30, 2016:

It’s true but FreeRTOS have this :

/* Critical section macros. */
void vPortEnterCritical( void );
void vPortExitCritical( void );
#define portENTER_CRITICAL(){						  \
    extern volatile UBaseType_t uxCriticalNesting;		\
	microblaze_disable_interrupts();					\
	uxCriticalNesting++;								\
}

#define portEXIT_CRITICAL()	{						  \
    extern volatile UBaseType_t uxCriticalNesting;		\
	/* Interrupts are disabled, so we can */			\
	/* access the variable directly. */					\
	uxCriticalNesting--;								\
	if( uxCriticalNesting == 0 )						\
	{													\
        /* The nesting has unwound and we 				  \
              can enable interrupts again. */				 \   
               portENABLE_INTERRUPTS();						  \
	}													\
}

So it can not be used with FreeRTOS+Trace on Microblaze ?

6

davedoors wrote on Sunday, October 30, 2016:

Microblaze is the Xilinx softcore, Zynq is an ARM CortexA9. Dont follow the instructions for a totally different part!

7

debugasm wrote on Monday, October 31, 2016:

Sorry, I know Zynq and Microblaze very well, I have used and use both long enough.

Before you start writing a line of code I have documented “on principle of operation of trace”.

Know that Zynq and Microblaze are different, and that are not compatible. But understand the principle, you can work on any cpu.

I followed the examples and guides:

 To integrate the recorder trace library in an existing FreeRTOS project, the following steps are suggested:

(1) Check that your FreeRTOS version is v7.3 or later.
(2) Locate the Snapshot recorder library package via the Help menu 
        in Tracealyzer and extract its contents.
(3) Add the .c files from GenericRecorderLibSrc in your 
        build project.
(4) Add GenericRecorderLibSrc/Include to your compiler's 
        include path (i.e., in the project settings).
(5) Copy GenericRecorderLibSrc/ConfigurationTemplate/trcConfig.h 
        to GenericRecorderLibSrc/Include.
(6) Open trcConfig.h and...
        
        - Set SELECTED_PORT to match your target processor.
        - Verify that the FREERTOS_VERSION setting matches your 
            version of FreeRTOS.
        - Check the N[Type] settings (e.g., NTask, NQueue, etc.). 
            They define the maximum allowed number of simultaneously
            active objects, i.e., objects that have been created by 
            not deleted.
        - Check the other settings, especially TRACE_RECORDER_STORE_MODE
            and EVENT_BUFFER_SIZE. You don't need to change them, but we
            recommend you have a look. 
            
(7) In your FreeRTOSConfig.h, make sure you have the following
         setting.

        - #define configUSE_TRACE_FACILITY 1

(8) Add the following line in the very end of your FreeRTOSConfig.h

        #ifndef __ASSEMBLER__
            #if configUSE_TRACE_FACILITY == 1
            	#define TRACE_ENTER_CRITICAL_SECTION() portENTER_CRITICAL()
                #define TRACE_EXIT_CRITICAL_SECTION() portEXIT_CRITICAL()
		        #include "trcKernelPort.h"
            #endif
       #endif

(9) In your main() routine, call vTraceInitTraceData() as early as
         possible. It must be placed before any other calls to the
         recorder or FreeRTOS.
    
(10) Call uiTraceStart() at the point you wish to begin recording,
           in the startup or at some later point in the application code.
           Remember to check the return value. A value of 1 indicates
           that the recorder configuration was OK. Otherwise an error
           message is found in RecorderDataPtr->systemInfo.

Compilation successful, but FreeRTOs crash. After comment “portENTER_CRITICAL()” and “portEXIT_CRITICAL()” all work but I can not say if properly although the trace buffer is filled normally.

Other ideas ?

debugasm

8

rtel wrote on Monday, October 31, 2016:

I posted this yesterday, but for some reason it has not shown up. It might show up twice now…

I’m reading the page I just linked to and can’t see why you are coming to that conclusion. From the page:

"If you are using a FreeRTOS port that doesn’t implement portSET_INTERRUPT_MASK_FROM_ISR and portCLEAR_INTERRUPT_MASK_FROM_ISR, you need to define TRACE_ENTER_CRITICAL_SECTION and TRACE_EXIT_CRITICAL_SECTION yourself (see trcKernelPortFreeRTOS.h.). However, in such FreeRTOS ports there are no nested interrupts, so critical sections are only needed in task-context. In this case, a generic solution is:

#define TRACE_ENTER_CRITICAL_SECTION() if (IN_TASK_CONTEXT) portENTER_CRITICAL()
#define TRACE_EXIT_CRITICAL_SECTION()  if (IN_TASK_CONTEXT) portEXIT_CRITICAL()

Another solution is to implement a something like the portSET_INTERRUPT_MASK_FROM_ISR solution above. That design is required if your chip allows for nested interrupts."

So it looks like you just need to implement IN_TASK_CONTEXT for the Microblaze, in other words, find a way of knowing if you are inside a critical section or not, and only call portENTER/EXIT_CRITICAL() when you are.

Is there a register you can read in the Microblaze that tells you if you are in an interrupt or not? Normally there would be such a thing.

…just looking now.

The mfmsr() function returns the ‘machine status register’ which has a Exception In Progress bit (EIP, bit 22). If that is 1 then a hardware exception is in process. I’m not sure if it is also set to 1 when a standard interrupt, rather than an error exception, is in process but you can check that. So IN_TASK_CONTEXT would then become:

#define IN_TASK_CONTEXT ( ( mfmsr() & ( 1 << 22 ) ) == 0 )

NOTE: I have not checked if the bit is set inside an interrupt, as well as an exception, so please check that and find an alternative if it isn’t. These are not FreeRTOS questions though, but hardware questions, so I will leave that to you ;o)

9

debugasm wrote on Tuesday, November 01, 2016:

As always with your help, you reach always the goal.

I thought that FreeRTOS+Trace was already integrated enough, but from what
you’ve made me read, not for all CPUs.

Taking a cue from what you have written, I have documented better.

Basically:

Machine Status Register (MSR)

EIP = bit 22
	
	Exception In Progress
	
	0 = No hardware exception in progress
	1 = Hardware exception in progress
	Only available if configured with exception support

Hardware Exceptions (EIP) trap the following internal error conditions: 

	illegal instruction
	instruction and data bus error
	unaligned access

with hardware divider 

	divide exception

with hardware floating point unit:

	underflow
	overflow
	float division-by-zero
	invalid operation
	denormalized operand error

with a hardware Memory Management Unit:

	Illegal Instruction Exception
	Data Storage Exception
	Instruction Storage Exception
	Data TLB Miss Exception
	Instruction TLB Miss Exception

This bit remain untouched during interrupt therefore it can not be used for the
purpose.

Documenting best I found this:

Machine Status Register (MSR)

IE = bit 30

	Interrupt Enable
	
	0 = Interrupts disabled
	1 = Interrupts enabled
	
The processor disables future interrupts by clearing the IE bit in the MSR. The 
IE bit is automatically set again when executing the RTID instruction.

So during any interrupts in the processor clear MSR “IE” (bit 30) to prevent
others interrupts (prevents nesting).

So I modified the code suggested by you in this way:

/* During any interrupts in the processor clear MSR "IE" (bit 30) */
#define IN_TASK_CONTEXT ( ( mfmsr() & ( 1 << 30 ) ) == 1 )

#define TRACE_ENTER_CRITICAL_SECTION() if (IN_TASK_CONTEXT) portENTER_CRITICAL()
#define TRACE_EXIT_CRITICAL_SECTION()  if (IN_TASK_CONTEXT) portEXIT_CRITICAL()

Compiling again with these changes and restarting the debugging, now everything
works perfectly.

I also modified version of “Trace-Streaming” using a serial port and it works
perfectly well with this.

I do other tests, if not finding other problems, I would say that everything
works fine.

Inspecting the first recorded Trace would say that the tool is amazing, you can
see many things more. Really nice.

Thanks very much.

debugasm

10

johankraft wrote on Monday, November 07, 2016:

Many thanks for documenting your Microblaze fix. Since FreeRTOS is available for so many architectures, we have not yet been able to provide official ports for all. The current Microblaze port is tagged as “unofficial”, meaning that we have received it from a third party but not yet verified it ourselves. But we will take this opportunity integrate your fix, verify and provide it as an official port.
I hope this can be done by the Tracealyzer v3.1.1 release, probably in december. The upcoming release v3.1.0 is in code freeze now, a shame I didn’t see this earlier…

For questions regarding the FreeRTOS trace recorder, I recommend contacting support (at) percepio.com, where our friendly support engineer Niclas will help you out.

11

debugasm wrote on Tuesday, November 08, 2016:

The working code is this:

#if (SELECTED_PORT == PORT_XILINX_MICROBLAZE)
	#define TRACE_SR_ALLOC_CRITICAL_SECTION() int __irq_status;

    /* Byte Order */
    #if XPAR_MICROBLAZE_ENDIANNESS
        /* 
         * MicroBlaze Little-Endian Working
         *
         * If interrupt enable, interrupt clearing the IE bit in the MSR 
         */
        #define IN_TASK_CONTEXT ( ( mfmsr() & ( 1 << 1 ) ) != 0 )
    #else
        /* 
         * MicroBlaze Big-Endian Not Tested !!! 
         *        
         * If interrupt enable, interrupt clearing the IE bit in the MSR
         */
        #define IN_TASK_CONTEXT ( ( mfmsr() & ( 1 << 30 ) ) != 0 )
    #endif
    
	#define TRACE_ENTER_CRITICAL_SECTION() __irq_status = IN_TASK_CONTEXT; if (__irq_status) portENTER_CRITICAL()
	#define TRACE_EXIT_CRITICAL_SECTION() if (__irq_status) portEXIT_CRITICAL()
#endif

Add this at the end of “FreeRTOSConfig.h”

/* Prevent the function prototypes being included from asm files. */
#ifndef __ASSEMBLER__
	#if configUSE_TRACE_FACILITY == 1
		#include "trcKernelPort.h"
	#endif
#endif

If user want same address for recordi data use linker script to place recorder data on section of memory:

RecorderDataType RecorderData __attribute__((section(".trace")));

On Linker script:

MEMORY
{
...
TRACE_BASEADDR : ORIGIN = 0x84000000, LENGTH = 0x00400000
...
}

.trace (NOLOAD) : {
   . = ALIGN(8);
   _trace = .;
} > TRACE_BASEADDR

I did not forget anything, I think.

debugasm

12

I went through this exercise on a more recent release of Xilinx tools (2019.1) and Percepio Tracealyzer (4.4.1). So just updating this topic with what I found.

  • First off, because the Xilinx IDE controls/automates the FreeRTOSConfig.h file, including trcRecorder.h with FreeRTOS.h avoids undesired loss of changes to the FreeRTOSConfig.h file. This sort of change is also better supported by the Xilinx tool, which allows for custom code repos. That is, you can clone the FreeRTOS source code to a new location, make the needed changes, then provide this path to the Xilinx IDE. Such custom repos will take precedence over default ones.

  • So I made changes by cloning /data/embeddedsw/ThirdParty/bsp/freertos10_xilinx_v1_3 to a new location and then adding all the new Tracealyzer files (trc*.h) to that location. Unfortunately, the tools ignored the trc*.c files, so I still had to add these to the application project.

  • Below is a summary of all the mods I made to this cloned location. Mostly to the additional Tracealyzer files.


diff -bur embeddedsw_def/ThirdParty/bsp/freertos10_xilinx_v1_3/src/Source/include/FreeRTOS.h embeddedsw_mb/ThirdParty/bsp/freertos10_xilinx_v1_3/src/Source/include/FreeRTOS.h

--- embeddedsw_def/ThirdParty/bsp/freertos10_xilinx_v1_3/src/Source/include/FreeRTOS.h  2021-02-12 15:02:10.265999000 -0500

+++ embeddedsw_mb/ThirdParty/bsp/freertos10_xilinx_v1_3/src/Source/include/FreeRTOS.h   2021-02-04 17:51:52.313377069 -0500

@@ -55,7 +55,11 @@

 /* Application specific configuration options. */

 #include "FreeRTOSConfig.h"

 

-#include "trcRecorder.h" /* Should be in the end, after the #define. */

+#ifndef __ASSEMBLER__

+   #if configUSE_TRACE_FACILITY == 1

+       #include "trcRecorder.h" /* Should be in the end, after the #define. */

+   #endif

+#endif

 

 /* Basic FreeRTOS definitions. */

 #include "projdefs.h"

diff -bur embeddedsw_def/ThirdParty/bsp/freertos10_xilinx_v1_3/src/Source/include/trcConfig.h embeddedsw_mb/ThirdParty/bsp/freertos10_xilinx_v1_3/src/Source/include/trcConfig.h

--- embeddedsw_def/ThirdParty/bsp/freertos10_xilinx_v1_3/src/Source/include/trcConfig.h 2020-11-11 10:47:58.000000000 -0500

+++ embeddedsw_mb/ThirdParty/bsp/freertos10_xilinx_v1_3/src/Source/include/trcConfig.h  2021-01-14 10:57:40.158798000 -0500

@@ -61,7 +61,7 @@

  * required at least for the ARM Cortex-M port, that uses the ARM CMSIS API.

  * Try that in case of build problems. Otherwise, remove the #error line below.

  *****************************************************************************/

-#error "Trace Recorder: Please include your processor's header file here and remove this line."

+//#error "Trace Recorder: Please include your processor's header file here and remove this line."

 

 /*******************************************************************************

  * Configuration Macro: TRC_CFG_HARDWARE_PORT

@@ -81,7 +81,7 @@

  * See trcHardwarePort.h for available ports and information on how to

  * define your own port, if not already present.

  ******************************************************************************/

-#define TRC_CFG_HARDWARE_PORT TRC_HARDWARE_PORT_NOT_SET

+#define TRC_CFG_HARDWARE_PORT TRC_HARDWARE_PORT_XILINX_MICROBLAZE

 

 /*******************************************************************************

  * Configuration Macro: TRC_CFG_RECORDER_MODE

@@ -124,7 +124,7 @@

  * TRC_FREERTOS_VERSION_10_4_0             If using FreeRTOS v10.4.0

  * TRC_FREERTOS_VERSION_10_4_1             If using FreeRTOS v10.4.1 or later

  *****************************************************************************/

-#define TRC_CFG_FREERTOS_VERSION FREERTOS_VERSION_NOT_SET

+#define TRC_CFG_FREERTOS_VERSION TRC_FREERTOS_VERSION_10_1_1

 

 /*******************************************************************************

  * TRC_CFG_SCHEDULING_ONLY

diff -bur embeddedsw_def/ThirdParty/bsp/freertos10_xilinx_v1_3/src/Source/include/trcHardwarePort.h embeddedsw_mb/ThirdParty/bsp/freertos10_xilinx_v1_3/src/Source/include/trcHardwarePort.h

--- embeddedsw_def/ThirdParty/bsp/freertos10_xilinx_v1_3/src/Source/include/trcHardwarePort.h   2020-11-11 10:47:58.000000000 -0500

+++ embeddedsw_mb/ThirdParty/bsp/freertos10_xilinx_v1_3/src/Source/include/trcHardwarePort.h    2021-02-04 18:06:22.857377069 -0500

@@ -339,7 +339,7 @@

    

    #define TRC_HWTC_TYPE TRC_OS_TIMER_DECR

    #define TRC_HWTC_COUNT XTmrCtr_GetTimerCounterReg( XPAR_TMRCTR_0_BASEADDR, 0 )

-   #define TRC_HWTC_PERIOD (XTmrCtr_mGetLoadReg( XPAR_TMRCTR_0_BASEADDR, 0) + 1)

+   #define TRC_HWTC_PERIOD (XTmrCtr_GetLoadReg( XPAR_TMRCTR_0_BASEADDR, 0) + 1)

    #define TRC_HWTC_DIVISOR 16

    #define TRC_HWTC_FREQ_HZ (TRACE_TICK_RATE_HZ * TRC_HWTC_PERIOD)

    #define TRC_IRQ_PRIORITY_ORDER 0

diff -bur embeddedsw_def/ThirdParty/bsp/freertos10_xilinx_v1_3/src/Source/include/trcKernelPort.h embeddedsw_mb/ThirdParty/bsp/freertos10_xilinx_v1_3/src/Source/include/trcKernelPort.h

--- embeddedsw_def/ThirdParty/bsp/freertos10_xilinx_v1_3/src/Source/include/trcKernelPort.h 2020-11-11 10:47:58.000000000 -0500

+++ embeddedsw_mb/ThirdParty/bsp/freertos10_xilinx_v1_3/src/Source/include/trcKernelPort.h  2021-01-14 12:22:44.322798000 -0500

@@ -249,6 +249,29 @@

 #endif /* configUSE_TIMERS == 1*/

 #endif /* configUSE_TIMERS */

 

+/* For MicroBlaze - see https://forums.freertos.org/t/microblaze-trace/6224/11 */

+#if (SELECTED_PORT == PORT_XILINX_MICROBLAZE)

+   /* Byte Order */

+   #if XPAR_MICROBLAZE_ENDIANNESS

+       /*

+        * MicroBlaze Little-Endian Working

+        *

+        * If interrupt enable, interrupt clearing the IE bit in the MSR

+        */

+       #define IN_TASK_CONTEXT ( ( mfmsr() & ( 1 << 1 ) ) != 0 )

+   #else

+       /*

+        * MicroBlaze Big-Endian Not Tested !!!

+        *

+        * If interrupt enable, interrupt clearing the IE bit in the MSR

+        */

+       #define IN_TASK_CONTEXT ( ( mfmsr() & ( 1 << 30 ) ) != 0 )

+   #endif

+   #define TRACE_ALLOC_CRITICAL_SECTION() int __irq_status;

+   #define TRACE_ENTER_CRITICAL_SECTION() { __irq_status = IN_TASK_CONTEXT; if (__irq_status) portENTER_CRITICAL(); }

+   #define TRACE_EXIT_CRITICAL_SECTION() { if (__irq_status) portEXIT_CRITICAL(); }

+#endif

+

 /* For ARM Cortex-M devices - assumes the ARM CMSIS API is available */

 #if (defined (__CORTEX_M)) 

    #define TRACE_ALLOC_CRITICAL_SECTION() uint32_t __irq_status;

  • The menu option Xilinx>Repositories can then be used to make the Xilinx IDE aware of this cloned code

  • I didn’t have any success using the Eclipse plug-in, but this page was key in working around this: Percepio Tracealyzer Documentation. There you will find a note about Xilinx SDK and using XSCT to dump the trace data. The great thing about this option is it eliminates the need for gdb, allowing use of the native System Debugger (TCF) instead.

  • I also didn’t have any success loading the trace data into Tracealyzer on Ubuntu (tried on multiple systems). But it loaded fine on Windows, which I guess isn’t so surprising since it’s a native Windows program (maybe mono works on some setups, but didn’t work on mine).

  • Note that the method I used (custom source repo) is also supported by the newer Xilinx Vitis tools, so these steps should work for the more recent Xilinx releases as well. WOuld apply to other devices as well (e.g. Zynq).