How do you debug system

Kernel
1

Hi All,

I have a question for people who have spent a lot of time debugging FreeRTOS systems in the field.

When a task is not behaving as expected, what is your typical debugging approach? Do you first look at task states (Ready/Running/Blocked/Suspended), stack usage, priorities, or something else?

Also, what tools do you normally use? Just the debugger and FreeRTOS APIs, or any other tools?

Another area is heaping corruption. In embedded systems these issues can be very painful because the crash often happens far away from the actual bug. How do you usually track down heap corruption and identify the offending task or code path?

Finally, how do you measure CPU utilization in a real project? Do you use runtime stats, idle task monitoring, or external tracing tools? And what’s your preferred way of finding which task is consuming the most CPU time?

I’d appreciate hearing about real debugging experiences and lessons learned rather than textbook explanations.

2

Debugger and logs are normally sufficient. For timing related issues, Tracelyzer is also a very useful tool.

You are right that memory corruptions are hard to track down. I have found data breakpoints to be useful in debugging them.

3

A real debugger is vital to find many issues, you do need to be careful where you put break points as you can change which task you are in.

We also tend to have a task connected to a debug console with which we can use commands to interogate various system state and pull up the task status report.

It is also not uncommon to have a GPIO (or several) that tasks can toggle to report conditions that we can monitor on a scope.

For usage we normally use the run time stats, and I wrote an extension that clears the current accumulation so we can see the usage in given conditions.

Memory corruption (of which heap corruption is really just a specia case) is hard to find. Being careful in the first place is the best answer so you don’t get into that situation. And having someone else review the code is a good solution if you do run into a problem.

One other thing to do is have your system periodically check for system consistancy, as is being “defensive” in your programming and reporting errors when something isn’t right.

4

Some type of timestamped persistent logging is extremely helpful. You can log out of memory, asserts, and other important states, which allows one to piece quite a bit together afterwards. IMO debugging production builds using debug mechanisms is going to have limited use.

You can also kick a watchdog when all important threads are validated as not blocked, and/or blink an led

5

I assume you wrote it as a new API which call to clear accumulation. Do you think it is something that can be included in the kernel code itself?

6

It was written using freertos_tasks_c_additions.h method, and adds a function you can call to reset all the performance counters, so we use it as part of the “kernel code itself”. I have slightly modified it from what we use, as it add some additional proprietary operations. It will need some work for SMP

/*
* Internal Function to clear the accumulation for all tasks on a specific list
*/
void vTaskClearUsageSingleList(List_t *pxList)
{
	configLIST_VOLATILE TCB_t *pxNextTCB, *pxFirstTCB;

	if( listCURRENT_LIST_LENGTH( pxList ) > ( UBaseType_t ) 0 ) {
		listGET_OWNER_OF_NEXT_ENTRY( pxFirstTCB, pxList );

		/* Clear Usage for each task that is referenced from
		pxList.  See the definition of TaskStatus_t in task.h for the
		meaning of each TaskStatus_t structure member. */
		do {
			listGET_OWNER_OF_NEXT_ENTRY( pxNextTCB, pxList );
			pxNextTCB->ulRunTimeCounter = 0;
		} while( pxNextTCB != pxFirstTCB );
	}
}

#if tskKERNEL_VERSION_MAJOR < 11
# define SWITCH_TIME ulTaskSwitchedInTime
# define RUN_TIME    ulTotalRunTime
#else
# if ( configNUMBER_OF_CORES == 1 )
#  define SWITCH_TIME ulTaskSwitchedInTime[0]
#  define RUN_TIME    ulTotalRunTime[0]
# else
# endif
#endif
/*
* Clear performance statistics for all tasks
* 
* Warning, uses a long critical section so care needs to be taken 
* in actual real-time production system
*/
void vTaskClearUsage() {
	// Need full disable since we added ISR Tracking.
    taskENTER_CRITICAL();
	UBaseType_t uxQueue = configMAX_PRIORITIES;
	/* Clear runtime counter for each task in the Ready State */
	do {
		uxQueue--;
		vTaskClearUsageSingleList(&( pxReadyTasksLists[ uxQueue ] ));
	} while( uxQueue > ( UBaseType_t ) tskIDLE_PRIORITY ); /*lint !e961 MISRA exception as the casts are only redundant for some ports. */

	/* Clear runtime counter for each task in the Blocked state. */
	vTaskClearUsageSingleList(( List_t * ) pxDelayedTaskList);
	vTaskClearUsageSingleList(( List_t * ) pxOverflowDelayedTaskList);

#if( INCLUDE_vTaskDelete == 1 )
	/* Clear runtime counter for each task that has been deleted but not yet cleaned up. */
	vTaskClearUsageSingleList(&xTasksWaitingTermination);
#endif

#if ( INCLUDE_vTaskSuspend == 1 )
	/* Clear runtime counter for each task in the Suspended state. */
	vTaskClearUsageSingleList(&xSuspendedTaskList);
#endif
	SWITCH_TIME = 0;
	RUN_TIME = 0;
	vConfigureTimerForRunTimeStats();
    taskEXIT_CRITICAL();
}
7

Thank you for sharing!