Double rtos kernel using trustzone

amorniroli · December 6, 2022, 8:14pm

Hi,

we are evaluating some strategies about where to place rtos in a system based on trustzone (e.g. cortex m33).

rtos running in non-secure zone
rtos running in secure-zone
double rtos (1 instance running in non-secure zone and 1 instance running in secure zone)

We managed to implement all three solutions (PROS/CONS for each case, but I don’t want to focus on these) using FreeRTOS, but we stumbled upon a problem using solution 3). I’ll try to describe it.

TEST scenario:

boot in secure zone and do basic configuration (also set to ‘1’ AIRCR.PRIS, so non secure zone will run always with lower priority). Also create a NSC function void dummy (void) { vTaskDelay (1) }
configure freertos (using secure zone banked interrupts systick / pendsv / svc), create a single task (blocking on vTaskDelay (forever)) and start kernel
in secure domain freertos idle task hook tick, call non_secure_init routine (it will never return, so idle task acts as the entrypoint to non secure domain
in non secure domain, configure a separate freertos kernel (using non secure domain systick / pendsv / svc interrupts) and start it.

At this point, the situation is:

SECURE DOMAIN

1 task waiting forever / 1 task in “execution” (idle task, blocked on non_secure_init routine since it will never return)

NON SECURE DOMAIN

1 task in execution (idle task)

So, non secure domain is in execution (exception done for secure/non secure domain switch due RTOSs tick handler ISRs)

at this point, in non secure domain idle task tick hook, call NSC function, which (as said before), will call vTaskDelay (1)
secure domain switch is done due to NSC function, which will be executed in idle task context
configAssert will be triggered because of taskSELECT_HIGHEST_PRIORITY_TASK, since there are no ready task in secure domain rtos

Obviously this is an architecture lack (double rtos / blocking NSC function / etc etc), but my question is:

couldn’t FreeRTOS kernel function taskSELECT_HIGHEST_PRIORITY_TASK just wait for a task to become ready, instead of triggering configASSERT( uxTopPriority )?

I hope I was able to explain good enough the test scenario.

Best regards,

Alessandro

rtel · December 6, 2022, 8:48pm

Is option 3 feasible? You can run the kernel on the non-secure side and a monitor/microvisor application with some form of scheduling on the secure side, but I’m not sure about two independent RTOS instances because ARMv8-M is a single core device (albeit split into secure and non secure worlds) so can only run one task at a time.

This is the kind of problem I would envision because what would the RTOS on the nonsecure side execute while it was waiting?

amorniroli · December 6, 2022, 9:16pm

Yes, option is feasible (at least, I managed to compile two separate freertos instances with different configtick rate hz) and run them without problems (except for the scenario I described before, where NSC call blocks on task delay / semaphore / etc and no other task in secure domain are ready to run).

I would say that secure domain would run a RTOS, and non secure domain would run a OS (no real time guaranteed since higher priority of non secure domain task could be blocked by secure domain tasks).

Sure, the core is just one, so only one task at a time will run. AIRCR.PRIS could guarantee that non secure domain will run with lower priority than secure domain (I could reserve higher half priorities to secure domain and lower half priorities to non secure domain). So for example, if non secure domain ISR disable global irq and loop on while (1) ; secure domain RTOS will continue to run without interferences.

The idea behind two separate kernel is to increase separation between something I want to be secure and safe (secure domain) at 99.99999% and something I don’t really care (non secure domain, e.g. third party application code). Probably, the same could be done using single kernel and MPU. As said, every solution we tried has PROS/CONS, depending on the final application.

Well, using two separate kernel, a call to NSC function from the nonsecure domain is seen as a blocking function (but the kind of while (1) blocking function). So, it depends. For example, if a context switch happens on nonsecure domain (e.g. due to higher priority task), than that task would be executed.

Tomorrow I’ll try to share a STM32 cube ide example project based on stm32u5 nucleo running this scenario.

rtel · December 7, 2022, 2:58am

Curious how you leave the secure side when the RTOS there always has the idle task to execute.

amorniroli · December 7, 2022, 7:48am

S domain idle task is always in ready state, but since NS_init routine (which is “blocking”) was called inside it, a task context switch to idle task will also trigger a domain switch from S to NS, since from RTOS perspective, idle task is still executin NS_init routine.

The idle task is “sacrificed”: when S domain RTOS has nothing to do (idle), then switch to NS kernel is done.

aggarg · December 7, 2022, 10:49am

This probably means that you are running non-secure RTOS with TZ support enabled so that each task has a secure stack associated, right?

Even the idle task is not ready? Is it because you called ns_init from it? If yes, can you not do that from a dedicated task at priority 0 and not from idle task?

Looking forward to that as I am not sure I completely understand your setup.

Thanks.

amorniroli · December 7, 2022, 11:55am

Hi,

I’ve uploaded an example on github, based on NUCLEO-U575 board (compile and run w/ STM32CubeIde, but it should be easily portable to other boards).

(project is based on STM32 GPIO_IOToggle_TrustZone example)

This is the flow of project loaded on github:

configure S domain
configure led1/led2 of the board as GPIO (both of them are S)
start freertos kernel
in vApplicationDaemonTaskStartupHook, create S thread that will toggle led1 at 1Hz (please notice that tick rate of S domain FreeRTOS is 1000)
as S domain idle task runs, call NS entry point
NS entry point just starts its FreeRTOS kernel
in vApplicationDaemonTaskStartupHook of NS, create NS thread that will toggle led2 at 10Hz (please notice that tick rate of NS domain FreeRTOS is 100). led2 toggle is performed by NSC function SECURE_ToggleLed2

Everything seems to work.

In example loaded on github, it works without allocating secure stack (NS domain is not aware of what is running in S domain. It could be baremetal or another RTOS). NSC functions will be executed from idle task stack.

See next two images:

Stack pointer of S idle task when jumping to NS

Stack pointer of when NSC function call is executed (S idle task stack)

I’ll try to investigate on this.

amorniroli · December 7, 2022, 1:37pm

Nice hint! The trick works!

EDIT: nevermind, it’s not working. I mean, I’ve just noticed a thing: I thought that NSC calls were always executed from task context which called NS_init routine, but it’s not true.

Consider this example:

NSC routine

CMSE_NS_ENTRY void
SECURE_ToggleLed2 (const char* ptrTask)
{
	volatile const char* ptr = ptrTask;

	(void) ptr;

	vTaskDelay (2000);

    HAL_GPIO_TogglePin(GPIOB, GPIO_PIN_7);
}

SECURE DOMAIN

static void
sThread (void* ptrArg)
{
    (void) ptrArg;

    while (1)
    {
    	vTaskDelay (1000);

        HAL_GPIO_TogglePin (GPIOC, GPIO_PIN_7);
    }
}

static void
sIdle (void* ptrArg)
{
	NonSecure_Init ();
}

void
vApplicationDaemonTaskStartupHook (void)
{
	xTaskCreate ((TaskFunction_t)sThread, "secureThread",  1024, NULL, 16, NULL);
	xTaskCreate ((TaskFunction_t)sIdle,   "secureIdle",    1024, NULL, 0, NULL);
}

NON SECURE DOMAIN

static void
nsThread (void* ptrArg)
{
	const char* ptrName;

	ptrName = ptrArg;

	if (strcmp (ptrName, "NS1") == 0)
	{
		vTaskDelay (100);
	}

	(void) ptrName;

    while (1)
    {
    	vTaskDelay (10);

    	SECURE_ToggleLed2 (ptrName);
    }
}


void
vApplicationDaemonTaskStartupHook (void)
{
	vTaskSuspendAll ();

	xTaskCreate ((TaskFunction_t)nsThread, "nonSecureThread0", 1024, "NS0", 20, NULL);
	xTaskCreate ((TaskFunction_t)nsThread, "nonSecureThread1", 1024, "NS1", 30, NULL);

	xTaskResumeAll ();
}

So:

NS1 thread start and wait for 1s prior to call SECURE_ToggleLed2. Context switch to NS2
NS2 thread start and call SECURE_ToggleLed2
domain switch
SECURE_ToggleLed2 executed from sIdle context. vTaskDelay will trigger switch context to idle task

vTaskDelay (100) of NS1 expired, contex switch in NS kernel, and call to SECURE_ToggleLed2
domain switch
SECURE_ToggleLed2 executed from FreeRTOS idle task

vTaskDelay triggers configAssert

amorniroli · December 21, 2022, 11:57am

Hi,

I managed to come at the following solution which seems to work as my original intention (don’t focus on PRO/CONS for now) which would be:

running double RTOS kernel (1 instance in Secure domain / 1 instance in Non Secure domain);

From this point:

S->Secure domain
NS->Non Secure Domain
NSC->Non Secure Callable function

The following requirements shall be satisfied:

(1) NS RTOS shall be execute when S RTOS is in idle task;
(2) NS IRQs shall not preempt S IRQs and tasks (e.g. NS systick shall not trigger if S task is running, since it may schedule NS task to execute preempting S tasks);
(3) NSC shall always be called from same ‘context’ (task stack) used entering NS;

My solution:

(A) create ad hoc task used to enter NS. This task shall have same priority of idle task (0). Also configUSE_TIME_SLICING and configIDLE_SHOULD_YIELD shall be enabled.

	static void
	nonSecureTask(void* ptrArg)
	{
		NonSecure_Init ();
	}

(B) set ARM AIRCR->PRIS bit to ‘1’, so NS IRQs are de-prioritized.

(C) modify S (only) kernel functions/macro vPortEnterCritical, vPortExitCritical, portDISABLE_INTERRUPTS, portENABLE_INTERRUPTS, PendSV_Handler as follows:

file port.c

static uint32_t last;

void vPortEnterCritical( void ) /* PRIVILEGED_FUNCTION */
{
	uint32_t temp;

	temp = ulSetInterruptMask();

    if (ulCriticalNesting++ == 0)
    {
    	last = temp;
    }

    /* Barriers are normally not required but do ensure the code is
     * completely within the specified behaviour for the architecture. */
    __asm volatile ( "dsb" ::: "memory" );
    __asm volatile ( "isb" );
}
/*-----------------------------------------------------------*/

void vPortExitCritical( void ) /* PRIVILEGED_FUNCTION */
{
    configASSERT( ulCriticalNesting );

    if( --ulCriticalNesting == 0 )
    {
    	vClearInterruptMask (last);
    }
}

file portmacro.h

extern void vPortEnterCritical (void);
extern void vPortExitCritical  (void);

#define portDISABLE_INTERRUPTS()            vPortEnterCritical()
#define portENABLE_INTERRUPTS()             vPortExitCritical()

file portasm.c

/* Weak, this is implemented in main.c */
__attribute__( ( weak ) ) void
vRestoreBasePriAfterTaskSwitchContext (void)
{
	__asm volatile
	(
        "    mov r0, #0\n"		/* r0 = 0. */
	    "    msr basepri, r0 \n"/* Enable interrupts. */
	);
}

void PendSV_Handler( void ) /* __attribute__ (( naked )) PRIVILEGED_FUNCTION */
//other original code, changes are below

        "   mov r0, %0                                      \n"/* r0 = configMAX_SYSCALL_INTERRUPT_PRIORITY */
        "   msr basepri, r0                                 \n"/* Disable interrupts upto configMAX_SYSCALL_INTERRUPT_PRIORITY. */
        "   dsb                                             \n"
        "   isb                                             \n"
        "   bl vTaskSwitchContext                           \n"
    	"	bl vRestoreBasePriAfterTaskSwitchContext		\n"
//        "   mov r0, #0                                      \n"/* r0 = 0. */
//        "   msr basepri, r0                                 \n"/* Enable interrupts. */

file main.c

void
vRestoreBasePriAfterTaskSwitchContext (void)
{
    TaskHandle_t handle;

    /* Get current handle in execution. */
    handle = xTaskGetCurrentTaskHandle ();

    /* Jumping to non secure domain? */
    if (nsDomainTaskHandle == handle)
    {
    	/* Clear basepri (allow NS IRQs). */
    	__set_BASEPRI (0);
    }
    else
    {
    	/* Inhibit NS IRQs. */
    	__set_BASEPRI (0x80);
    }
}

vRestoreBasePriAfterTaskSwitchContext is responsible to inhibit NS IRQs when jumping to S tasks. If jumping to NS domain, then basepri is cleared (to let NS domain to run).

vPortEnterCritical, vPortExitCritical, portDISABLE_INTERRUPTS, portENABLE_INTERRUPTS changes are necessary since otherwise any portENABLE_INTERRUPTS would set basepri to ‘0’, letting NS domain to (potentially) preempt a S running task.

(1) is satisfied by (A)
(2) is satisfied by (B), (C)
(3) is satisfied by (C)

In case of “blocking” NSC, this solution is safe since S kernel idle task would run, and NS IRQs/scheduler are disabled by vRestoreBasePriAfterTaskSwitchContext (NS kernel is stuck on task which called NSC function).

What do you think?

EDIT: updated code on github GitHub - amorniroli/TrustZone_FreeRTOS_experiment: FreeRTOS experiments using ARMv8-M architecture (Nucleo-U575ZI-Q board)

(project can be imported in stm32cube ide. .project file is inside double_kernel/GPIO_IOToggle_TrustZone/STM32CubeIDE folder).

amorniroli · December 21, 2022, 1:46pm

Tested a bit and it seems to work properly now.

I’ve tried these situations:

blocking NS IRQ w/ while (1)

    	__set_BASEPRI (0x1);

    	while (1) {}

S domain is not influenced and kernel keeps working correctly. NS domain is stuck in IRQ

add blocking function (e.g. vTaskDelay) to NSC routine

CMSE_NS_ENTRY void
SECURE_ToggleLed2 (void)
{
		vTaskDelay (1000);

	HAL_GPIO_TogglePin(LED2_GPIO_Port, LED2_Pin);
}

CMSE_NS_ENTRY void
SECURE_ToggleLed3 (void)
{
	vTaskDelay (1000);

	HAL_GPIO_TogglePin(LED3_GPIO_Port, LED3_Pin);
}

S domain is not influenced and kernel keeps working correctly. Since NSC are executed from S task used to enter NS domain, NS kernel (and also IRQs) are stuck till the end of vTaskDelay (due to S basepri set to 0x80).

blocking NS task w/ while (1), for example

CMSE_NS_ENTRY void
SECURE_ToggleLed2 (void)
{
    while (1)
}

CMSE_NS_ENTRY void
SECURE_ToggleLed3 (void)
{
	HAL_GPIO_TogglePin(LED3_GPIO_Port, LED3_Pin);
}

S domain is not influenced and kernel keeps working correctly. NS kernel behavior depends on task NS priorities (in the example, SECURE_ToggleLed3 is called from higher priority task, so it will be scheduled normally).

What would be PRO/CONS of this solution? I’d say:

PRO

NS IRQs cannot interfere with both S IRQs and tasks (e.g. in other TZM solutions with single kernel, NS IRQ could stuck in while (1) preventing any task to RUN). I think this is the most important PRO
NS application may be implemented using a different OS then the one running in S domain (NS could be also implemented as bare metal application)
full separation between S and NS applications

CONS

double systick S/NS IRQs → double overhead
NS domain real time is not guaranteed, since NS IRQs are masked when S domain is in execution;
blocking NSC function would not schedule any other ready NS task, but would schedule S idle task;

aggarg · December 30, 2022, 11:09am

Thank you for sharing your code. I looked at it and I am not sure that it will work in all the scenarios.

The following are the possible states and modes -

Secure State
- Thread Mode
- Handler Mode
Non-Secure State
- Thread Mode
- Handler Mode

You are using the CM33_NTZ (modified) port on both sides. The context save code assumes that the hardware stores partial context on the interrupted task’s stack and stores the additional context on the same stack.

This assumption does not hold true always. Lets try to list the possible scenarios.

The following transactions take the control to the Secure PendSV Handler –

Secure PendSV when Secure Task was running. HW stores the partial context on the Secure Task’s stack (PSP_S). SW stores the additional context on the Secure Task’s stack correctly.
Secure PendSV when Non-Secure Task was running. HW stores the partial context on the Non-Secure Task’s stack (PSP_NS). SW stores the additional context on the Secure Task’s stack (PSP_S) incorrectly.
Secure PendSV when any Non-Secure interrupt was running. HW stores the partial context on the Non-Secure Main stack (MSP_NS). SW stores the additional context on the secure task’s stack (PSP_S) incorrectly.

The following transactions take the control to the Non-Secure PendSV Handler –

Non-Secure PendSV when Non-Secure Task was running. HW stores the partial context on the Non-Secure Task’s stack (PSP_NS). SW stores the additional context on the Non-Secure Task’s stack correctly.
Non-Secure PendSV when NSC function was called and as a result Secure Task nsEntryThread was running. HW stores the partial context AND additional context on nsEntryThread task’s stack. SW stores the additional context on the Non-Secure Task’s stack (PSP_NS) unnecessarily and incorrectly.

The same problems would happen during the context restore as well and you’d end up restoring wrong context and potentially returning to the wrong state.

amorniroli · January 4, 2023, 4:09pm

Hi @aggarg,

first of all thank you for taking time to this and for the accurate explanation.

Please don’t hate me ( ) but I still have some doubts about the steps you described as “incorrectly”.

In my scenario, using the modified CM33_NTZ port, the entry point to NS domain is done by a dedicated secure task (nonSecureTask in my previous post): I would say it’s correct that additional context stores by Secure PendSV Handler are executed on PSP_S of this (and ONLY THIS) secure task. Whenever nonSecureTask is scheduled again, context is restored and jump to NS is performed.

For example (NS task is running and S PendSV happens):

HW stores partial context on PSP_NS
SW stores additional context on PSP_S of nonSecureTask and context is switched
other S task is executed (please be aware that NS domain is completely inhibited if nonSecureTask is not in RUNNING state due to __set_BASEPRI (0x80))
nonSecureTask scheduled again: S PendSV handler restores context from step (2). Last operation is bx EXC_RETURN (0xffffffbd → NS stack used)
back to NS domain

I would say that’s correct (e.g. a low priority NS task that called a NSC function could be preempted by a higher priority NS task).

For example, from my previous post:

amorniroli:

blocking NS task w/ while (1), for example
CMSE_NS_ENTRY void
SECURE_ToggleLed2 (void)
{
    while (1)
    {
        asm ("nop");
    }
}

CMSE_NS_ENTRY void
SECURE_ToggleLed3 (void)
{
	HAL_GPIO_TogglePin(LED3_GPIO_Port, LED3_Pin);
}
S domain is not influenced and kernel keeps working correctly. NS kernel behavior depends on task NS priorities (in the example, SECURE_ToggleLed3 is called from higher priority task, so it will be scheduled normally).

I can’t see a case where I would end up restoring wrong context and potentially return to the wrong state, but I’ll investigate and test deeper.

Thank you again.

Alessandro