SMP Porting Checklist - A53 *4 as reference

Hi all,

Please see below a rough checklist for porting to SMP freertos. Let me know if i am missing something or something is wrong.

Regards
Srinivas

Porting Checklist for FreeRTOS SMP

  1. Change: Definition in freertosconfig
    configNUM_CORES: The number of cores available to run tasks. May be set to anything greater than 0, provided the hardware and port supports it.
    configRUN_MULTIPLE_PRIORITIES: Set to 1 to allow tasks with different priority levels to run simultaneously on different cores. Set to 0 to only allow tasks with the same priority level to run simultaneously.
    configUSE_CORE_AFFINITY: Gets the core affinity mask for a task, i.e., the cores on which a task can run.
    configTICK_CORE: Core in which tick should run
    configUSE_TASK_PREEMPTION_DISABLE: In SMP FreeRTOS application, if configUSE_TASK_PREEMPTION_DISABLE is defined as 1, then individual tasks can be set to either pre-emptive or co-operative mode using the vTaskPreemptionDisable and vTaskPreemptionEnable API functions.
    portCRITICAL_NESTING_IN_TCB: Storing critical nesting count per task in TCB.

Example for Cortex A53:
configNUM_CORES – 4
configRUN_MULTIPLE_PRIORITIES – 1
configUSE_CORE_AFFINITY – 1
configTICK_CORE - 0
configUSE_TASK_PREEMPTION_DISABLE – 0
portCRITICAL_NESTING_IN_TCB - 1

  1. Change: xPortStartScheduler should start scheduler on the number of cores specified by configNUM_CORES specified in FreeRTOSConfig.h file. Each core must end this function by restoring the context of the task assigned to it.
    Example for Cortex A53:
    Primary core (core 0) executes xPortStartScheduler, wakes up other 3 cores which are in sleep state, and all cores execute vPortRestoreContext.

  2. Change: pxCurrentTCB has been renamed to pxCurrentTCBs and made into an array indexed by core. Whenever task state needs to be saved or restored, this array must now be indexed first to get the stack pointer.
    Example for Cortex A53:
    In portSAVE_CONTEXT and portRESTORE_CONTEXT , get the portGET_CORE_ID of the current core executing. Based on the current core, move to that index of pxCurrentTCBs.

  3. Change: portDisable_interrupts to return interrupt mask prior to disabling.
    Example for Cortex A53:
    Use the ICC_PMR_EL1 to save priority mask.

  4. Change: If portCRITICAL_NESTING_IN_TCB is set to 1.
    portENTER_CRITICAL() → vTaskEnterCritical()
    portEXIT_CRITICAL() → vTaskExitCritical()
    portSET_INTERRUPT_MASK_FROM_ISR → This must call vTaskEnterCritical() in addition to returning the interrupt mask prior to calling it
    portCLEAR_INTERRUPT_MASK_FROM_ISR → This must first call vTaskExitCritical() before setting the interrupt mask to the value in x.
    Reference : FreeRTOS-Kernel/portmacro.h at 4832377117b4198db43009f2b548497d9cdbf8da · FreeRTOS/FreeRTOS-Kernel · GitHub

  5. Change: portRESTORE_INTERRUPTS This must set the interrupt mask to the value in x. It is used after calling portDISABLE_INTERRUPTS() to restore the interrupt mask to what it was prior to calling portDISABLE_INTERRUPTS(). The value x should be what was returned by portDISABLE_INTERRUPTS().
    Example for Cortex A53:
    Use the ICC_PMR_EL1 to restore priority mask saved when portDISABLE_INTERRUPTS.

  6. Change: Tick interrupt should be enabled only on one core (preferably core 0).
    Example for Cortex A53:
    In function xPortStartScheduler get the portGET_CORE_ID, if it is primary core call configSETUP_TICK_INTERRUPT.

  7. Change: Two locks must be implemented (portGET_TASK_LOCK portRELEASE_TASK_LOCK) and (portGET_ISR_LOCK, portRELEASE_ISR_LOCK). This should be recursive spinlock implementation.
    Example for Cortex A53:
    Use FreeRTOS-Kernel/portmacro.h at 4832377117b4198db43009f2b548497d9cdbf8da · FreeRTOS/FreeRTOS-Kernel · GitHub for reference.

  8. Change: portGET_CORE_ID(). This must return the core ID (between 0 and configNUM_CORES-1) of the calling core.
    Example for Cortex A53:
    Use the MPIDR_EL1 register to get the core ID.

  9. Change: portYIELD_CORE(coreID) , This must interrupt the core with ID x. The core that is interrupted must behave as if it called portYIELD() (i.e. save the current task context, call vTaskSwitchContext(), and then restore the context of the new current task).
    Example for Cortex A53:
    Send an interrupt from the running core to “coreID” core using SPI/PPI .
    In the interrupt handler call portYield_FROM_ISR(pdTrue) ,which will make the core to do context switch

  10. Change: portCHECK_IF_IN_ISR This must return pdTRUE if it is called from within an ISR, otherwise pdFALSE if it is not.
    Example for Cortex A53:
    Use ISR_EL1 register to check if it’s in ISR.

  11. Change: Pass coreid to vTaskSwitchContext as argument.
    Use portGET_CORE_ID to get the coreID.

  12. Change:
    Integration of ARM GIC – changing interrupt address
    Reference:
    RTOS for ARM Cortex-A

  13. Change: Call xTaskIncrementTick in function FreeRTOS_Tick_Handler within portSET_INTERRUPT_MASK_FROM_ISR and portCLEAR_INTERRUPT_MASK_FROM_ISR.

4 Likes

This looks very good. The only issue that I noticed in my first pass was in the example for step 6. It appears you copied the example from step 4.

Let us see if there are any other comments and then put it in the SMP docs folder.

Looks like you already ported FreeRTOS with SMP Support to the ARM A53. I’m currently working on a port for the Zynq Ultrascale+ MPSOC platform which uses the same processing core. So I’m very interested in your work. Did you contribute your port back to the FreeRTOS Repositories? If so where can I find it?

The A53 SMP port done for TI is located here:

Good Luck

Sadly this port seems outdated since the implemented functions do not match the required ones for the latest FreeRTOS release. This also applies for the Porting Checklist. Is anyone able to update the checklist?

These functions need to be implemented additionally when using SMP:

portGET_CORE_ID
portYIELD_CORE
portSET_INTERRUPT_MASK
portCLEAR_INTERRUPT_MASK
portRELEASE_TASK_LOCK
portGET_TASK_LOCK
portRELEASE_ISR_LOCK
portGET_ISR_LOCK
portENTER_CRITICAL_FROM_ISR
portEXIT_CRITICAL_FROM_ISR

Is there documentation on what these functions need to do?

Hi jen,

Thank you for listing the port macros changed in main branch.

The checklist above is for porting to FreeRTOS SMP branch. FreeRTOS SMP branch is deprecated now. I update the checklist with RP2040 port example.

1. Define SMP configuration in FreeRTOSConfig.h
Based on the sample confiugration, the following are required configuration to be added to enable SMP scheduler.

#define configNUMBER_OF_CORES     /* The number of cores in the platform. Set core numbers to greater than 1 to enable SMP scheduler */
#define configUSE_PASSIVE_IDLE_HOOK   0 /* 

You may reference SMP Specific Configuration Options for more information about other SMP configurations.

2. Define portGET_CORE_ID()
Returns the core ID ( between 0 and configNUMBER_OF_CORES - 1 ) of the calling core.
RP2040 port makes use of the get_num_core() API from PICO SDK to implement this function.

3. Define portYIELD_CORE( xCoreID )
The core calling this API must interrupt the other core with ID xCoreID. The interrupted core must behave as if it call portYIELD().
RP2040 makes use of a FIFO to interrupt other cores.

4. Define portSET_INTERRUPT_MASK() and portCLEAR_INTERRUPT_MASK( ulState )
Interrupt mask should be returned and cleared with these macros.
Example RP2040 port implementation

5. Define portGET_TASK_LOCK() and portGET_ISR_LOCK()
Acquire the task or ISR lock. TASK and ISR lock are recursive lock. They should be able to be locked by the same core multiple times.

6. Define portRELEASE_TASK_LOCK() and portRELEASE_ISR_LOCK()
Release the task or ISR lock. If these locks are locked by the same core multiple times, they should be released as many times as it is locked.
Example RP2040 recursive lock implementation.

7. Define portENTER_CRITICAL() and portEXIT_CRITICAL()
These macros should be implemented with vTaskEnterCritical() and vTaskExitCritical().
Example RP2040 implementation.

8. Define portENTER_CRITICAL_FROM_ISR() and portEXIT_CRITICAL_FROM_ISR()
Enter/exit critical section from ISR. These macros should be implemented with vTaskEnterCriticalFromISR() and vTaskExitCriticalFromISR( x ).
Example RP2040 implementation.

9. Define portCRITICAL_NESTING_IN_TCB
Port can decide to keep the critical nesting count in TCB or in port by setting portCRITICAL_NESTING_IN_TCB in portmacro.h file. RP2040 port provides an example of keeping critical nesting count in the port.

10. Update portSET_INTERRUPT_MASK_FROM_ISR and portCLEAR_INTERRUPT_MASK_FROM_ISR implementation in port
These macros should now be implemented as the macro name suggested, set or clear interrupt mask from ISR if nested interrupt are supported.
Example RP2040 implementation

11. Call xTaskIncrementTick in critical section in port
Access shared kernel data in interrupt handler now need to be performed in critical section due to multiple cores consideration.
Example RP2040 implementation.

12. Update xPortStartScheduler
The port should start scheduler on the number of cores specified by configNUMBER_OF_CORES. Each core must end this function by restoring the context of the task assigned to it.

13. Use pxCurrentTCBs in the port
Pointers to Current TCB now are an array indexed by core ID. Whenever task state needs to be saved or restored, this array must now be indexed first to get the stack pointer in TCB_t.
Example Index the pxCurrentTCBs in RP2040 port

14. Pass core ID to vTaskSwitchContext as parameter
vTaskSwitchContext now takes core ID as parameter to switch context of the core ID in pxCurrentTCBs.

This checklist may be rough and not completed. Feedback and update to the checklist are welcomed.
The template port also include SMP support. It can be used as a starting point for developing a new FreeRTOS SMP port.

3 Likes

I have some quesions about this check list:

  1. About point 4 and 9
    The implement of portSET_INTERRUPT_MASK() and portSET_INTERRUPT_MASK_FROM_ISR() seems to be the same. They also read primask and disable irq, what is the difference between them? In last smp version v202110, portSET_INTERRUPT_MASK_FROM_ISR() should call vTaskEnterCritical(), isn’t it necessary in this version?

  2. About point 14
    vTaskSwitchContext now takes core ID as parameter to switch context of the core ID in pxCurrentTCBs.
    vTaskSwitchContext( xCoreID ) will change pxCurrentTCBs[xCoreID]. Does it mean a core can change the running task on another core? In my understanding, vTaskSwitchContext() is called when the core needs to do context switch after a interrupt and only switch the context in inself.

Hi Saiiijchan,

  1. portSET/CLEAR_INTERRUPT_MASK is called in a FreeRTOS task and portSET/CLEAR_INTERRUPT_MASK_FROM_ISR is called in an ISR. They may use the same implementation to stop interrupts which may access FreeRTOS-Kernel shared data in the ISR. If the port doesn’t support nested interrupt, portSET/CLEAR_INTERRUPT_MASK_FROM_ISR can be ignored in the port.
    The portSET_INTERRUPT_MASK_FROM_ISR() should be implemented as the name suggested, vTaskEnterCritical() is not required. Instead, vTaskEnterCriticalFromISR() should be called in portENTER_CRITICAL_FROM_ISR().

  2. No, a core won’t change the running task on another core. We will look further if this can be removed.

1 Like

I’m done porting to the Zynq Ultrascale+ MPSOC Platform. I need to do a bit more testing to ensure stability but it look promising. When I’m done, I’ll contribute the port back to the community.

7 Likes

Hi @jen,

Can I kindly ask if you are still intending to contribute your port? As I need to do the same port and it will save me a lot of time!

Thank you.

Kind Regards,

Pete

Hi @pete-pjb,

yes I will, though I need a bit more time to perfect what I’ve written (and verify if it really is working). If you need it quick, I can send it to you directly without any further modifications.

Hi @jen,

Thank you for your prompt reply!

If it’s not an imposition would you mind sending it? I would be most grateful and of course if I find any issues I would be happy to work on and share any fixes back to you. I assume there is just 3 or 4 files, would you be able to send a zip to pete@one.co.uk please? :smiling_face:
I am currently attempting to get a system up and running with SMP FreeRTOS on an Enclustra board (to date I have just used one core) and I am working on +TCP and +FAT (SD & onboard eMMC) integration which seems to be working okay, I am also using the Displayport Interface, along with attempting to get a host USB XHCI port running. So if any of my efforts might help you also please feel free to email me if you could benefit from any of those parts.

Thanks again.

Kind Regards,

Pete

1 Like

Hi @pete-pjb,

I’ve sent you an email.

1 Like

Hi @jen

I also need to use zynqMP SOC to run freertos SMP recently, I saw this post of yours and think you did a very good job, Would you send me a copy of your port layer as well, thanks!
e-mail: yafeng.yun@mindcruise.ai

Kind Regards
Yafeng

@jen
With a few requests for the port work, would you consider a PR so we can more broadly support the community?

Of course I will contribute my port. Currently however I have an issue with the recursive spinlock implementation. After my own implementation was not working I tried the existing implementation of a Partner Supported Port for a TI Cortex A53 CPU (found here: Source/portable/ThirdParty/Partner-Supported-Ports/TI/CORTEX_A53_64-BIT_TI_AM64_SMP/portASM.S.) However this also results in the TASK_LOCK being locked indefinitely (Partner Supported and mine).
The Code I’m running is the Full_Demo of the Sync Ultrascale+ Demo (here: Demo/CORTEX_A53_64-bit_UltraScale_MPSoC). I narrowed to problem down to the tick Hook (vFullDemoTickHook). Whenever I comment out the function calls within, the System runs fine. Any idea why I experience such behavior?

Also I encountered this: [BUG] vTaskDelay raises configASSERT when SMP is on · Issue #1068 · FreeRTOS/FreeRTOS-Kernel · GitHub

Can you try pining all the tasks to one core? You can probably use configTASK_DEFAULT_CORE_AFFINITY for this.

I’ve finished porting FreeRTOS SMP to ZynqMPSOC, and I’m currently testing it, and I haven’t found any problems at the moment

1 Like

Whenever you feel you are ready, feel free to raise a PR to the following repo - GitHub - FreeRTOS/FreeRTOS-Kernel-Community-Supported-Ports.