Putting tasks to waiting lists from interrupt handlers - experiments using syscalls instead of FreeRTOS task API functions in order to access queues

Kernel
1

I’m planning to make some experiments using the RISC-V port of FreeRTOS where instead of employing xQueueReceive() or xQueueSend() from a task, these functionalities are implemented as system calls, using the instruction ECALL to enter into kernel mode. From the trap handler, the respective FreeRTOS API calls are used (xQueueReceiveFromISR(), xQueueSendFromISR()) and if these calls return pdFALSE, the interrupt handler would put the calling task into the corresponding waiting lists simply using

vTaskPlaceOnEventList( &( pxQueue->xTasksWaitingToReceive ), xTicksToWait);

However, the Queue_t object is available only to the queue.c module, therefore, in order to make it work properly, I needed to slightly patch the queue.c module with some wrapper function like

void vQueueSuspendReceive(QueueHandle_t xQueue, TickType_t xTicksToWait)
{
Queue_t * const pxQueue = xQueue;
vTaskPlaceOnEventList( &( pxQueue->xTasksWaitingToReceive ), xTicksToWait);
}

to make it visible for the interrupt handler. It works well, but is there any alternative to this approach, i.e. not patching the core modules but some (more) public functions that can be used to add the task to waiting (or delayed) lists from interrupts?

thanks in advance, A.

2

I suppose my first question is WHY? It sounds like what you are doing is just trying to duplicate what the MPU ports are doing to handle elevation of privileges.

Note, a “system call” while perhaps seeming like an ISR, are not really interrupts in the sense FreeRTOS uses the term.

3

Yes, it is good question why :slight_smile: As noted above, it is a kind of experiment, but i’d say that there are some benefits (according to these recently gained experiences with these type of ports):

  • If a task is added to a waiting list since it is waiting for a queue, then the syscall handler can directly perform the yield, there is no need for further repetition for context switching (i.e. the scheduler immediately yields either to another task or puts the CPU into WFI state). Similarly, if an interrupt wakes up a task, then the interrupt directly returns to the task what just recently awaken by this interrupt, there is no need for an async yield (like PendSV). It can make the whole system more responsive with less latency.
  • The code footprint is definitely smaller since a few of queue related functions are not linked to the final binary (including xQueueSend() and xQueueReceive() themselves).
  • Such a “syscall” can more-or-leass easily be simulated/implemented on architectures even where there are no such CPU features (like AVR or MSP430, i checked these two as well).
  • Especially on RISC-V, tasks can then be run in U-mode, i.e. without doing any kind of juggling with M-mode registers, including the lack of critical sections and disabling/re-enabling interrupts. This property adds some extra security/safety and a more streamlined software design.

I’m not really familiar with MPU ports, but i suppose that it somehow an orthogonal feature (however, indeed, it can be overlapped, see e.g. U-mode in RISC-V ports). So, I’d say that besides that two wrapper functions that needed to be added to queue.c (and one extra is needed to added to tasks.c to expose prvAddCurrentTaskToDelayedList() to the port in order to implement delaying), it is not a big cost. FreeRTOS kernel itself is not modified.

thanks, A.

4

perhaps you need to look a the code base to understand what actually is happening.

Putting a task on the waiting list itself doesn’t do any part of the context switch away from the task. That all happens inside the PendSV interrupt (for ports that use that sort of method). The whole concept of a task requires that a context save happens to leave the task, and a context load to return to it.

Yes, there are ports that don’t do this via something like a PendSV but just use a call to the scheduler that saves and reloads the context, but this has a side effect of not allowing nested interrupts. (If an interrupt that nested by interrupting another ISR changed its return to a new task, the interrupted ISR won’t get run until the task that it interrupted gets rescheduled again).

You should look at the MPU ports, as a lot of what you are talking about sounds just like the moving between “restricted” (U-Mode) and “permitted” (M-Mode) operation modes.

Note, if you are putting your tasks into the restricted domain, then ANY action that need a critical section is going to need some sort of “syscall” to perform its action, which is exactly the sort of thing the MPU wrappers provide.

5

Looking at this community contributed port may also be helpful as this one has MPU support for RISC-V port.

6

Thank you, this RISC-V_RV32_MPU is indeed looks good and as an entry point, it is indeed what i’m looking for. Now i’m searching for where this array of extern uxSystemCallImplementations[] is actually implemented (for RISC-V and/or for something else). For instance, how/where the syscall number of 36 (SYSTEM_CALL_xQueueReceive) is defined. It is may or may not architecture dependent… and it seems that it is not in this port and also not in the kernel itself. But maybe i’m wrong…

7

I found that the above array shall call a function named MPU_xQueueReceiveImpl (related to xQueueReceive) but it is not defined anywhere in the full FreeRTOS source tree… or at least, i’m unable to find it…

8

Here it is - FreeRTOS-Kernel/portable/Common/mpu_wrappers_v2.c at main · FreeRTOS/FreeRTOS-Kernel · GitHub.

9

Thanks! So this syscall implementation basically calls the same xQueueReceive but from an ISR(-alike) context? What would happen if the caller task needs to go sleep? Would it loop within the SVC ISR? And what would happen in the case of RISC-V when there is no nesting?

10

You are missing that a “syscall” really isn’t an “ISR like” context. A syscall is a SYCRONOUS interrupt, and the code executed by that is strongly tied to the task context that initiated it. The execution thread inside a syscall can be switched away from just like user code. Being in a SYSCALL synchronous “interrupt” should not block real asynchronous interrupts (like the timer) from occurring.

11

Yes, if nesting is allowed and syscall (SVC) does not have that high priority then indeed it is not “really” an ISR like thing… but that’s why i was just asking or digging into this problem a bit because in RISC-V, where syscall is much more resembling to an ISR (i.e. it has exactly the same handler and we need to distinguish whether it was an ISR-interrupt or syscall or any other sync/async trap, you need to analyze context->mcause). Otherwise entering/leaving is the same and such a syscall cannot be preempted/interrupted unless we deliberately reenable mstatus | (1<<3) (MIE) or something equivalent within the ISR.

12

I will admit that I haven’t studied the RISC-V architecture in details. But if it mixes the synchronous internally generated trap of a ECALL with external interrupts (like a timer) in a way that totally blocks external interrupts for the duration of the ECALL there is a design issue.

For most processors, a Syscall “interrupt” will use a different vector than external interrupts. If a given RISC-V architecture puts them all through the same common vector, then an early step of such a vector should be to categorize the interrupt, and then enable all “higher priority” interrupts. A processor that can’t do that is not really suitable for a Real-Time system.

If the hardware can do this, but the base software package doesn’t implement this, than that software package is deficient for real-time work.

13

Below is a step-by-step description of how a system call (vTaskDelay) is executed in a FreeRTOS Cortex-M MPU port:

  1. The unprivileged task calls MPU_vTaskDelay (Note that the application code calls vTaskDelay, which the pre-processor changes to MPU_vTaskDelay).
  2. A software interrupt (SVC) is raised with the system call index corresponding to the entry in this table.
  3. The SVC Handler invokes vSystemCallEnter, which does the following:
    • Copies the stack frame from the task stack to a privileged only system call stack.
    • Configures the stack frame such that when the SVC Handler returns:
      • Execution begins at the system call indicated by the index from step 2.
      • The code uses the privileged only system call stack.
      • The code runs with elevated privileges.
      • The code returns to vRequestSystemCallExit upon system call completion.
  4. The net effect of the step 3 is that the system call executes in Thread Mode (not in ISR context), with elevated privileges, on a separate privileged-only stack.
  5. When the system call completes, vRequestSystemCallExit raises another SVC which invokes vSystemCallExit. This handler performs the reverse of step 3:
    • Copies the stack frame back to the task stack.
    • Configures the stack frame such that when the SVC Handler returns:
      • Execution resumes at the point immediately after the system call in the application code.
      • The code uses the task stack.
      • The code runs in unprivileged mode.

The system call executes in Thread Mode—not in ISR context—which means it is safe for the system call to block.

14

Hi, I would just add a bit of detail here. RISC-V allows for several implementations so depending on the hardware implementation your mileage will vary.

It is possible (but not required) that the traps for both synchronous (like ecall) and asynchronous (like a timer IRQ) events are handled by the same code or in a vectored fashion depending on the implementation and settings of the mtvec.mode CSR.

Regardless, when entering in a trap, the interrupts are disabled (e.g. mstatus.mie is cleared) until the program either exits the trap (assuming mstatus.mpie was not modified) or it explicitly re-enables the Interrupts.

15

So, if i interpret the above correctly, these steps are more-or-less equivalent to perform the syscall operation in a kind-of task which is only run if a syscall/SVC explicitly yields to it (and therefore it has more privilege than a normal task because accessing the memory regions needed by the syscall - but otherwise it is a more-or-less normal task)?

16

And, in addition to this, i’ve seen (and tried and tested) two approaches for yielding in FreeRTOS RISC-V ports: you can either do it synchronously (using ECALL, like a syscall, but such ports use ECALL only to do yielding, nothing else) or asynchronously (using CLINT, i.e. more-or-less equivalently what PendSV does). Both are okay. Unfortunately, mpu_syscall_numbers.h does not contain a yield syscall, so doing with syscalls in RISC-V implies in this regard to use async yield.

In my current tests for these ports, delaying is implemented something like below. There is only single entry point for traps/interrupts which is setup and used like this:

void attribute (( noinline, naked )) __restore_context_return(void)
{
 portRESTORE_CONTEXT();
 asm volatile ( “mret” );
}
void attribute (( noinline, naked, aligned(4) )) __vector_0 (void)
{
 portSAVE_CONTEXT();
 asm volatile ( “mv  a0, sp” :: );
 asm volatile ( “jal __vector_0_handler” :: );
 asm volatile ( “j __restore_context_return” :: );
}
BaseType_t attribute ((naked)) xPortStartScheduler( void )
{
 /* Setup the hardware to generate the tick. */
 prvSetupTimerInterrupt();

 /* setup vector table: */
 csr_write(mtvec, (uint32_t)__vector_0);

 /* enable external interrupts (MEIE), needed by PLIC. MEIE is MIE[11]: */
 asm volatile ( “li     a0, (1<<11)” :: );
 asm volatile ( “csrs   mie, a0”  :: );

 asm volatile ( “j __restore_context_return” :: );
}

There is a normal RISC-V function performing the actual duties of the handler:

void __vector_0_handler(riscv_context *context)
{
 BaseType_t     woken;

 woken = pdFALSE;

 if ( context->mcause & (1<<31) )
  {     uint32_t        interrupt;
        interrupt = context->mcause & 0xFFF;
        /* MEIP: */
        if ( interrupt == 11 )
         {      int     hwint;
                while ( 0 < (hwint = PLIC0->CONTEXT[1].CLAIM_COMPLETE) )
                 {      if ( plic_irqv[hwint].irqv_callback != NULL )
                         {      plic_irqv[hwint].irqv_callback(&woken);
                         }
                        PLIC0->CONTEXT[1].CLAIM_COMPLETE = hwint;
                 }
         }
        else
         {      /* kind of panic, unhandled interrupt */
                (void)0;
         }
  }

 /* syscalls: */
 else if ( context->mcause == 8 || context->mcause == 11 )
  {     /* yield */
        if ( context->a7 == SYSTEM_CALL_vYield )
         {      /* ECALL is always 4-bytes long, just jump it over: */
                context->mepc += 4;
                woken = pdTRUE;
         }

        /* vPortDelay(): */
        else if ( context->a7 == SYSTEM_CALL_vTaskDelay )
         {      TickType_t              delay;
                delay = context->a0;
                if ( context->vrdelay_flags == 0 )
                 {      woken = pdTRUE;
                        context->vrdelay_flags = 1;
                        vTaskDelayFromISR(delay);
                 }
                else
                 {      context->vrdelay_flags = 0;
                        context->mepc += 4;
                 }

         }
        /* another syscalls: */
        else
         {      /* ... */
         }
  }
 /* another traps: */
 else
  {     /* ... */
  }

 /* yield if needed: */
 if ( woken )
        vTaskSwitchContext();
}

And this is the patch (extension) to tasks.c what actually exposes the private function which actually adds the task to the delayed list:

void vTaskDelayFromISR( const TickType_t xTicksToDelay )
{
  if ( xTicksToDelay > ( TickType_t ) 0U )
   {    prvAddCurrentTaskToDelayedList( xTicksToDelay, pdFALSE );
  }
}
17

You are still confusing the concept of what is considered an actual ISR and what isnt (as far as FreeRTOS is concerned). The fact that the system call mechanism goes through the same code segment as “nornal” asynchronous interrupts doesn’t mean they should use the same APIs. The argument of “what if it doesn’t do the ‘right’ thing” is just admitting that the code is wrong.

The vTaskDelayFromISR you are trying to define shows the error of your ways, as an “interrupt” doesn’t care, nor should it know, what task it interrupted (if it even was a task and not a nesting of interrupts if allowed) and thus it delaying whatever task it interrupted is a meaningless act. By basic concept “FromISR” APIs don’t HAVE a concept of the “current task” as they are NOT running in the context of a task.

This is why the code inside syscall / ECALL shouldn’t be thought of as in an interrupt, as it most definitely DOES have a task context it is running in. Thus it is imperative that your master interrupt handler should quickly determine the general type of interrupt, and for these “software interrupts” working as system call, restore the system state to “task” (but privileged) context quickly, and thus that code doesn’t need to use the “FromISR” routines, as it isn’t in what is considered an ISR, but just privileged task context.

18

It is not “kind-of-task” but it is the same task and there is no explicit yield needed (unless you are calling the above mentioned SVCs as yield). When using MPU, each task contains 2 stacks - a privileged only stack which is used for system call execution and a normal stack which is used for everything else.

It is still unclear what are you trying to achieve. It would be helpful if you can describe what you want to achieve leaving aside how to achieve it.