Running FreeRTOS (RISC-V) on gem5

Hi everyone,

I am working on a project to run FreeRTOS on gem5, specifically on a RISC-V processor.
To get started, I used the QEMU project "RISC-V_RV32_QEMU_VIRT_GCC", but I am open to switching to a different project if there is a better alternative.

What I Have Done So Far:

  • have configured a gem5 Full System (FS) simulation with a RISC-V architecture.
  • After encountering some issues (described here, thank you again for the support :folded_hands:), I compiled FreeRTOS for a 64-bit architecture.
  • I managed to boot the system on gem5, but execution seems stuck at the very beginning.

Current Issue:

  • Debugging with GDB, I found that execution hangs when calling function: xQueueCreate.
  • Specifically, it seems stuck on the set of instructions with label freertos_risc_v_trap_handler, at the instruction: portcontextSAVE_CONTEXT_INTERNAL.

However, I am not so expert in debugging assembly, so I could be wrong on the specific instruction.

What I Need Help With:

Does anyone have experience running FreeRTOS on gem5?

  • Are there known issues with the RISC-V_RV32_QEMU_VIRT_GCC project on gem5?
  • Are there better reference configurations for running FreeRTOS on gem5?
  • Any suggestions on what could be causing this trap handler issue when calling xQueueCreate?

Any help or insights would be greatly appreciated!

Thank you very much in advance! :blush:

I don’t know much about gem5. But in that demo, unless you modified something in the Makefile. There is the expectation that the FPU is present in the hardware.
If gem5 does not support it, you may be executing flw, fld, etc. Which would be “undefined instructions” and drop you into the trap handler. Did you check the mcause CSR?

1 Like

If FPU is the issue as @cubidesj suspects, you can set configENABLE_FPU to 0 and it should work.

1 Like

Hi guys, thank you for your support!

I recompiled the application as you suggested, but the application seems stuck in the same place. Do you have any other ideas?

I’ll report you the Makefile, I shouldn’t modified anything (except adding the possibility of changing this flag, as you were suggesting me).

OUTPUT_DIR := ./output
IMAGE := RTOSDemo.elf

# The directory that contains the /source and /demo sub directories.
FREERTOS_ROOT = ./../../../..

CC = riscv64-unknown-elf-gcc
LD = riscv64-unknown-elf-gcc
SIZE = riscv64-unknown-elf-size
MAKE = make

# Generate GCC_VERSION in number format
GCC_VERSION = $(shell $(CC) --version | grep ^$(CC) | sed 's/^.* //g' | awk -F. '{ printf("%d%02d%02d"), $$1, $$2, $$3 }')
GCC_VERSION_NEED_ZICSR = "110100"

ifeq ($(RVA23),1)
  $(info ----- Using RVA23 build -----)
  # For the time being, we don't include the vector extensions.
  MARCH = rv64imafdc_zicsr_zicntr_zihpm_ziccif_ziccrse_ziccamoa_zicclsm_za64rs_zihintpause_zic64b_zicbom_zicbop_zicboz_zfhmin_zkt_zihintntl_zawrs
  MABI = lp64d
  MCMODEL = medany
  ifeq ($(FPU),1)
    $(info ------- Enabling the FPU) -------)
    CFLAGS+=-DconfigENABLE_FPU=1
  else
    CFLAGS+=-DconfigENABLE_FPU=0
  endif
else
  $(info ----- Using RV32 build -----)
  ifeq ($(shell test $(GCC_VERSION) -ge $(GCC_VERSION_NEED_ZICSR) && echo true),true)
    MARCH=rv32imac_zicsr
  else
    MARCH=rv32imac
  endif
  MABI=ilp32
  MCMODEL=medlow
endif

INCLUDE_DIRS += -I$(KERNEL_PORT_DIR)/chip_specific_extensions/RV32I_CLINT_no_extensions

CFLAGS += $(INCLUDE_DIRS) -fmessage-length=0 \
          -march=$(MARCH) -mabi=$(MABI) -mcmodel=$(MCMODEL) -ffunction-sections -fdata-sections \
          -Wno-unused-parameter -nostartfiles -g3 -Os


ifeq ($(PICOLIBC),1)
CFLAGS += --specs=picolibc.specs -DPICOLIBC_INTEGER_PRINTF_SCANF
else
CFLAGS += --specs=nano.specs -fno-builtin-printf
endif

LDFLAGS += -nostartfiles -Xlinker --gc-sections -Wl,-Map,$(OUTPUT_DIR)/RTOSDemo.map \
           -T./fake_rom.ld -march=$(MARCH) -mabi=$(MABI) -mcmodel=$(MCMODEL) -Xlinker \
           --defsym=__stack_size=350 -Wl,--start-group -Wl,--end-group

ifeq ($(PICOLIBC),1)
LDFLAGS += --specs=picolibc.specs --oslib=semihost --crt0=minimal -DPICOLIBC_INTEGER_PRINTF_SCANF
else
LDFLAGS +=  -Wl,--wrap=malloc \
           -Wl,--wrap=free -Wl,--wrap=open -Wl,--wrap=lseek -Wl,--wrap=read -Wl,--wrap=write \
           -Wl,--wrap=fstat -Wl,--wrap=stat -Wl,--wrap=close -Wl,--wrap=link -Wl,--wrap=unlink \
           -Wl,--wrap=execve -Wl,--wrap=fork -Wl,--wrap=getpid -Wl,--wrap=kill -Wl,--wrap=wait \
           -Wl,--wrap=isatty -Wl,--wrap=times -Wl,--wrap=sbrk -Wl,--wrap=puts -Wl,--wrap=_malloc \
           -Wl,--wrap=_free -Wl,--wrap=_open -Wl,--wrap=_lseek -Wl,--wrap=_read -Wl,--wrap=_write \
           -Wl,--wrap=_fstat -Wl,--wrap=_stat -Wl,--wrap=_close -Wl,--wrap=_link -Wl,--wrap=_unlink \
           -Wl,--wrap=_execve -Wl,--wrap=_fork -Wl,--wrap=_getpid -Wl,--wrap=_kill -Wl,--wrap=_wait \
           -Wl,--wrap=_isatty -Wl,--wrap=_times -Wl,--wrap=_sbrk -Wl,--wrap=__exit -Wl,--wrap=_puts
endif

# -Wl,--wrap=_exit
#
# Kernel build.
#
KERNEL_DIR = $(FREERTOS_ROOT)/Source
KERNEL_PORT_DIR += $(KERNEL_DIR)/portable/GCC/RISC-V
INCLUDE_DIRS += -I$(KERNEL_DIR)/include \
				-I$(KERNEL_PORT_DIR)
VPATH += $(KERNEL_DIR) $(KERNEL_PORT_DIR) $(KERNEL_DIR)/portable/MemMang
SOURCE_FILES += $(KERNEL_DIR)/tasks.c
SOURCE_FILES += $(KERNEL_DIR)/list.c
SOURCE_FILES += $(KERNEL_DIR)/queue.c
SOURCE_FILES += $(KERNEL_DIR)/timers.c
SOURCE_FILES += $(KERNEL_DIR)/event_groups.c
SOURCE_FILES += $(KERNEL_DIR)/stream_buffer.c
SOURCE_FILES += $(KERNEL_DIR)/portable/MemMang/heap_4.c
SOURCE_FILES += $(KERNEL_DIR)/portable/GCC/RISC-V/port.c
ASM_SOURCE_FILES += $(KERNEL_DIR)/portable/GCC/RISC-V/portASM.S

#
# Common demo files for the "full" build, as opposed to the "blinky" build -
# these files are build by all the FreeRTOS kernel demos.
#
DEMO_ROOT = $(FREERTOS_ROOT)/Demo
COMMON_DEMO_FILES = $(DEMO_ROOT)/Common/Minimal
INCLUDE_DIRS += -I$(DEMO_ROOT)/Common/include
VPATH += $(COMMON_DEMO_FILES)
SOURCE_FILES += (COMMON_DEMO_FILES)/AbortDelay.c
SOURCE_FILES += (COMMON_DEMO_FILES)/BlockQ.c
SOURCE_FILES += (COMMON_DEMO_FILES)/blocktim.c
SOURCE_FILES += (COMMON_DEMO_FILES)/countsem.c
SOURCE_FILES += (COMMON_DEMO_FILES)/death.c
SOURCE_FILES += (COMMON_DEMO_FILES)/dynamic.c
SOURCE_FILES += (COMMON_DEMO_FILES)/EventGroupsDemo.c
SOURCE_FILES += (COMMON_DEMO_FILES)/GenQTest.c
SOURCE_FILES += (COMMON_DEMO_FILES)/integer.c
SOURCE_FILES += (COMMON_DEMO_FILES)/IntSemTest.c
SOURCE_FILES += (COMMON_DEMO_FILES)/MessageBufferAMP.c
SOURCE_FILES += (COMMON_DEMO_FILES)/MessageBufferDemo.c
SOURCE_FILES += (COMMON_DEMO_FILES)/PollQ.c
SOURCE_FILES += (COMMON_DEMO_FILES)/QPeek.c
SOURCE_FILES += (COMMON_DEMO_FILES)/QueueOverwrite.c
SOURCE_FILES += (COMMON_DEMO_FILES)/QueueSet.c
SOURCE_FILES += (COMMON_DEMO_FILES)/QueueSetPolling.c
SOURCE_FILES += (COMMON_DEMO_FILES)/recmutex.c
SOURCE_FILES += (COMMON_DEMO_FILES)/semtest.c
SOURCE_FILES += (COMMON_DEMO_FILES)/StaticAllocation.c
SOURCE_FILES += (COMMON_DEMO_FILES)/StreamBufferDemo.c
SOURCE_FILES += (COMMON_DEMO_FILES)/StreamBufferInterrupt.c
SOURCE_FILES += (COMMON_DEMO_FILES)/TaskNotify.c
SOURCE_FILES += (COMMON_DEMO_FILES)/TaskNotifyArray.c
SOURCE_FILES += (COMMON_DEMO_FILES)/TimerDemo.c

#
# Application entry point.  main_blinky is self contained.  main_full builds
# the above common demo (and test) files too.
#
DEMO_PROJECT = ./../..
# DEMO_PROJECT = $(DEMO_ROOT)/RISC-V_RV32_QEMU_VIRT_GCC
VPATH += $(DEMO_PROJECT)
INCLUDE_DIRS += -I$(DEMO_PROJECT)
SOURCE_FILES += (DEMO_PROJECT)/main.c
SOURCE_FILES += (DEMO_PROJECT)/main_blinky.c
SOURCE_FILES += (DEMO_PROJECT)/main_full.c
SOURCE_FILES += (DEMO_PROJECT)/ns16550.c
SOURCE_FILES += (DEMO_PROJECT)/riscv-virt.c
# Lightweight print formatting to use in place of the heavier GCC equivalent.
ifneq ($(PICOLIBC),1)
SOURCE_FILES += ./printf-stdarg.c
endif
ASM_SOURCE_FILES += ./start.S
ASM_SOURCE_FILES += ./RegTest.S
ASM_SOURCE_FILES += ./vector.S


#Create a list of object files with the desired output directory path.
OBJS = $(SOURCE_FILES:%.c=%.o) $(ASM_SOURCE_FILES:%.S=%.o)
OBJS_NO_PATH = $(notdir $(OBJS))
OBJS_OUTPUT = $(OBJS_NO_PATH:%.o=$(OUTPUT_DIR)/%.o)

#Create a list of dependency files with the desired output directory path.
DEP_FILES := $(SOURCE_FILES:%.c=$(OUTPUT_DIR)/%.d) $(ASM_SOURCE_FILES:%.S=$(OUTPUT_DIR)/%.d)
DEP_FILES_NO_PATH = $(notdir $(DEP_FILES))
DEP_OUTPUT = $(DEP_FILES_NO_PATH:%.d=$(OUTPUT_DIR)/%.d)

all: $(OUTPUT_DIR)/$(IMAGE)

%.o : %.c
$(OUTPUT_DIR)/%.o : %.c $(OUTPUT_DIR)/%.d Makefile
	$(CC) $(CFLAGS) -MMD -MP -c $< -o $@

%.o : %.S
$(OUTPUT_DIR)/%.o: %.S $(OUTPUT_DIR)/%.d Makefile
	$(CC) $(CFLAGS) -MMD -MP -c $< -o $@

$(OUTPUT_DIR)/$(IMAGE): $(OBJS_OUTPUT) Makefile
	@echo ""
	@echo ""
	@echo "--- Final linking ---"
	@echo ""
	$(LD) $(OBJS_OUTPUT) $(LDFLAGS) -o $(OUTPUT_DIR)/$(IMAGE)
	$(SIZE) $(OUTPUT_DIR)/$(IMAGE)

$(DEP_OUTPUT):
include $(wildcard $(DEP_OUTPUT))

clean:
	rm -f $(OUTPUT_DIR)/$(IMAGE) $(OUTPUT_DIR)/*.o $(OUTPUT_DIR)/*.d $(OUTPUT_DIR)/*.map

#use "make print-[VARIABLE_NAME] to print the value of a variable generated by
#this makefile.
print-%  : ; @echo $* = $($*)

.PHONY: all clean

@cubidesj could you please explain to me where I can find the information you are asking for?
I don’t know where I can read the mcause CSR.

Edit: I am seeing also that you modified the project on GitHub yesterday, should I take the last version of the project or I can work with the previous version?

You can use the GDB command info all or p/x $mcause.

New changes are for vector extension support. You do not necessarily need them to debug this issue.

I am not familiar with gem5 but I’d suggest to get a bare metal code working first and then add FreeRTOS.

Hi Gaurav,

Thank you again for your answer! I will split my answer to analyse two different aspects.

Testing a bare-metal program

Regarding this observation of yours, I tried executing a very simple bare-metal program on gem5 (print-test.c):

#include <stdio.h>

int main() {
    printf("\n\nThis is just a test program!\n");
    printf("Last update: April 3, 2025\n\n");
    return 0;
}

Compiling the testing program

Then I compiled the application using the RISC-V tool-chain (statically, to be sure to have all the necessary dependencies)

  $ riscv64-unknown-elf-gcc print-test.c --static -o print-test-static

Executing the testing program on gem5

From what I know, gem5 can work in two different modes: “system emulation” and “full system”. The first one emulates the Linux system calls and is more similar to qemu, the second one tries to better model a real physical architecture.

If I execute my program in system emulation, it works correctly printing on the screen the two messages. However, if I try to execute the same application in full-system, I encounter an error:

src/mem/xbar.cc:368: fatal: Unable to find destination for [0xffffffffffffffe0:0xffffffffffffffe8] on system.membus

Error explanation

From my understanding, this happens because I haven’t set up a Linux kernel or bootloader in full-system mode.
Therefore, not having the necessary source file, the system is not able to correctly start-up.

So, if I would like to execute this simple program on gem5, I should configure a kernel (or at least a bootloader) and then I can correctly execute it. However, I think that this step would be useless for executing FreeRTOS (more details below).

Consideration related to FreeRTOS

  • System Emulation Mode → Likely doesn’t work because FreeRTOS doesn’t rely on Linux system calls.
  • Full System Mode → Since FreeRTOS is a minimal OS, I assume I don’t need a bootloader, and I should be able to run it directly.

Conclusions

I hope that all my assumptions are correct. I am new to all these different concepts, if I made some mistakes please correct me.
Would you recommend configuring a bootloader even for FreeRTOS, or should gem5 be able to run it as a standalone binary?

Thanks again for your help!

Your understanding is correct and FreeRTOS application should be able to run as a standalone binary.

1 Like

Hi again, guys, thank you to both of you for your support.

According to Gaurav’s comments

Here are the outputs of the two commands immediately after typing si (step into) on the function xQueueCreate(...). In that moment, I am on line 311 of file portASM.S on the instruction: portcontextSAVE_EXCEPTION_CONTEXT

(gdb) p/x $mcause
$1 0x6
(gdb) info all
zero           0x0	0
ra             0x80000078	0x80000078 <_text+120>
sp             0x800b715e	0x800b715e <ucHeap+223710>
gp             0x80080800	0x80080800 <ucHeap+128>
tp             0x0	0x0
t0             0x200	512
t1             0x0	0
t2             0x0	0
fp             0x0	0x0
s1             0x0	0
a0             0x1	1
a1             0x0	0
a2             0x0	0
a3             0x0	0
a4             0x0	0
a5             0x80002701	2147493633
a6             0x0	0
a7             0x0	0
s2             0x0	0
s3             0x0	0
s4             0x0	0
s5             0x0	0
s6             0x0	0
s7             0x0	0
s8             0x0	0
s9             0x0	0
s10            0x0	0
s11            0x0	0
t3             0x0	0
t4             0x0	0
t5             0x0	0
t6             0x0	0
pc             0x800023ac	0x800023ac <freertos_risc_v_exception_handler>
ft0            {float = 0, double = 0}	(raw 0x0000000000000000)
ft1            {float = 0, double = 0}	(raw 0x0000000000000000)
ft2            {float = 0, double = 0}	(raw 0x0000000000000000)
ft3            {float = 0, double = 0}	(raw 0x0000000000000000)
ft4            {float = 0, double = 0}	(raw 0x0000000000000000)
ft5            {float = 0, double = 0}	(raw 0x0000000000000000)
ft6            {float = 0, double = 0}	(raw 0x0000000000000000)
ft7            {float = 0, double = 0}	(raw 0x0000000000000000)
fs0            {float = 0, double = 0}	(raw 0x0000000000000000)
fs1            {float = 0, double = 0}	(raw 0x0000000000000000)
fa0            {float = 0, double = 0}	(raw 0x0000000000000000)
fa1            {float = 0, double = 0}	(raw 0x0000000000000000)
fa2            {float = 0, double = 0}	(raw 0x0000000000000000)
fa3            {float = 0, double = 0}	(raw 0x0000000000000000)
fa4            {float = 0, double = 0}	(raw 0x0000000000000000)
fa5            {float = 0, double = 0}	(raw 0x0000000000000000)
fa6            {float = 0, double = 0}	(raw 0x0000000000000000)
fa7            {float = 0, double = 0}	(raw 0x0000000000000000)
fs2            {float = 0, double = 0}	(raw 0x0000000000000000)
fs3            {float = 0, double = 0}	(raw 0x0000000000000000)
fs4            {float = 0, double = 0}	(raw 0x0000000000000000)
fs5            {float = 0, double = 0}	(raw 0x0000000000000000)
fs6            {float = 0, double = 0}	(raw 0x0000000000000000)
fs7            {float = 0, double = 0}	(raw 0x0000000000000000)
fs8            {float = 0, double = 0}	(raw 0x0000000000000000)
fs9            {float = 0, double = 0}	(raw 0x0000000000000000)
fs10           {float = 0, double = 0}	(raw 0x0000000000000000)
fs11           {float = 0, double = 0}	(raw 0x0000000000000000)
ft8            {float = 0, double = 0}	(raw 0x0000000000000000)
ft9            {float = 0, double = 0}	(raw 0x0000000000000000)
ft10           {float = 0, double = 0}	(raw 0x0000000000000000)
ft11           {float = 0, double = 0}	(raw 0x0000000000000000)
ustatus        0x0	0
fflags         0x0	NV:0 DZ:0 OF:0 UF:0 NX:0
frm            0x0	FRM:0 [RNE (round to nearest; ties to even)]
fcsr           0x0	NV:0 DZ:0 OF:0 UF:0 NX:0 FRM:0 [RNE (round to nearest; ties to even)]
uie            0x0	0
utvec          0x0	0
uscratch       0x0	0
uepc           0x0	0
ucause         0x0	0
utval          0x0	0
uip            0x0	0
sstatus        0x200002200	8589943296
sedeleg        0x0	0
sideleg        0x0	0
sie            0x0	0
stvec          0x0	0
scounteren     0x7	7
senvcfg        0x0	0
sscratch       0x0	0
sepc           0x0	0
scause         0x0	0
stval          0x0	0
sip            0x0	0
satp           0x0	0
mstatus        0xa00003a00	SD:0 VM:00 MXR:0 PUM:0 MPRV:0 XS:0 FS:1 MPP:3 HPP:1 SPP:0 MPIE:0 HPIE:0 SPIE:0 UPIE:0 MIE:0 HIE:0 SIE:0 UIE:0
misa           0x800000000034112d	RV64ACDFIMSUV
medeleg        0x0	0
mideleg        0x0	0
mie            0x0	0
mtvec          0x80002701	2147493633
mcounteren     0x7	7
mscratch       0x0	0
mepc           0x800023b0	2147492784
mcause         0x6	6
mtval          0x800b716e	2148233582
mip            0x0	0
hstatus        0x75626d656d5f6d65	8458443332549307749
hedeleg        0x6464616461625f73	7234013985222319987
hideleg        0x6e6f707365725f72	7957702707512500082
hie            0x3c6f69705f726564	4354815296049145188
hip            0x3338384d223d6420	3690761798568207392
cycle          0x0	0
time           0x0	0
mvendorid      0x0	0
marchid        0x0	0
mimpid         0x0	0
mhartid        0x0	0
htvec          0xa3e656c7469742f	738138905419281455
hscratch       0x696620687461703c	7594793454608150588
hepc           0x656e6f6e223d6c6c	7308901764080430188
hcause         0x656b6f7274732022	7308057357709418530
hbadaddr       0x226b63616c62223d	2480185289878938173
placeholder    0x362d2c30	908930096

At this moment, I notice that the line in which I am stuck differs from the one I mentioned at the beginning of this topic… I have no idea if this is because now I am using the latest version of this project (disabling VPU and FPU), or if I made a mistake in my previous debug session…

If you have any ideas or suggestions to help me start the project it would be very appreciated!

Thank you very much again for your support!

This is a macro. It would be helpful to know which exact instruction is faulting. Can you run display /i $pc so that the next instruction is always displayed on the terminal? This way you can find the faulting instruction while stepping through the assembly using the si command.

According to the spec, this is Store/AMO address misaligned exception. Finding the instruction and the address that the faulting instruction is trying to access, would be helpful here.

$mcause is 6 which means “store address misaligned” and looking at $mtval I indeed see an unaligned address 0x800b716e which seems to be in your stack ($sp is 0x800b715e). The question is now, what put $sp in that condition.

Hi all,

I tracked the issue down to the stack alignment. The exception was triggered because the store instruction was using an unaligned address.

The linker script sets up the stack like this:

	.stack :
	{
		. = ALIGN(16);
		. += __stack_size;
		_stack_top = .;
	} >ram AT>ram

By default, the __stack_size is equal to 350 bytes via the linker flags in the default makefile, but 350 isn’t a multiple of 16.

This meant that even though the stack section started aligned (thanks to ALIGN(16)), adding 350 bytes resulted in a _stack_top (and thus sp) that wasn’t 16‑byte aligned. That’s why $mcause was 6 (“store address misaligned”) and $mtval showed an unaligned address within the stack.

I modified __stack_size to 352, which ensured that _stack_top is properly aligned on a 16‑byte boundary.

After making that change, the misaligned store exception stopped occurring.

Hope this helps anyone facing similar problems!

3 Likes

A way to be defensive about the stack_size issue is to use code like:

.stack :
	{
		. = ALIGN(16);
		. += __stack_size;
		. = ALIGN(16);
		_stack_top = .;
	} >ram AT>ram

Then stack_top will also always be aligned.

1 Like

Thank you for sharing the solution and raising the PR!

1 Like

Thank you for your comment!

You are right! I restored the previous value in the makefile and tested your solution, and it worked!
You should propose a pull request with this change, too!

I will continue working on keeping your modification in the linker script!