FreeRTOS hangs after vTaskEndScheduler in Posix_GCC port

I have problems with Posix GCC port.

I’m using FreeRTOS on x86-64 to run some tests and at the end call vTaskEndScheduler().

I’ve created MRE based on blinky example app:

diff --git a/FreeRTOS/Demo/Posix_GCC/main.c b/FreeRTOS/Demo/Posix_GCC/main.c
index e89da3b9e..9a2fc0d3d 100644
--- a/FreeRTOS/Demo/Posix_GCC/main.c
+++ b/FreeRTOS/Demo/Posix_GCC/main.c
@@ -82,11 +82,13 @@
 #endif
 
 /* Demo type is passed as an argument */
-#ifdef USER_DEMO
-    #define     mainSELECTED_APPLICATION    USER_DEMO
-#else /* Default Setting */
-    #define    mainSELECTED_APPLICATION     FULL_DEMO
-#endif
+// #ifdef USER_DEMO
+//     #define     mainSELECTED_APPLICATION    USER_DEMO
+// #else /* Default Setting */
+//     #define    mainSELECTED_APPLICATION     FULL_DEMO
+// #endif
+
+#define     mainSELECTED_APPLICATION    USER_DEMO
 
 /* This demo uses heap_3.c (the libc provided malloc() and free()). */
 
diff --git a/FreeRTOS/Demo/Posix_GCC/main_blinky.c b/FreeRTOS/Demo/Posix_GCC/main_blinky.c
index c599c933a..d451d2841 100644
--- a/FreeRTOS/Demo/Posix_GCC/main_blinky.c
+++ b/FreeRTOS/Demo/Posix_GCC/main_blinky.c
@@ -174,9 +174,9 @@ void main_blinky( void )
      * there was insufficient FreeRTOS heap memory available for the idle and/or
      * timer tasks	to be created.  See the memory management section on the
      * FreeRTOS web site for more details. */
-    for( ; ; )
-    {
-    }
+    // for( ; ; )
+    // {
+    // }
 }
 /*-----------------------------------------------------------*/
 
@@ -205,6 +205,10 @@ static void prvQueueSendTask( void * pvParameters )
          * will not block - it shouldn't need to block as the queue should always
          * have at least one space at this point in the code. */
         xQueueSend( xQueue, &ulValueToSend, 0U );
+
+        if (xNextWakeTime > 400) {
+            vTaskEndScheduler();
+        }
     }
 }
 /*-----------------------------------------------------------*/

I’ve compiled as cmake .; make.

Running in loop with until ! nice -n 20 ./posix_demo; do date; done.

After some time (10-100 loops) posix_demo hangs.

Attaching to gdb shows following:

# sudo gdb -p 35553                                                                                                                                                                                                                                                   
GNU gdb (Ubuntu 15.1-1ubuntu2) 15.1
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 35553
[New LWP 35559]
[New LWP 35557]
[New LWP 35556]
[New LWP 35555]
[New LWP 35554]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x0000719b4b845ed8 in __GI___sigtimedwait (set=set@entry=0x7fff1df3ff70, info=info@entry=0x7fff1df3fea0, timeout=timeout@entry=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:31

warning: 31     ../sysdeps/unix/sysv/linux/sigtimedwait.c: Nie ma takiego pliku ani katalogu
(gdb) info thr
  Id   Target Id                                           Frame
* 1    Thread 0x719b4ba7e740 (LWP 35553) "Scheduler"       0x0000719b4b845ed8 in __GI___sigtimedwait (set=set@entry=0x7fff1df3ff70, info=info@entry=0x7fff1df3fea0, timeout=timeout@entry=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:31
  2    Thread 0x719b484006c0 (LWP 35559) "Scheduler timer" 0x0000719b4b8f38e3 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x719b483ffe70, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:48
  3    Thread 0x719b498006c0 (LWP 35557) "IDLE"            0x0000719b4b89dffd in __futex_abstimed_wait_common64 (private=0, futex_word=0x6218c9f76e60, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57
  4    Thread 0x719b4a2006c0 (LWP 35556) "TX"              futex_wait (futex_word=0x6218c9f76fa0, expected=2, private=0) at ../sysdeps/nptl/futex-internal.h:146
  5    Thread 0x719b4ac006c0 (LWP 35555) "Rx"              0x0000719b4b89dffd in __futex_abstimed_wait_common64 (private=29083, futex_word=0x6218c9f76a24, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57
  6    Thread 0x719b4b6006c0 (LWP 35554) "TzCtrl"          0x0000719b4b89dffd in __futex_abstimed_wait_common64 (private=0, futex_word=0x6218c9f562f4, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57
(gdb) thr 4
[Switching to thread 4 (Thread 0x719b4a2006c0 (LWP 35556))]
#0  futex_wait (futex_word=0x6218c9f76fa0, expected=2, private=0) at ../sysdeps/nptl/futex-internal.h:146
warning: 146    ../sysdeps/nptl/futex-internal.h: Nie ma takiego pliku ani katalogu
(gdb) bt
#0  futex_wait (futex_word=0x6218c9f76fa0, expected=2, private=0) at ../sysdeps/nptl/futex-internal.h:146
#1  __GI___lll_lock_wait (futex=futex@entry=0x6218c9f76fa0, private=0) at ./nptl/lowlevellock.c:49
#2  0x0000719b4b8a5541 in lll_mutex_lock_optimized (mutex=0x6218c9f76fa0) at ./nptl/pthread_mutex_lock.c:48
#3  ___pthread_mutex_lock (mutex=0x6218c9f76fa0) at ./nptl/pthread_mutex_lock.c:93
#4  0x00006218a6eb583e in event_signal (ev=0x6218c9f76fa0) at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/utils/wait_for_event.c:104
#5  0x00006218a6eb536c in vPortCancelThread (pxTaskToDelete=0x6218a6f1f320 <xTimerTaskTCB.0>) at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c:445
#6  0x00006218a6eb13d7 in prvDeleteTCB (pxTCB=0x6218a6f1f320 <xTimerTaskTCB.0>) at /data/code/FreeRTOS/FreeRTOS/Source/tasks.c:6432
#7  0x00006218a6eade64 in vTaskDelete (xTaskToDelete=0x6218a6f1f320 <xTimerTaskTCB.0>) at /data/code/FreeRTOS/FreeRTOS/Source/tasks.c:2319
#8  0x00006218a6eaf0aa in vTaskEndScheduler () at /data/code/FreeRTOS/FreeRTOS/Source/tasks.c:3778
#9  0x00006218a6e90abc in prvQueueSendTask (pvParameters=0x0) at /data/code/FreeRTOS/FreeRTOS/Demo/Posix_GCC/main_blinky.c:210
#10 0x00006218a6eb53f1 in prvWaitForStart (pvParams=0x719b4ba7cfe8) at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c:465
#11 0x0000719b4b8a1e2e in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#12 0x0000719b4b933a4c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
(gdb) fr 3
#3  ___pthread_mutex_lock (mutex=0x6218c9f76fa0) at ./nptl/pthread_mutex_lock.c:93
warning: 93     ./nptl/pthread_mutex_lock.c: Nie ma takiego pliku ani katalogu
(gdb) p mutex.__data
$1 = {__lock = 2, __count = 0, __owner = 35558, __nusers = 1, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}
(gdb)

To summarize: task TX waits on mutex. Its owner is task 35558 which no longer exists. It was Tmr Svc task. To add - Tmr Svc task is first task deleted after calling vTaskEndScheduler(). With vTaskSuspendAll() before vTaskEndScheduler() app still hangs.

Problem occurs on current master (2dcc47ecb) and 202406.01 LTS.

I wonder it I’m using vTaskEndScheduler incorrectly or there some issue with that port.

There is a recent PR to address a hang in the POSIX port’s vTaskEndScheduler - POSIX Simulator: Handle `pthread`s not created by FreeRTOS differently by johnboiles · Pull Request #1223 · FreeRTOS/FreeRTOS-Kernel · GitHub. Would you please try that and see if it address your issue?

I’ve tested and nothing changed. The issue remains the same - TX task waits on mutex with owner Tmr Svc which is deleted.

I’m not sure that’s my situation after reading PR description - vTaskEndScheduler is called from FreeRTOS task and all tasks in example are FreeRTOS tasks.

I tried running your sample code for a long time and I am unable to repro this. This indicates that it is likely a race condition. I am trying to read the code and your call stack to determine the cause of the hang. Would you please test the following patch: posix_port.patch (614 Bytes).

Without PR 1223 and with patch I immediately get following crash (multiple times):

gdb ./posix_demo                        ─╯
GNU gdb (Ubuntu 15.1-1ubuntu2) 15.1
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./posix_demo...
(gdb) break configASSERT
Function "configASSERT" not defined.
Make breakpoint pending on future shared library load? (y or [n]) n
(gdb) break vAssertCalled
Breakpoint 1 at 0x376d: file /data/code/FreeRTOS/FreeRTOS/Demo/Posix_GCC/main.c, line 328.
(gdb) r
Starting program: /data/code/FreeRTOS/FreeRTOS/Demo/Posix_GCC/posix_demo 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff7a006c0 (LWP 48280)]

Trace started.
The trace will be dumped to disk if a call to configASSERT() fails.
Starting echo blinky demo
[New Thread 0x7ffff70006c0 (LWP 48281)]
[New Thread 0x7ffff66006c0 (LWP 48282)]
[New Thread 0x7ffff5c006c0 (LWP 48283)]
[New Thread 0x7ffff52006c0 (LWP 48284)]
[New Thread 0x7ffff48006c0 (LWP 48285)]
Message received from task
Message received from task
[Thread 0x7ffff52006c0 (LWP 48284) exited]
[Switching to Thread 0x7ffff66006c0 (LWP 48282)]

Thread 4 "TX" hit Breakpoint 1, vAssertCalled (pcFileName=0x55555557e870 "/data/code/FreeRTOS/FreeRTOS/Source/tasks.c", ulLine=5132) at /data/code/FreeRTOS/FreeRTOS/Demo/Posix_GCC/main.c:328
328         volatile uint32_t ulSetToNonZeroInDebuggerToContinue = 0;
(gdb) bt
#0  vAssertCalled (pcFileName=0x55555557e870 "/data/code/FreeRTOS/FreeRTOS/Source/tasks.c", ulLine=5132) at /data/code/FreeRTOS/FreeRTOS/Demo/Posix_GCC/main.c:328
#1  0x000055555557733e in vTaskSwitchContext () at /data/code/FreeRTOS/FreeRTOS/Source/tasks.c:5132
#2  0x000055555557c0e2 in prvPortYieldFromISR () at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c:309
#3  0x000055555557c11b in vPortYield () at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c:321
#4  0x0000555555574ea2 in vTaskDelete (xTaskToDelete=0x5555555c6260 <xIdleTaskTCB.2>) at /data/code/FreeRTOS/FreeRTOS/Source/tasks.c:2331
#5  0x00005555555760d3 in vTaskEndScheduler () at /data/code/FreeRTOS/FreeRTOS/Source/tasks.c:3785
#6  0x0000555555557abc in prvQueueSendTask (pvParameters=0x0) at /data/code/FreeRTOS/FreeRTOS/Demo/Posix_GCC/main_blinky.c:210
#7  0x000055555557c3f1 in prvWaitForStart (pvParams=0x7ffff7fa1fe8) at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c:465
#8  0x00007ffff7ca1e2e in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#9  0x00007ffff7d33a4c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

I have no hard evidence but feel like running demo with lower priority and some benchmark (e.g. `openssl speed -multi 8) increases likelihood of hang. But hangs also occurs without load.

Crash with your patch was without extra load.

It looks like same issue as 1217 in FreeRTOS-Kernel (cannot post link).