I’ll try disabling projENABLE_TRACING/configUSE_TRACE_FACILITY.
Without projENABLE_TRACING/configUSE_TRACE_FACILITY there’s also hang.
(gdb) info thr
Id Target Id Frame
* 1 Thread 0x74ddda5e4740 (LWP 265340) "Scheduler" 0x000074ddda245ed8 in __GI___sigtimedwait (set=set@entry=0x7ffc4d7ab040, info=info@entry=0x7ffc4d7aaf70, timeout=timeout@entry=0x0)
at ../sysdeps/unix/sysv/linux/sigtimedwait.c:31
2 Thread 0x74ddcb4006c0 (LWP 265345) "Scheduler timer" 0x000074ddda2f38e3 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x74ddcb3ffe70,
rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:48
3 Thread 0x74ddd3e006c0 (LWP 265344) "Tmr Svc" 0x000074ddda29dffd in __futex_abstimed_wait_common64 (private=0, futex_word=0x5677cc76ae38, expected=0, op=393, abstime=0x0, cancel=true)
at ./nptl/futex-internal.c:57
4 Thread 0x74ddd8c006c0 (LWP 265343) "IDLE" 0x000074ddda2f38e3 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x74ddd8bffe40,
rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:48
5 Thread 0x74ddd96006c0 (LWP 265342) "TX" 0x000074ddda29dffd in __futex_abstimed_wait_common64 (private=0, futex_word=0x5677cc76aab8, expected=0, op=393, abstime=0x0, cancel=true)
at ./nptl/futex-internal.c:57
6 Thread 0x74ddda0006c0 (LWP 265341) "Rx" 0x000074ddda29dffd in __futex_abstimed_wait_common64 (private=0, futex_word=0x5677cc76a878, expected=0, op=393, abstime=0x0, cancel=true)
at ./nptl/futex-internal.c:57
(gdb) bt
#0 0x000074ddda245ed8 in __GI___sigtimedwait (set=set@entry=0x7ffc4d7ab040, info=info@entry=0x7ffc4d7aaf70, timeout=timeout@entry=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:31
#1 0x000074ddda2455ac in __GI___sigwait (set=0x7ffc4d7ab040, sig=0x7ffc4d7ab03c) at ../sysdeps/unix/sysv/linux/sigwait.c:28
#2 0x00005677b1f20887 in xPortStartScheduler () at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c:293
#3 0x00005677b1f1bca3 in vTaskStartScheduler () at /data/code/FreeRTOS/FreeRTOS/Source/tasks.c:3739
#4 0x00005677b1f0444a in main_blinky () at /data/code/FreeRTOS/FreeRTOS/Demo/Posix_GCC/main_blinky.c:169
#5 0x00005677b1f04090 in main () at /data/code/FreeRTOS/FreeRTOS/Demo/Posix_GCC/main.c:176
(gdb) thr 2
[Switching to thread 2 (Thread 0x74ddcb4006c0 (LWP 265345))]
#0 0x000074ddda2f38e3 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x74ddcb3ffe70, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:48
warning: 48 ../sysdeps/unix/sysv/linux/clock_nanosleep.c: Nie ma takiego pliku ani katalogu
(gdb) bt
#0 0x000074ddda2f38e3 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x74ddcb3ffe70, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:48
#1 0x000074ddda301597 in __GI___nanosleep (req=req@entry=0x74ddcb3ffe70, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
#2 0x000074ddda33353a in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31
#3 0x00005677b1f20b40 in prvTimerTickHandler (arg=0x0) at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c:453
#4 0x000074ddda2a1e2e in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#5 0x000074ddda333a4c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
(gdb) thr 3
[Switching to thread 3 (Thread 0x74ddd3e006c0 (LWP 265344))]
#0 0x000074ddda29dffd in __futex_abstimed_wait_common64 (private=0, futex_word=0x5677cc76ae38, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57
warning: 57 ./nptl/futex-internal.c: Nie ma takiego pliku ani katalogu
(gdb) bt
#0 0x000074ddda29dffd in __futex_abstimed_wait_common64 (private=0, futex_word=0x5677cc76ae38, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (futex_word=0x5677cc76ae38, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5677cc76ae38, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0)
at ./nptl/futex-internal.c:139
#3 0x000074ddda2a0b6d in __pthread_cond_wait_common (cond=0x5677cc76ae10, mutex=0x5677cc76ade0, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_wait (cond=0x5677cc76ae10, mutex=0x5677cc76ade0) at ./nptl/pthread_cond_wait.c:627
#5 0x00005677b1f21058 in event_wait (ev=0x5677cc76ade0) at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/utils/wait_for_event.c:84
#6 0x00005677b1f20ddb in prvSuspendSelf (thread=0x5677b1f69178 <uxTimerTaskStack+262104>) at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c:604
#7 0x00005677b1f20dad in prvSwitchThread (pxThreadToResume=0x5677cc76a738, pxThreadToSuspend=0x5677b1f69178 <uxTimerTaskStack+262104>)
at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c:582
#8 0x00005677b1f209fe in prvPortYieldFromISR () at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c:377
#9 0x00005677b1f20a13 in vPortYield () at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c:385
#10 0x00005677b1f1fcf3 in prvProcessTimerOrBlockTask (xNextExpireTime=2000, xListWasEmpty=0) at /data/code/FreeRTOS/FreeRTOS/Source/timers.c:828
#11 0x00005677b1f1fc25 in prvTimerTask (pvParameters=0x0) at /data/code/FreeRTOS/FreeRTOS/Source/timers.c:774
#12 0x00005677b1f20d39 in prvWaitForStart (pvParams=0x5677b1f69178 <uxTimerTaskStack+262104>) at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c:546
#13 0x000074ddda2a1e2e in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x000074ddda333a4c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
(gdb) thr 4
[Switching to thread 4 (Thread 0x74ddd8c006c0 (LWP 265343))]
#0 0x000074ddda2f38e3 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x74ddd8bffe40, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:48
warning: 48 ../sysdeps/unix/sysv/linux/clock_nanosleep.c: Nie ma takiego pliku ani katalogu
(gdb) bt
#0 0x000074ddda2f38e3 in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x74ddd8bffe40, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:48
#1 0x000074ddda301597 in __GI___nanosleep (req=req@entry=0x74ddd8bffe40, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/nanosleep.c:25
#2 0x000074ddda33353a in usleep (useconds=<optimized out>) at ../sysdeps/posix/usleep.c:31
#3 0x00005677b1f040c8 in vApplicationIdleHook () at /data/code/FreeRTOS/FreeRTOS/Demo/Posix_GCC/main.c:226
#4 0x00005677b1f1d710 in prvIdleTask (pvParameters=0x0) at /data/code/FreeRTOS/FreeRTOS/Source/tasks.c:5838
#5 0x00005677b1f20d39 in prvWaitForStart (pvParams=0x5677b1f89258 <uxIdleTaskStack.1+131032>) at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c:546
#6 0x000074ddda2a1e2e in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#7 0x000074ddda333a4c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
(gdb) thr 5
[Switching to thread 5 (Thread 0x74ddd96006c0 (LWP 265342))]
#0 0x000074ddda29dffd in __futex_abstimed_wait_common64 (private=0, futex_word=0x5677cc76aab8, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57
warning: 57 ./nptl/futex-internal.c: Nie ma takiego pliku ani katalogu
(gdb) bt
#0 0x000074ddda29dffd in __futex_abstimed_wait_common64 (private=0, futex_word=0x5677cc76aab8, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (futex_word=0x5677cc76aab8, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5677cc76aab8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0)
at ./nptl/futex-internal.c:139
#3 0x000074ddda2a0b6d in __pthread_cond_wait_common (cond=0x5677cc76aa90, mutex=0x5677cc76aa60, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_wait (cond=0x5677cc76aa90, mutex=0x5677cc76aa60) at ./nptl/pthread_cond_wait.c:627
#5 0x00005677b1f21058 in event_wait (ev=0x5677cc76aa60) at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/utils/wait_for_event.c:84
#6 0x00005677b1f20ddb in prvSuspendSelf (thread=0x74ddda5e2fe8) at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c:604
#7 0x00005677b1f20dad in prvSwitchThread (pxThreadToResume=0x5677b1f89258 <uxIdleTaskStack.1+131032>, pxThreadToSuspend=0x74ddda5e2fe8)
at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c:582
#8 0x00005677b1f209fe in prvPortYieldFromISR () at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c:377
#9 0x00005677b1f20a13 in vPortYield () at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c:385
#10 0x00005677b1f1af4b in xTaskDelayUntil (pxPreviousWakeTime=0x74ddd95ffe78, xTimeIncrement=200) at /data/code/FreeRTOS/FreeRTOS/Source/tasks.c:2425
#11 0x00005677b1f04497 in prvQueueSendTask (pvParameters=0x0) at /data/code/FreeRTOS/FreeRTOS/Demo/Posix_GCC/main_blinky.c:201
#12 0x00005677b1f20d39 in prvWaitForStart (pvParams=0x74ddda5e2fe8) at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c:546
#13 0x000074ddda2a1e2e in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x000074ddda333a4c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
(gdb) thr 6
[Switching to thread 6 (Thread 0x74ddda0006c0 (LWP 265341))]
#0 0x000074ddda29dffd in __futex_abstimed_wait_common64 (private=0, futex_word=0x5677cc76a878, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57
57 in ./nptl/futex-internal.c
(gdb) bt
#0 0x000074ddda29dffd in __futex_abstimed_wait_common64 (private=0, futex_word=0x5677cc76a878, expected=0, op=393, abstime=0x0, cancel=true) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (futex_word=0x5677cc76a878, expected=0, clockid=0, abstime=0x0, private=0, cancel=true) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5677cc76a878, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0)
at ./nptl/futex-internal.c:139
#3 0x000074ddda2a0b6d in __pthread_cond_wait_common (cond=0x5677cc76a850, mutex=0x5677cc76a820, clockid=0, abstime=0x0) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_wait (cond=0x5677cc76a850, mutex=0x5677cc76a820) at ./nptl/pthread_cond_wait.c:627
#5 0x00005677b1f21058 in event_wait (ev=0x5677cc76a820) at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/utils/wait_for_event.c:84
#6 0x00005677b1f20ddb in prvSuspendSelf (thread=0x5677cc76a738) at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c:604
#7 0x00005677b1f20dad in prvSwitchThread (pxThreadToResume=0x74ddda5e2fe8, pxThreadToSuspend=0x5677cc76a738) at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c:582
#8 0x00005677b1f209fe in prvPortYieldFromISR () at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c:377
#9 0x00005677b1f20a13 in vPortYield () at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c:385
#10 0x00005677b1f17c0d in xQueueReceive (xQueue=0x5677cc74a6b0, pvBuffer=0x74ddd9fffe84, xTicksToWait=18446744073709551615) at /data/code/FreeRTOS/FreeRTOS/Source/queue.c:1614
#11 0x00005677b1f0455d in prvQueueReceiveTask (pvParameters=0x0) at /data/code/FreeRTOS/FreeRTOS/Demo/Posix_GCC/main_blinky.c:248
#12 0x00005677b1f20d39 in prvWaitForStart (pvParams=0x5677cc76a738) at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c:546
#13 0x000074ddda2a1e2e in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#14 0x000074ddda333a4c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
(gdb) p xTickCount
$2 = 2
(gdb)
- I see that the value of
xTickCount
is 2. Does it mean that it stops incrementing? Can you put a a breakpoint invPortSystemTickHandler
function and see if it is getting called periodically? - Do you have access to 24.04? I want to see if this is 24.10 specific.
- Also, update the OS by running
sudo apt update
andsudo apt upgrade
.
-
Yes, it’s stop incrementing. I attach gdb to process when I notice hang - 10 minutes, 1 hour or after weekend. I’ll check if
vPortSystemTickHandler
is called. -
I’ll try “LiveCD” tomorrow.
-
I have up-to-date system.
vPortSystemTickHandler
is not called.
sudo gdb -p 103526
GNU gdb (Ubuntu 15.1-1ubuntu2) 15.1
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 103526
[New LWP 103531]
[New LWP 103530]
[New LWP 103529]
[New LWP 103528]
[New LWP 103527]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x0000776ecc445ed8 in __GI___sigtimedwait (set=set@entry=0x7fffc91935d0, info=info@entry=0x7fffc9193500,
timeout=timeout@entry=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:31
warning: 31 ../sysdeps/unix/sysv/linux/sigtimedwait.c: Nie ma takiego pliku ani katalogu
(gdb) p xTickCount
$1 = 1
(gdb) break vPortSystemTickHandler
Breakpoint 1 at 0x562ae4f29ba2: file /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c, line 475.
(gdb) c
Continuing.
//nothing
^C
Thread 2 "Scheduler timer" received signal SIGINT, Interrupt.
[Switching to Thread 0x776ec9a006c0 (LWP 103531)]
__GI___pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ./nptl/pthread_kill.c:69
warning: 69 ./nptl/pthread_kill.c: Nie ma takiego pliku ani katalogu
(gdb) p xTickCount
$2 = 1
Can you put a breakpoint here and see if we are periodically issuing SIGALRM?
Yes, I hit this breakpoint multiple times.
Attaching to process 1178646
[New LWP 1178651]
[New LWP 1178650]
[New LWP 1178649]
[New LWP 1178648]
[New LWP 1178647]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007c3a42c45ed8 in __GI___sigtimedwait (set=set@entry=0x7ffd324600e0, info=info@entry=0x7ffd32460010,
timeout=timeout@entry=0x0) at ../sysdeps/unix/sysv/linux/sigtimedwait.c:31
warning: 31 ../sysdeps/unix/sysv/linux/sigtimedwait.c: Nie ma takiego pliku ani katalogu
(gdb) break pthread_kill
Breakpoint 1 at 0x7c3a42ca3e00: pthread_kill. (2 locations)
(gdb) c
Continuing.
[Switching to Thread 0x7c3a3be006c0 (LWP 1178651)]
Thread 2 "Scheduler timer" hit Breakpoint 1.1, __GI___pthread_kill (threadid=136589646759616, signo=14)
at ./nptl/pthread_kill.c:83
warning: 83 ./nptl/pthread_kill.c: Nie ma takiego pliku ani katalogu
(gdb) bt
#0 __GI___pthread_kill (threadid=136589646759616, signo=14) at ./nptl/pthread_kill.c:83
#1 0x0000576e2698eb36 in prvTimerTickHandler (arg=0x0)
at /data/code/FreeRTOS/FreeRTOS/Source/portable/ThirdParty/GCC/Posix/port.c:452
#2 0x00007c3a42ca1e2e in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#3 0x00007c3a42d33a4c in __GI___clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
(gdb) c
Continuing.
Thread 2 "Scheduler timer" hit Breakpoint 1.1, __GI___pthread_kill (threadid=136589646759616, signo=14)
at ./nptl/pthread_kill.c:83
83 in ./nptl/pthread_kill.c
(gdb) c
Continuing.
Thread 2 "Scheduler timer" hit Breakpoint 1.1, __GI___pthread_kill (threadid=136589646759616, signo=14)
at ./nptl/pthread_kill.c:83
83 in ./nptl/pthread_kill.c
(gdb) c
Continuing.
Thread 2 "Scheduler timer" hit Breakpoint 1.1, __GI___pthread_kill (threadid=136589646759616, signo=14)
at ./nptl/pthread_kill.c:83
83 in ./nptl/pthread_kill.c
(gdb) c
Continuing.
Thread 2 "Scheduler timer" hit Breakpoint 1.1, __GI___pthread_kill (threadid=136589646759616, signo=14)
at ./nptl/pthread_kill.c:83
83 in ./nptl/pthread_kill.c
(gdb)
I run Ubuntu 24.04.1 LTS from “LiveCD” with same binary. After 2 hours app didn’t hanged.
On Ubuntu 24.10 hang occurs in 5 minutes or earlier.
This means that we are issuing SIGALRM but it is not getting delivered to the target thread. Are you manipulating signal mask in your application? Also, can you apply the following patch and share the output: logging.patch (1.9 KB).
So it is likely 24.10 specific.
With patch on Ubuntu 24.10.
hang.7z (13.9 KB)
Each app execution is separated by 4 lut 2025
.
Log is created by until ! nice -n 20 ./posix_demo 2>&1; do date; done | tee -a hang.txt
.
After attaching gdb:
(gdb) p xTickCount
$1 = 3
I didn’t add any signal manipulating, still using app as described in first post.