I’m really happy, as my SMP port is barely working: I see my tasks running correctly on both core, displaying the expected trace on UART terminal, aaand… it gets blocked. There is no Fault or else. If I look with the debuggers, I see my two cores are traped in the loop to lock the spinlocks.
to summarize the state a little time before the trap :
core 0 owns the ISR lock / core 1 does not own any lock
in vTaskSwitchContext, the core 1 takes the task lock successfully
in vTaskEnterCritical, the core 0 tries to take the task lock, but can’t as it’s now owned by the core 1 : it gets trapped into an infinite loop until it can correctly take it.
still in vTaskSwitchContext, the core 1 tries to take the ISR lock, but can’t as it’s stillowned by the core 0 : it gets trapped into an infinite loop too.
And at this point, I have the two cores trapped for ever. hum hum
I’ve two starting points to try to handle this.
First, even if I can’t modify it, I take a look at tasks.c (currently the only file using these two spinlocks.) I’m surprised by what I see. I expected to see the same count of get and release for the ISR lock : it’s the case for the task lock, and seems pertinent, as it would maintain a kind of balance ? But I count 5 get and 6 release for the ISR lock. I also expected to have the same amount of get for both task and ISR lock, but I only have 4 get/release for the task lock. how can I explain this ?
The second track is about how I implemented my software spinlock: It will goes in an infinite loop until the lock is correctly… locked. Maybe it’s too overzealous ??
You are describing a fundamental lock deadlock. Two independent paths on a pair of Mutex-like locks must take them in the same order, or you get this sort of deadlock.
I am not sure how FreeRTOS defines its locks, but my first guess is that you aren’t supposed to take the ISR lock without the Task Lock (or at least so you can’t try to take the task lock while holding the ISR lock).
It’s indeed what I expected to see, after I read this line in tasks.c. I see portGET_TASK_LOCK() followed by portGET_ISR_LOCK() as mentioned in the commentary. And it made me think that the release must be in the opposite order, to prevent my situation: portRELEASE_ISR_LOCK() and then portRELEASE_TASK_LOCK(). but it’s not possible given the number of calls to these four macros, and it’s not the case everywhere. The most confusing for me is the lines 842 and 855. I planed to start drawing a graph of all the calls after a break, to make it clearer.
The locks are defined by myself, and are used through the predefined macro I mentioned above. It works as expected when I do simple unit tests, but it’s no longer the case when it’s used more actively by the OS. However, I don’t use them in the application nor in port.c. All the calls I counted are in tasks.c, that’s why I’m reticent to replace all the single calls to the macro by the sequences I wrote.
The order that FreeRTOS operations need to get your various locks to do the operations it does will determine the ordering that all parts of the code must follow using the locks.
Releasing locks does not need to be in reverse of taking, but you can not take (or re-take) a lock that is earlier in the order tree than any lock you currently hold.
FreeRTOS does not IMPLEMENT these various locks but defines the methodology. for taking and releasing and their order.
The port needs to define how to actually implement the locks, and any use it has of these locks needs to follow the FreeRTOS ordering rules.
Looking at task.c, there is a comment where it says that portGET_TASK_LOCK needs to be called before portGET_ISR_LOCK. It also seems that portGET_ISR_LOCK should only be called in an ISR or with the interrupts disabled.
This sequence seems to indicate that the core0 is executing some task and takes ISR lock without first taking task lock. That does not seem correct. Can you check the code sequence executing on core0.
I’m not sure to understand, does it mean that it would be pertinent if my software spinlock is implemented accordingly to the locks used by FreeRTOS ? Currently, my implementation doesn’t have any clue of FreeRTOS, the spinlock can be used without the OS, and it doesn’t care if there is 1, 2, or 3 existing locks.
I’m absolutely sure of what I read yesterday, even if it’s surprising. I don’t know why, but this morning I can’t reproduce it to verify the sequence. Instead, I met another issue: A core is trying to release a lock (it breaks after the STREX, so it’s never released) and the other core is eternally locked in loop trying to take the same lock.
I handled this by modifying the unlock function: Instead of breaking when the STREX fails, it reload the spinlock state (so reset the tag on the address) and try again to unlock. It works ! The port is running without troubles since 10 minutes now !!
but I can’t explain the solution. the lock function never try its STREX: it loop back when it sees the lock is owned by the other core, and do a new LDREX. My interpretation is that doing a LDREX is considered as a modification of the address, but I can’t find any source for it. I looked at the ARM reference manual, and it says that, when 2 cores do a LDREX at the same address, the state is marked as UNKNOWN… So both cores knows the state of the exclusive monitor of the other core ? where is this information in memory ??
The model for the use of a Load-Exclusive/Store-Exclusive instruction pair, accessing a non-aborting memory
address x is:
• The Load-Exclusive instruction reads a value from memory address x.
• The corresponding Store-Exclusive instruction succeeds in writing back to memory address x only if no other
observer, process, or thread has performed a more recent store to address x. The Store-Exclusive operation
returns a status bit that indicates whether the memory write succeeded.
A Load-Exclusive instruction tags a small block of memory for exclusive access.
If your apin locks are being used by FreeRTOs as part of the implementation of the portGET_TASK_LOCK and portGET_ISR_LOCK, then they need to be implemented, and used, according to the rules that FreeRTOS uses, and part of that is that the TaskLock can not be taken when you have the ISR lock, and the ISR lock must only be taken inside a critical section.
I don’t know why, but this morning I can’t reproduce it to verify the sequence. Instead, I met another issue: A core is trying to release a lock (it breaks after the STREX, so it’s never released) and the other core is eternally locked in loop trying to take the same lock.