Notification not received after wake from low power

Kernel
#1

I am rather new to FreeRTOS…

I have an embedded application running multiple tasks. One for the display, one for sensors, one for buttons, a main task that coordinates between buttons and the display (and a few other things not yet implemented) and one for a long calculation (the calculation is broken up into short chunks).

The application uses NXP’s FreeRTOS version (Amazon) and it’s running on an LPC54102 (Arm 4) processor.

I have implemented a low power mode that is entered by holding a button. When this is detected, I send a “SLEEP” notification to all tasks. The tasks do any housekeeping required to enter the low power mode. This includes powering down sensors, the display, etc. All the tasks uses xTaskNotifyWait to get the notifications. All notifications are sent with xTaskNotify with eSetBits.

The way NXP (it may the same in other ports, I don’t have the experience to know) implements low power is in the idle task:

			/* Sleep until something happens.  configPRE_SLEEP_PROCESSING() can
			set its parameter to 0 to indicate that its implementation contains
			its own wait for interrupt or wait for event instruction, and so wfi
			should not be executed again.  However, the original expected idle
			time variable must remain unmodified, so a copy is taken. */
			xModifiableIdleTime = xExpectedIdleTime;
			configPRE_SLEEP_PROCESSING( xModifiableIdleTime );
			if( xModifiableIdleTime > 0 )
			{
				__asm volatile( "dsb" ::: "memory" );
				__asm volatile( "wfi" );
				__asm volatile( "isb" );
			}
			configPOST_SLEEP_PROCESSING( xExpectedIdleTime );

The two macros config*_SLEEP_PROCESSING are defined by me in FreeRTOSConfig.h and they call functions that configure for deep sleep if required. In that case, I enable a wakeup source, call an SDK function to enable a Power Down mode and set xModifiableIdleTime is set to zero (so that the above doesn’t do anything). If power down is not required, the macros set xModifiableIdleTime to 1 and the processor enters a low power wait state (Sleep) as it should.

My assumption is that with the SLEEP notifications sent to all the tasks, the system doesn’t enter idle until all the tasks process the notifications. Is this a safe assumption? It appears to work as all power down code executes giving us a power down current of ~25 uA.

On processor wake, I send notifications to all tasks to WAKE which causes the tasks to reconfigure their associated hardware from power down to running (powering up the display, sensors, etc).

I do not stop/delete any tasks.

The problem I’m having is that the occasionally (1 out of 20 to 30 times) on wake, one task does not respond to the notification. The notification is sent and I have traced what I think is a symptom of the problem in the function xTaskGenericNotify(). The first part of this function does find the task, and set the pxTCB->ulNotifiedValue to the notification sent. The second part of this function has an IF statement:

			/* If the task is in the blocked state specifically to wait for a
			notification then unblock it now. */
			if( ucOriginalNotifyState == taskWAITING_NOTIFICATION )
			{
				...
			}

When the task doesn’t respond to the notification, this IF statement is not true and the code is never entered as if the task isn’t waiting for notifications, but I think it is - or last should be.

This task that doesn’t get the notification does have one interrupt that is from a timer - it sends a notification to take measurements. This timer is stopped during power down and is not active when the WAKE notification is sent.

I have gone through the FAQ and I don’t think I’m doing any of the things it says not to do. Unfortunately, I am limited on what code I can post.

Thanks!

#2

I am trying to find someone internally who has an NXP setup or has recent exposure to this subsystem.

#3

First the normal stuff: Do you have configASSERT() defined and stack overflow checking set to 2? Those two things catch most issues. Assuming that doesn’t help, then:

If I understand correctly, there are two methods of entering low power. First is handled by the SDK and enters deep sleep. Second is handled by the code you post. Does the issue only occur after a deep sleep (I think this is the case, but want to check)? Please post the part of the SDK source code that enters the deep sleep, and what it executes afterwards too.

Are you able to trap that with a breakpoint in the debugger so we can see what ucOriginalNotifyState is set to? Also, do you have a thread aware debugger, so we can see where the task is if its not waiting for the notification?

#4

configASSERT is defined. I had set configCHECK_FOR_STACK_OVERFLOW to 1. I’ll set it to 2 and try again.

The sleep mode handled by the SDK is called “SLEEP” and it’s essentially a lower power mode that turns off the main clock. All peripherals are clocked. To me, it’s really more of an idle mode that is used all the time when the application is running. Current draw is probably half of normal run mode. SLEEP is the term used in the LPC54102 reference manual. The deep sleep and power down currents are 5-6 orders of magnitude less than normal run mode.

I think the answer to both of those is yes.

I’ve set configCHECK_FOR_STACK_OVERFLOW to 2.

When the error occurs, usOriginalNotifyState is 0 (taskNOT_WAITING_NOTIFICATION).

#5

from my understanding, there will always be a time where a task is NOT waiting for a notification, otherwise that task would not do anything.

I am not familiar enough with notificatins, but it makes intuitive sense to me to only notify tasks that are currently waiting for a notification.

You may want to use some other syncronization mechanism such as a semaphore with count n for n tasks that need to react to your global sleep notification. Some more bookkeeping may be required to ensure tbat no task catches the semaphore twice.

#6

The task notification is latched, so if the task is not waiting for it when it arrives, it won’t block the next time it looks for it (because it’s already there, just like a semaphore). If the task isn’t waiting for the notification then it must be executing something else. Obviously I’ve not seen the code to see all options, but broadly speaking, either the task never got to that point in the code before the processor entered SLEEP mode, or it had already moved past that point in the code before the processor exited SLEEP mode. The latter could happen if the notification bit was already set from the previous run through that function.

#7

That certainly makes sense. Just not sure how to stop if from happening. It’s odd that it only happens with one task.

Was my assumption that the idle task does not get processor time if other tasks have notifications not correct?

Thanks.

#8

That depends on their priority. Tasks that run at the idle priority will share time with the idle task. Tasks that have a higher priority will always run before the idle task.

#9

If you are sending the notifications from the Idle task, if you notify a task that will run for a while, that will delay the Idle task from notifying other tasks.

You may want to make sure you notify tasks in order from highest to lowest or disable the scheduler while you notify them to avoid getting trapped in a priority inversion.

#10

Thanks, I will give that a try.

Is there a better way to do this kind of thing? I can’t be the only one try to have tasks do something after waking from a low power mode.

#11

I double checked all the priorities. The Idle task is running at 0, all of the tasks I create are at 3. I tried wrapping the notification calls in the idle task sleep function in vTaskSuspendAll() and xTaskResumeAll() and it didn’t appear to make any difference.

#12

configPRE_SLEEP_PROCESSING() should either set xModifiableIdleTime to zero or leave it alone. Setting it to one is not correct and will cause the MCU to wake and go back to sleep once per tick period (eg, every millisecond).
(EDIT: Correction - setting it to one has no effect.)

No, that’s not safe. While handling some notification, a task may block waiting for something – vTaskDelay() for example – and FreeRTOS will still attempt tickless idle during that time, even though one of your tasks is not ready for deep sleep (and is ostensibly working on it). As others have pointed out, you need to add some positive feedback from your tasks that they are ready for deep sleep.

A question for you. When the error occurs, is the “one task” that doesn’t get the wake-up notification always the same task?

#13

Thanks, I will make sure that is what is happening.

I think you hit the nail on the head here. The sensors task is blocking on an SPI transfer, I think it is going to sleep during that call. I’m going to implement some something to allow the sleep code to know it’s safe for everything to go to sleep.

It would seem like a counting semaphore would be the way to go (only sleep when count is zero). Each task takes the semaphore when it can’t sleep.

Yes. This task probably does more blocking reads (I2C and SPI) than any other task.

#14

One key thing is if your task is blocking on an interrupt that should be happening soon, DON’T block for portMAX_DELAY, but block for just slightly longer than the expected period (perhaps checking the result and reblocking if going a bit to slow).

The sleep code is looking at the time the next activity is “scheduled”, and uses block times as its guide. If actions are happening that will (or will likely) interrupt shortly, it needs to know about that. Either use short timeouts for these, or you need to set a flag to say activity is happening, don’t go to sleep, and incorporate that into the sleep logic.

#15

This makes sense. By the way, is this SPI transfer blocking on a task notification? If so, it should use a dedicated task notification index. Using the same notification index as the blocking for the “wake” command would cause you to lose the “wake” notification if it arrives while you are waiting for the SPI transfer. If I understand correctly this is the problem you originally started looking for – missing the notification. Precisely speaking, you lose the “notified” state, but you don’t lose the bit set in the notification value. Task notifications have a separate state and value.

Task notification indexes are relatively new. If you are not using them explicitly, then you are using index 0 for everything.

#16

I am using zero for everything.

The SPI (and I2C) transfers are part of the NXP SDK that is made to work with their port of FreeRTOS. It uses a semaphore that is released in an interrupt. It takes the semaphore, starts the transfer and then blocks on xSemaphoreTake(portMAX_DELAY). The interrupt at the end of the transfer calls xSemaphoreGiveFromISR().

#17

OK that’s safe. My own code still uses semaphores for SPI/I2C completion.

Maybe you weren’t ever really losing the notified status then, but entering deep sleep in the middle of an SPI/I2C transfer caused that task simply to hang permanently, waiting for the SPI/I2C transfer to finish after wake up. But the transfer was aborted by the use of deep sleep.

#18

Just becaue it is provided by the Chip Manufacturer, doesn’t mean it is written weil.

I have long found that it is frequently worthwhile to copy out their drivers, review them and write my own (based on them). I tend to make an outer API that is largely processor agnostic, so basic I/O tasks aren’t tied to a given processor.

That portMAX_DELAY says that if something goes wrong, your system just hangs, and the system has no idea if it is actually busy doing something without a lot of explicit code.

#19

No kidding. I haven’t been real impressed with NXP’s coding. Unfortunately, I don’t have the time to rewrite all their code.

#20

That’s what I was thinking.

Every task I create looks like this (at the top):

	/*
	 *
	 */
	while (true)
	{
		/*
		 * Wait for notification
		 */
		port_sleep_dec_ref_count();
		xTaskNotifyWait(0x0, 0xFFFFFFFF, &notifications, portMAX_DELAY);
		port_sleep_inc_ref_count();

I added the port_sleep_dec_ref_count() and port_sleep_inc_ref_count() that look like this:

/*
 * See header file for documentation
 */
void port_sleep_inc_ref_count(void)
{
	xSemaphoreTake(port_sleep_count, 0);
}

/*
 * See header file for documentation
 */
void port_sleep_dec_ref_count(void)
{
	xSemaphoreGive(port_sleep_count);
}

Where the port_sleep_count semaphore was created using xQueueCreateCountingSemaphore().

This way, it only sleeps when all of my tasks are waiting for a notification (it’s a gross solution, but should work, I would think).

It still fails. The one task still doesn’t receive the wake notification.