Using Task Notifications on Windows Port

aiyerk wrote on Tuesday, March 14, 2017:

I am trying to write a small thread abstraction wrapper on top of the freeRTOS implementation and am testing it using the Windows port. I’m running into an issue which could be due to my use of Task Notifications but I’m unable to confirm.

This is my Create Thread Function

static void ThreadLauncher( void* arg )
{
  KThread* pThread = ( KThread* )arg;
  for( ;; ) {
    TaskHandle_t hJoin;
    ulTaskNotifyTake(pdTRUE, portMAX_DELAY );
    pThread->fn( pThread->arg );
    KMutexLock( &pThread->joinMutex, WAIT_FOREVER );
    hJoin = pThread->hJoinRequestTask;
    KMutexUnlock( &pThread->joinMutex );
    if( hJoin ) {
      xTaskNotifyGive( hJoin );
    }
    vTaskDelete(NULL);
  }
}

bool KThreadCreate( KThread* pThread, const KThreadCreateParams* pParams )
{
  bool retval = false;
  int err = 0;
  if ( pThread &&
      pParams &&
      pParams->fn &&
      pParams->pStack &&
      pParams->stackSizeInBytes / sizeof( StackType_t ) > configMINIMAL_STACK_SIZE  ) {
    StackType_t *pStack = ( StackType_t * ) pParams->pStack;
    memset( pThread, 0, sizeof( KThread ) );
    pThread->fn = pParams->fn;
    pThread->arg = pParams->threadArg;
    pThread->isComplete = false;
    pThread->sanity = THREAD_SANITY_CHECK;
    if ( pParams->pThreadName ) {
      strncpy( pThread->threadName, pParams->pThreadName, sizeof( pThread->threadName ));
    } else {
      memset( pThread->threadName, 0, sizeof( pThread->threadName ));
    }
    if ( KMutexCreate( &pThread->joinMutex, NULL ) ) {
      pThread->hTask = xTaskCreateStatic( ThreadLauncher,
                                          pThread->threadName,
                                          pParams->stackSizeInBytes / sizeof( StackType_t ),
                                          pThread,
                                          pParams->threadPriority,
                                          pStack,
                                          &pThread->task );
      if ( pThread->hTask ) {
        xTaskNotifyGive( pThread->hTask );
        retval = true;
      }
    }
  }
  return retval;
}

The thread also creates a companion mutex which I’m using a part of my implementation to join. Since I’m storing the handle to the thread returned by xTaskCreateStatic (which is use in my wrapped ThreadLauncher function) I want to make sure that the created thread doesn’t really start running till pThread->hTask is properly set. Of
course this would mean a context switch when the created thread is of higher priority than the creating thread.

I try to accomplish the wait by

  1. Having the created thread wait on Notification flag using xTaskNotifyTake( pdTRUE, portMAX_DELAY );
  2. Give this notification in the creating thread using xTaskNotifyGive( pThread->hTask )

** I run into a problem when the creating thread is of higher priority. For some reason, the xTaskNotifyTake() completes even though the creating thread has not yet called xTaskNotifyGive(). The High priority thread runs to completion and when the lower priority (creating) thread resumes it gets blocked in the call to xTaskNotifyGive() specifically on the ReleaseMutex() call in the implementation to vPortExitCritical()**

I could use some help trying to figure out if what I did was wrong. I wrote a unit test which I’m running with embedded unit to test the problem ( the unit test runner itself runs in the freeRTOS task of the lowest priority, it is the first thread created before the task scheduler is launched) .

#include <embUnit/embUnit.h>
#include <ThreadInterface.h>
#include <MutexInterface.h>

typedef struct _ThreadTest1Data
{
  uint8_t value;
  KMutex mtx;
} Test1Data;

static Test1Data s_tst1 = { 0 };

static void TestThreadFunction( void *arg )
{
 // usleep( 1000000 );
  s_tst1.value = 1;
}

static void SetUp(void)
{

}
static void TearDown(void)
{

}

typedef struct _ThreadData
{
  KThread thread;
  uint8_t stack[ 1024 ];
}ThreadData;


static ThreadData s_testThreadA;
static void ThreadApiTest(void)
{
  KTHREAD_CREATE_PARAMS( s_testThreadAParams, 
                         "Test Thread", 
                         TestThreadFunction, 
                         NULL, 
                         s_testThreadA.stack, 
                         sizeof( s_testThreadA.stack ), 
                         SEMANTIC_THREAD_PRIORITY_MID );
  bool retval = KThreadCreate( &s_testThreadA.thread, KTHREAD_PARAMS( s_testThreadAParams ) );
  TEST_ASSERT( retval );
  retval = KThreadJoin( &s_testThreadA.thread );
  TEST_ASSERT( retval );
  TEST_ASSERT( s_tst1.value == 1 );
  retval = KThreadDelete( &s_testThreadA.thread );
  TEST_ASSERT( retval );
}

static void PremptTestThreadHi( void *arg )
{
  KMutexLock( &s_tst1.mtx, WAIT_FOREVER);
  s_tst1.value = 20;
  KMutexUnlock( &s_tst1.mtx );
}

static void PremptTestThreadMid( void *arg )
{
  uint32_t val = 0;
  bool keepRunning = true;
  while( keepRunning ) {
    KMutexLock( &s_tst1.mtx, WAIT_FOREVER);
    val = s_tst1.value;
    KMutexUnlock( &s_tst1.mtx );
    keepRunning = ( val == 0 ) ? true : false;
  }
}

static ThreadData s_testThreadMid;
static ThreadData s_testThreadHi;
static void TestBasicPremption( void )
{
  memset( &s_tst1, 0, sizeof( s_tst1 ));
  if( KMutexCreate( &s_tst1.mtx, "ThreadPremptTestMtx" )) {
    s_tst1.value = 0;
    KTHREAD_CREATE_PARAMS( threadParamsMid,
                           "ThreadPremptTestMid",
                           PremptTestThreadMid,
                           NULL,
                           s_testThreadMid.stack,
                           sizeof( s_testThreadMid.stack ),
                           SEMANTIC_THREAD_PRIORITY_LOWEST);
    KTHREAD_CREATE_PARAMS( threadParamsHi,
                           "ThreadPremptTestHi",
                           PremptTestThreadHi,
                           NULL,
                           s_testThreadMid.stack,
                           sizeof( s_testThreadMid.stack ),
                           SEMANTIC_THREAD_PRIORITY_HIGH );
    if( KThreadCreate( &s_testThreadMid.thread, KTHREAD_PARAMS( threadParamsMid ) )) {
      if( KThreadCreate( &s_testThreadHi.thread, KTHREAD_PARAMS( threadParamsHi ) )) {
        TEST_ASSERT( KThreadJoin( &s_testThreadMid.thread ) );
        TEST_ASSERT( KThreadJoin( &s_testThreadHi.thread ) );
        TEST_ASSERT( KThreadDelete( &s_testThreadMid.thread ) );
        TEST_ASSERT( KThreadDelete( &s_testThreadHi.thread ) );
      }
    }
    KMutexDelete( &s_tst1.mtx );
  }
}

TestRef KThreadTest_ApiTests()
{
  EMB_UNIT_TESTFIXTURES(fixtures) {
    new_TestFixture( "TreadApiTest", ThreadApiTest ),
    new_TestFixture( "BasicPremption", TestBasicPremption )

  };
  EMB_UNIT_TESTCALLER( KThreadBasic, "KThreadBasic", SetUp, TearDown, fixtures );
  return (TestRef)&KThreadBasic;
}

I don’t think it matters but here is the rest of the implementation of the Thread Wrapper functions

bool KThreadJoin( KThread* pThread )
{
  bool retval = false;
  if( xTaskGetCurrentTaskHandle() != pThread->hTask ) {
    if ( eTaskGetState( pThread->hTask ) != eDeleted ) {
      //Wait to be notified when deleted
      KMutexLock( &pThread->joinMutex, WAIT_FOREVER );
      if( !pThread->hJoinRequestTask ){
        pThread->hJoinRequestTask = xTaskGetCurrentTaskHandle();
        retval = true;
      }
      KMutexUnlock( &pThread->joinMutex );
      if( retval ) ulTaskNotifyTake(pdTRUE, portMAX_DELAY );
    } else {
      retval = true;
    }
  }
  return retval;
}

bool KThreadDelete( KThread* pThread )
{
  bool retval = false;
  if( pThread ) {
    KThreadJoin( pThread );
    pThread->hTask = 0;
    pThread->sanity = 0;

    KMutexDelete( &pThread->joinMutex );
    pThread->hJoinRequestTask = 0;
    retval = true;
  }
  return retval;
}

int32_t KThreadGetPriority( KThread* pThread )
{
  return (int32_t)uxTaskPriorityGet( pThread->hTask );
}

const char* KThreadGetName( KThread* pThread )
{
  return pThread->threadName;
}

void vAssertCalled( unsigned long ulLine, const char * const pcFileName )
{
  volatile uint32_t ulSetToNonZeroInDebuggerToContinue = 0;

  /* Called if an assertion passed to configASSERT() fails.  See
  http://www.freertos.org/a00110.html#configASSERT for more information. */

  /* Parameters are not used. */
  ( void )ulLine;
  ( void )pcFileName;

  printf( "ASSERT! Line %d, file %s\r\n", ulLine, pcFileName );

  taskENTER_CRITICAL();
  {
    /* You can step out of this function to debug the assertion by using
    the debugger to set ulSetToNonZeroInDebuggerToContinue to a non-zero
    value. */
    while ( ulSetToNonZeroInDebuggerToContinue == 0 ) {
      __asm { NOP };
      __asm { NOP };
    }
  }
  taskEXIT_CRITICAL();
}

/* configUSE_STATIC_ALLOCATION is set to 1, so the application must provide an
implementation of vApplicationGetIdleTaskMemory() to provide the memory that is
used by the Idle task. */
void vApplicationGetIdleTaskMemory( StaticTask_t **ppxIdleTaskTCBBuffer, StackType_t **ppxIdleTaskStackBuffer, uint32_t *pulIdleTaskStackSize )
{
  /* If the buffers to be provided to the Idle task are declared inside this
  function then they must be declared static - otherwise they will be allocated on
  the stack and so not exists after this function exits. */
  static StaticTask_t xIdleTaskTCB;
  static StackType_t uxIdleTaskStack[ configMINIMAL_STACK_SIZE ];

  /* Pass out a pointer to the StaticTask_t structure in which the Idle task's
  state will be stored. */
  *ppxIdleTaskTCBBuffer = &xIdleTaskTCB;

  /* Pass out the array that will be used as the Idle task's stack. */
  *ppxIdleTaskStackBuffer = uxIdleTaskStack;

  /* Pass out the size of the array pointed to by *ppxIdleTaskStackBuffer.
  Note that, as the array is necessarily of type StackType_t,
  configMINIMAL_STACK_SIZE is specified in words, not bytes. */
  *pulIdleTaskStackSize = configMINIMAL_STACK_SIZE;
}
/*-----------------------------------------------------------*/

/* configUSE_STATIC_ALLOCATION and configUSE_TIMERS are both set to 1, so the
application must provide an implementation of vApplicationGetTimerTaskMemory()
to provide the memory that is used by the Timer service task. */
void vApplicationGetTimerTaskMemory( StaticTask_t **ppxTimerTaskTCBBuffer, StackType_t **ppxTimerTaskStackBuffer, uint32_t *pulTimerTaskStackSize )
{
  /* If the buffers to be provided to the Timer task are declared inside this
  function then they must be declared static - otherwise they will be allocated on
  the stack and so not exists after this function exits. */
  static StaticTask_t xTimerTaskTCB;
  static StackType_t uxTimerTaskStack[ configTIMER_TASK_STACK_DEPTH ];

  /* Pass out a pointer to the StaticTask_t structure in which the Timer
  task's state will be stored. */
  *ppxTimerTaskTCBBuffer = &xTimerTaskTCB;

  /* Pass out the array that will be used as the Timer task's stack. */
  *ppxTimerTaskStackBuffer = uxTimerTaskStack;

  /* Pass out the size of the array pointed to by *ppxTimerTaskStackBuffer.
  Note that, as the array is necessarily of type StackType_t,
  configMINIMAL_STACK_SIZE is specified in words, not bytes. */
  *pulTimerTaskStackSize = configTIMER_TASK_STACK_DEPTH;
}

void KThreadInit()
{
  /* Start the scheduler so the demo tasks start to execute. */
  vTaskStartScheduler();

  /* vTaskStartScheduler() would only return if RAM required by the Idle and
  Timer tasks could not be allocated.  As this demo uses statically allocated
  RAM only, there are no allocations that could fail, and
  vTaskStartScheduler() cannot return - so there is no need to put the normal
  infinite loop after the call to vTaskStartScheduler(). */
}

This is my unit test launcher

#include<embUnit.h>
#include <ThreadInterface.h>

extern TestRef PoolTest_ApiTests();
extern TestRef KThreadTest_ApiTests();
extern TestRef PriorityWakeTest();
extern TestRef PriorityDonateChainTest();

#define TEST_RUNNER_STACK_SIZE          ( 1 << 14 )
static uint8_t s_testRunnerThreadStack[ TEST_RUNNER_STACK_SIZE ];
static KThread s_testRunnerThread;

static void TestRunner( void* arg )
{
  TestRunner_start();
  {
    TestRunner_runTest( PoolTest_ApiTests() );
    TestRunner_runTest( KThreadTest_ApiTests() );
    TestRunner_runTest( PriorityWakeTest() );
    //TestRunner_runTest( PriorityDonateChainTest() );
  }
  TestRunner_end();
  for ( ; ; );
}
int main (int argc, const char* argv[])
{
  KThreadCreateParams param = {
    .pThreadName = "TestRunnerThread",
    .threadPriority = SEMANTIC_THREAD_PRIORITY_LOWEST,
    .threadArg = NULL,
    .pStack = s_testRunnerThreadStack,
    .stackSizeInBytes = TEST_RUNNER_STACK_SIZE,
    .fn = TestRunner
  };
  KThreadCreate( &s_testRunnerThread, &param );
  KThreadInit();
}

The complete code with build scripts can be found here in case it is useful.

Again thanks a lot for the help

rtel wrote on Tuesday, March 14, 2017:

Are you using any Windows system calls in any of your wrappers? Or just
sticking to the FreeRTOS API within the wrappers? If the former then
that could be the issue - just stick with the FreeRTOS API even though
its running on Windows (I haven’t read through all the code).

heinbali01 wrote on Wednesday, March 15, 2017:

Hi Kartik,

I’d like to confirm that the Windows simulation of the scheduler is very realistic. It was tested with the task-notify functionality, all queue-types, mutexes and semaphores, and also with the event-groups.

The usage and behaviour of xTaskNotifyGive() / ulTaskNotifyTake() is exactly the same as when your code is running on an embedded system.

Richard is (as usually) completely right: thou shall not call any (Windows) system API from a FreeRTOS task. Think of stdio and the Windows thread- and process-related functions. It may happen then that two tasks will start running simultaneously.

Suppose a task calls printf("Hello world\n"). That function will call a Windows API and the API-call will put the thread asleep. The scheduler will start another task and while that runs, the API may return. At that moment, two tasks will be active.

At least in your published code, I don’t see Window API calls.

I extended your test function as:

static void TestThreadFunction( void *pvArgument )
{
BaseType_t xIndex;
SemaphoreHandle_t xSemaphore;

	s_tst1.value = 1;
	xSemaphore = xSemaphoreCreateBinary();
	vLoggingPrintf( "TestThreadFunction started\n" );
	for( xIndex = 1; xIndex < 4; xIndex++ )
	{
		xSemaphoreTake (xSemaphore, 1000);
		vLoggingPrintf( "TestThreadFunction: %d\n", xIndex );
	}
	vSemaphoreDelete( xSemaphore );
}

just to see some sign of live.

vLoggingPrintf() sends logging to stdout: this is being done from within a real Windows thread.
The communication between a FreeRTOS task and a Windows thread is very limited: threads know nothing about semaphores, while tasks are not supposed to use Windows mutexes.

I ran a bit of your code and what I saw happen is the following:

  1. The creating task has a higher priority:
Time Message
----------------------------------------
 147 [Main-task] xTaskCreateStatic()
 147 [Main-task] xTaskNotifyGive start
 147 [Main-task] xTaskNotifyGive end
 147 [Main-task] KThreadJoin start
 147 [Test Thread] ulTaskNotifyTake start
 147 [Test Thread] ulTaskNotifyTake end
 147 [Test Thread] TestThreadFunction started
1147 [Test Thread] TestThreadFunction: 1
2147 [Test Thread] TestThreadFunction: 2
3147 [Test Thread] TestThreadFunction: 3
3147 [Main-task] KThreadJoin end
3147 [Main-task] KThreadDelete start
3147 [Main-task] KThreadDelete end
3147 [Test Thread] Deleting task
  1. The creating task has a lower priority:
Time Message
----------------------------------------
 153 [Main-task] xTaskCreateStatic()
 153 [Test Thread] ulTaskNotifyTake start
 153 [Main-task] xTaskNotifyGive start
 153 [Test Thread] ulTaskNotifyTake end
 153 [Test Thread] TestThreadFunction started
 153 [Main-task] xTaskNotifyGive end
 153 [Main-task] KThreadJoin start
1153 [Test Thread] TestThreadFunction: 1
2153 [Test Thread] TestThreadFunction: 2
3153 [Test Thread] TestThreadFunction: 3
3153 [Test Thread] Deleting task
3154 [Main-task] KThreadJoin end
3154 [Main-task] KThreadDelete start
3154 [Main-task] KThreadDelete end

Please realise that if you xTaskCreate() a low-priority task, that task will not run until the creator goes in a blocking state. This only happens at KThreadJoin().

If you xTaskCreate() a high-priority task, that new task will run immediately, and xTaskCreate() will only return when the new task goes in a bocking state, which happens at ulTaskNotifyTake().

As far as I can see there is nothing wrong with the Windows implementation of the task-notify functions. Regards.

aiyerk wrote on Friday, March 17, 2017:

Hi Hein,

Thank you very much for getting back to me and for debugging some of my work. I’m not directly making any windows API calls but I will check the unit test runner code and make changes if needed. Also thanks a lot for pointing out the problem with the creation of threads in my test. I will work out the kinks try again and will update this thread.

Thanks again.
Kartik

aiyerk wrote on Sunday, March 26, 2017:

Hi Hein,

Sorry to be asking about this again. I have not been able to find an implementation in the source code of FreeRTOS v9.0.0 for vLoggingPrintf()

Can you please point me to the implementation.

Thanks
Kartik

aiyerk wrote on Sunday, March 26, 2017:

I think I found the code, please ignore my previous message.

heinbali01 wrote on Sunday, March 26, 2017:

Hi Kartik,

Just to make sure you find the right one, it’s in the FreeRTOS/Labs download

FreeRTOS-Plus/Demo/FreeRTOS_Plus_TCP_and_FAT_Windows_Simulator/demo_logging.c

Here you can find a short description of vLoggingPrintf().