Passing elements to task for writing SD Card

xanderwar wrote on Friday, December 05, 2014:

Hi all,

I’m working on a project with FreeRTOS including FATFs.
I’m currently with two tasks (but soon increasing to more): one that reads some values from a sensor connected to the microcontroller (EFM32LG) and sends this values to another task, which takes care of writing them to an SD Card.
To make this writing process the fastest possible, I’m writing in the SDcard every 512 bytes of data I have (sector size). The buffers containing this data must be normal arrays of bytes. The sensor puts out data with different length in bytes, at a certain frequency, and each type of data (temperature, acceleration…) has its own file (one buffer for each file).
The question I’m facing is, I want to have few buffers so I don’t occupy much memory, but also avoid a lot of copy operations in and out of queues. Which would be the best trade-off?

Should I simply pass values between tasks with a queue for each sensor output type and save them in buffers till I reach the sector size for each? Or should I use semaphores with ring buffers (but then having to copy again from this ring buffer to the normal buffer)?

The queues seem like the best option, but since I have different files/data length, I’m afraid I won’t be able to have just one task to take care of writing the files, since it can block in one queue while others are waiting…

Is it a good practice to keep a binary semaphore and just use the queues as ring buffers? This way I avoid blocking on a specific queue, since I know I’m expecting a certain amount data to be available (as we read different data, but from the same source and frequency).

rtel wrote on Friday, December 05, 2014:

If you only have one writer controlling a “write to” index and one reader controlling a “read from” index then you may not require a queue at all, a simple circular buffer could be used. With a bit of twiddling you could do this writing directly into the 512K buffer, so the producer fills the buffer as it it were any other circular RAM buffer, and once full the consumer passes the buffer to the file system with no extra copying (it must give the producer a new buffer to fill in the mean time of course).

If you have more than one task writing into the same buffer it become more complex as you would need to protect it from simultaneous access.

The FreeRTOS+TCP code (download link) uses some files called FreeRTOS_Stream_Buffer.c/h that does just this when reading and writing TCP data out of streams (hence the name).


heinbali01 wrote on Friday, December 05, 2014:

Hi Alexandre,

In case you’d like to use the mentioned circular buffer, here below you find an example along with some explanation:

    static xStreamBuffer *pxBuffer = NULL;

    void vCreateBuffer( )
        /* Declare a pointer to a circular buffer. */
        /* The maximum number of bytes it must be able to store. */
        const uint32_t ulLength = 1024;
        /* The number of bytes to be allocated. */
        uint32_t ulSize;

        /* when HEAD == TAIL, the buffer could either be 100% full or empty.
        In this implementation, 'HEAD == TAIL' means empty.
        For this reason, allocate 1 extra byte. */
        ulSize = sizeof( *pxBuffer ) - sizeof( pxBuffer->ucArray ) + ulLength + 1;

        pxBuffer = ( xStreamBuffer * )pvPortMalloc( ulSize );

        if( pxBuffer != NULL )
            /* Clear the markers of the stream */
            memset( pxBuffer, '\0', sizeof( *pxBuffer ) - sizeof( pxBuffer->ucArray ) );
            /* Note that 'LENGTH' is a const member. It indicates the allocated
            size of the ucArray[]. */
            pxBuffer->LENGTH = ( int32_t ) ( ulLength + 1 );
            /* After this, memset() should not be used because LENGTH would get
            erased as well */
            /* To clear a buffer, use: */
            vStreamBufferClear( pxBuffer );

    void vDataInterrupt( )
        BaseType_t xWoken = pdFALSE;
        int32_t lByteCount;
        unsigned char *pucBuffer;

        lByteCount = dma_get( &pucBuffer );

        /* Check for space and add bytes */
        if( ( lByteCount > 0 ) && ( lStreamBufferGetSpace( pxBuffer ) >= lByteCount ) )
            lByteCount = lStreamBufferAdd( pxBuffer, 0, pucBuffer, lByteCount );
            /* Wake-up the task which takes the data from the buffer */
            xSemaphoreGiveFromISR( xDataSemaphore, &xWoken );
        portEND_SWITCHING_ISR( xWoken );

    void vMyTask( void *pvParams )
        vCreateBuffer( );
        configASSERT( pxBuffer != NULL );

        for( ; ; )
            xSemaphoreTake( xDataSemaphore, 100 )
            if( lStreamBufferGetSize( pxBuffer ) > 0 )
                lByteCount = lStreamBufferGet( pxBuffer, 0, pucBuffer,
                    sizeof pucBuffer, pdFALSE );

The idea is as follows:

- Task-A (in this example an interrupt) is filling a buffer.
  It may only touch the volatile field 'lHead'.

  - Task-B (the task vMyTask()) is emptying the same buffer.
  It may only touch and change the volatile field 'lTail'.

Access functions like:


do not need a critical section as long as there is only one instance of Task-A and one instance of Task-B.

Hope that this explanation is a bit clear…


richard_damon wrote on Friday, December 05, 2014:

If each sensor is buffering up for a seperate write operation, then by definition you are going to need at least a 512 byte buffer for each sensor. You probably need some extra buffers so a sensor can continue collecting data while its buffer is being written (how many extras may be a bit hard to compute, it depends on how many sensors can fill their buffers in the time it takes to write the buffers).

An upper bound on the buffers would be two buffers per sensor.

If each sensor has its own buffer, then you just need to pass the buffer addresses and NEVER need to copy data. When a sensor fills its buffer, it queues a message with the data type and buffer address, and sends it to the writer task, then gets a new free buffer to continue working.

If this uses too much memory, you may need to change how you organize your data flow. Perhaps write a single stream of merged data, and elsewhere (either another task in the unit, or external to the unit) have something that splits this stream into its component files.

If you use another task, then you just need to keep track of where in your “temp” file its data starts, and how much data you have queue up for it. When a sensor aquires a full sector of data, you scan the temp file from the marked point, extract the saved data, and write out the seperated data. This will require that you have ample SD Memory bandwidth.

xanderwar wrote on Friday, December 05, 2014:

Thank you for the quick responses!
I think I’m on my way to find my solution.
I have only one task writing to the buffers. I may need to use the current and latest sample data, but since it’s just a couple of samples, I can pass them from the producer directly to another task, if required.
But taking in account these stream buffers, I noticed from the code, that I still need to copy into them and later read from them into the 512 bytes buffers to be passed to the file system. So… wouldn’t it be the same using queues (except that with the stream buffers I can ask directly for the 512, while with queues I need to call the function 512 times)?

I guess memory is not a problem, so maybe I opt for the 2 buffers for each sensor data. Another solution that seems to work is to have the single stream of data and parsing it right before writing in the SD Card.
Just to make it clearer: I have only one sensor, but providing different data, therefore I’m buffering for separate write operations!

xanderwar wrote on Thursday, December 11, 2014:

Hi guys,

just to let you know that I’ve adopted the stream buffer implementation you suggested, using the code from the FreeRTOS+TCP and the function code to initialize / create the buffer, provided in one of the posts in this topic.

I would suggest to maybe include this buffer implementation as circular/ring buffer in the normal FreeRTOS package, for people who require one, just like me, since a lot of applications of RTOS require such resources. It works very well and solved some problems I was having with another implementation of a buffer, also with another project.
Just my two cents!