hiccup when writting SD card, freeRTOS, fatfs, ZYNQ platform

system · May 30, 2017, 6:35am

gsing wrote on Tuesday, May 30, 2017:

Hi,
I am new to FreeRTOS, now I am working on FreeRTOS on Xilinx zynq platform, running 160919_FreeRTOS_Labs. I want to test the performance of SD card writing. SD card formated fat32 with windows, it is a 4G card, testing with windows, the write speed is about 5.5 MB/sec. Testing with ftp uploading, the speed is 1.59 MB/sec. testing with command "copy ", actual function is prvCOPYCommand, the speed is very slow, checking the code, I modify the buffer to 32K byte, the speed get improved(about 300 KB/sec), check the log, the problem is we will see that writing stop for about 10 sec, then continue. please see below log.
Does anyone see this? What I can do to improve this hiccup? to improve the writing performance. Thanks in advance for any suggestion!

each line of log means write 32K byte data, the second data is total data have written.
56.549.595 [UDP CLI ] read byte this time lBytesToRead = 32768, read total byte, lBytesRead = 3997696
56.572.595 [UDP CLI ] read byte this time lBytesToRead = 32768, read total byte, lBytesRead = 4030464
56.594.596 [UDP CLI ] read byte this time lBytesToRead = 32768, read total byte, lBytesRead = 4063232
56.617.595 [UDP CLI ] read byte this time lBytesToRead = 32768, read total byte, lBytesRead = 4096000

//why here has 10 seconds delay???

66.641.627 [UDP CLI ] read byte this time lBytesToRead = 32768, read total byte, lBytesRead = 4128768
66.663.627 [UDP CLI ] read byte this time lBytesToRead = 32768, read total byte, lBytesRead = 4161536

below is the chart for overview of writting time interval

rtel · May 30, 2017, 3:11pm

rtel wrote on Tuesday, May 30, 2017:

Looks like something is timing out somewhere - do you have any timeouts
set to 10 seconds?

system · May 30, 2017, 3:46pm

gsing wrote on Tuesday, May 30, 2017:

Thanks so much for the great help!
Here you point out timeouts, do you means in this 10 seconds, some higher priority tasks override the CLI task(default is tskIDLE_PRIORITY), so this copy functions were blocked for 10 seconds? I just increase the CLI priority to (tskIDLE_PRIORITY+2),** this hiccup gone**.
For the 10s timeout, I need to double check the code to confirm. But actually I want to store some stream data in SD card, this storing definitely will not run in IDLE priority. so here my problem have been fixed. Thanks again for the help!

By the way, May I ask why there is a gap between windows writing and FreeRtos writing? Windows can rich 5.5MB/s, while FreeRtos fatfs is about 1.7MB/s. What is the bottleneck? CPU? using a better performance SD card?

rtel · May 30, 2017, 3:59pm

rtel wrote on Tuesday, May 30, 2017:

By the way, May I ask why there is a gap between windows writing and
FreeRtos writing? Windows can rich 5.5MB/s, while FreeRtos fatfs is
about 1.7MB/s. What is the bottleneck? CPU? using a better performance
SD card?

Do you mean why can Windows write to an SD card faster than FreeRTOS can?

If so, then probably lots of reasons, like Windows will cache the data
and only commit it to the card in the background. Also Windows will
probably use larger buffers, and the full SD interface, whereas FreeRTOS
will use smaller buffers and an SPI (serial) interface.

system · May 31, 2017, 7:19am

gsing wrote on Wednesday, May 31, 2017:

Thanks for the help! Anyway I will try a higher performance SD card.

htibosch · May 31, 2017, 9:38am

heinbali01 wrote on Wednesday, May 31, 2017:

CLI task( default is tskIDLE_PRIORITY )

Sorry about the confusion. tskIDLE_PRIORITY is often used in FreeRTOS demo’s, but is rarely seen in real life applications.

I just tested the writing speed to an ordinary SDHC-card on my Zynq and have some interesting results.

I used a 512 KB buffer to write, filled with 0xE3E3E3E3.

Writing from a buffer to a new file showed an average writing speed of about 3 MB/sec.
Writing from a buffer to an existing file happens at about 17 MB/sec (!)
Copying a file from card to card would be slower, of course.

Access times may vary because the SD-card has its own software that can hold the BUSY line high for as long as it needs to.
And yes card differ a lot in quality. For best results use genuine cards of a higher class.

What you can do to optimise writing speed is :

Try to write blocks whose length is a multiple of 512 bytes ( sector size )
Try to write at offsets that are a multiple of 512 bytes
Use large buffers if possible
Make sure that the writing task doesn’t get starved because of higher-priority tasks

And know that writing to sectors of an existing file is much faster than writing to a new file.

When you produce short logging messages, it is better to write the data to buffers first and only flush them to disk at certain times.

When using small blocks, whose lengths are not a multiple of 512 bytes, the following option is very beneficial:

    #define ffconfigOPTIMISE_UNALIGNED_ACCESS    1

If gives the file a private “current sector buffer” of 512 bytes.

If your SD-card is only used in your embedded device, you might want to use a single copy of the FAT table:

    #define ffconfigWRITE_BOTH_FATS    0

That’ll also make it faster. It’ll still be compatible, but it’ll miss the FAT backup sectors.

Richard wrote:

and the full SD interface, whereas FreeRTOS will
use … an SPI (serial) interface

If I’m not mistaken the demo activates the full 4-bit SD-card interface. You can check if this function is called:

    s32 XSdPs_Change_BusWidth(XSdPs *InstancePtr)

and what width is actually used. Most probably 4-bits.

Please report your further findings.

htibosch · June 1, 2017, 5:08am

heinbali01 wrote on Thursday, June 01, 2017:

I forgot to mention that the theoretical maximum transfer speed of a normal high-speed SD-card is 25 MB/s. Make sure that XSdPs_Change_BusSpeed() is called.

Reading from a card has very little overhead, and the nett speed should be more than 23 MB/sec.

Writing to a card has a lot of overhead. Free clusters must be found and linked into a chain. The directory must be updated. The FAT table must be synced and the data sectors must be written. That is when the SD-card may use the BUSY signal.

What you can try is a different way of formatting. In ff_format.c around line 317, replace the if-block with :

    if( ( ucFATType == FF_T_FAT32 ) && ( ulSectorCount >= 0x100000UL ) )    /* Larger than 0.5 GB */
    {
    uint32_t ulRemaining;

		/* Skip the first 4MB erase block. */
        ulFATReservedSectors = 8192 - ulHiddenSectors;
        ulNonDataSectors = ulFATReservedSectors + iFAT16RootSectors;

		/* Let the data cluster be well aligned at 64 KB. */
        ulRemaining = (ulNonDataSectors + 2 * ulSectorsPerFAT) % 128;
        if( ulRemaining != 0 )
        {
            /* In order to get ClusterBeginLBA well aligned (on a 128 sector boundary) */
            ulFATReservedSectors += ( 128 - ulRemaining );
            ulNonDataSectors = ulFATReservedSectors + iFAT16RootSectors;
        }
        ulUsableDataSectors = ulSectorCount - ulNonDataSectors - 2 * ulSectorsPerFAT;
        ulUsableDataClusters = ulUsableDataSectors / ulSectorsPerCluster;
    }

The second 4MB block is optimised to hold FAT tables.
Also it is advised to use large clusters to minimise the access to the FAT tables.
If you’re interested in these erase blocks, here is how their sizes can be uncovered.

system · June 1, 2017, 3:52pm

gsing wrote on Thursday, June 01, 2017:

Hi, Hein,
Thanks so much for your so detailed explanation!

Here is the testing I have done:

XSdPs_Change_BusWidth called, so should be 4 bit SD interface
with low performance sandisk 4G card, 32Kbyte buffer, write speed is 1.8MB/sec
if set ffconfigWRITE_BOTH_FATS 0, performance increase from 1.8MB/sec to 2.1MB/sec,
change this setting back to 1 for below testing.
using high performance sandisk card, format cluster=32k, write speed is 2.4MB/sec
using high performance sandisk card, format cluster=64k, write speed is 3.7MB/sec
using high performance sandisk card, testing the read speed too, read speed is about 4.5MB/sec, while write speed is about 3.7MB/s
using your code in ff_format.c, did not see obvious different.
did not test ffconfigOPTIMISE_UNALIGNED_ACCESS, since I just test large data( I tested with a 52MByte file )
I don’t know why, when I change buffer from 32K to 64K, seems fatfs doesn’t work, so did not do the testing for more than 32K buffer. I am not familiar with FAT, don’t understand your code in ff_format.c, just do as you listed, maybe not as you expected.
for you observed, I also see the similar, I use ftp to uploading file, if the file exit, chose overwriting, speed is much higher than new file.

below is high performance sandisk card info, UHS-I card, I did not buy from here, but should be the same kind card:
https://www.amazon.com/SanDisk-Extreme-Memory-Speeds-Ready-SDSDQXP-064G-G46A/dp/B00NUB3530/ref=sr_1_1?ie=UTF8&qid=1496330323&sr=8-1&keywords=SDSDQXP-064G

I tested this card in windows, the writing speed can reach around 80MB/sec.

Summary:

increase cluster, using high performance card is the most help
ffconfigWRITE_BOTH_FATS can also improve the speed.

and one more info, the platform I am using is zybo, CPU clock is 666MHz, not sure if this CPU also a bottle neck for the performance.

BR
Guoxing zhang

htibosch · June 2, 2017, 9:55am

heinbali01 wrote on Friday, June 02, 2017:

Hi Guoxing zhang,

Thanks reporting back. We like to learn.

XSdPs_Change_BusWidth called, so should be 4 bit SD interface

And please also make sure that XSdPs_Change_BusSpeed() is called.

using high performance sandisk card, format cluster=64k, write speed is 3.7MB/sec

64k clusters is what I also use when throughput is more important than space. I’m looking at similar results here

using high performance sandisk card, testing the read speed too, read speed is
about 4.5MB/sec, while write speed is about 3.7MB/s

Your reading speed is very low. If the driver switches to 50 MHz, the theoretical throughput is 25 MByte/s (needing 2 clocks for a byte).
Can you make sure that the task receives enough CPU time?
Can you show the code of your reading test, along with buffers and sizes?

using your code in ff_format.c, did not see obvious different.

Then probably something else is not OK. For most cards it should really increase the speed of writing.

did not test ffconfigOPTIMISE_UNALIGNED_ACCESS, since I just test large data
( I tested with a 52MByte file )

And I suppose that the length of your buffer has a multiple of 512 bytes?

for you observed, I also see the similar, I use ftp to uploading file,
if the file exit, chose overwriting, speed is much higher than new file.

below is high performance sandisk card info, UHS-I card, I did not buy from here, but should be the same kind card:
https://www.amazon.com/SanDisk-Extreme-Memory-Speeds-Ready-SDSDQXP-064G-G46A/dp/B00NUB3530/ref=sr_1_1?ie=UTF8&qid=1496330323&sr=8-1&keywords=SDSDQXP-064G

I think that your SD/MMC peripheral can only support normal high-speeds cards. The UHS cards use different techniques (LVDS) for ultra-high speed, as described here.

I tested this card in windows, the writing speed can reach around 80MB/sec.

That sounds like an UHS card in am UHS adapter.

Summary:

increase cluster, using high performance card is the most help

ffconfigWRITE_BOTH_FATS can also improve the speed.

and one more info, the platform I am using is zybo, CPU clock is 666MHz,
not sure if this CPU also a bottle neck for the performance.

That should be the same as my Zynq ( MicroZed ): the project uses a single core at 666 MHz.

Just tested on my Zybo board as well. I used a 4MB buffer and each file was written or read in two calls to ff_fwrite() :

    Write /test0002.txt 8388608 took 2.209 sec avg 3.888 MB
    Write /test0003.txt 8388608 took 1.811 sec avg 4.742 MB
    Write /test0004.txt 8388608 took 2.186 sec avg 3.929 MB
    Write /test0005.txt 8388608 took 2.319 sec avg 3.702 MB
    
    Read /test0002.txt 8388608 took 0.347 secs avg 24.689 MB
    Read /test0003.txt 8388608 took 0.348 secs avg 24.679 MB
    Read /test0004.txt 8388608 took 0.348 secs avg 24.647 MB
    Read /test0005.txt 8388608 took 0.348 secs avg 24.672 MB

which is much the same as a Zynq would do.

And here is my read routine:

#define WRITE_BUF_SIZE        ( 4ul * 1024ul * 1024ul )
#define WRITE_BUF_COUNT        ( 2 )

/* The aligned attribute is important: the buffer may be passed directly to DMA. */
__attribute__ ((aligned (32))) uint8_t ucBuffer[ WRITE_BUF_SIZE ];

void read_test()
{
    uint64_t ullTimes[ WRITE_BUF_COUNT + 1 ];

    FF_FILE fp = ff_fopen( fname, "r" );
    if (fp == NULL )
    {
        FreeRTOS_printf( ( "ff_fopen %s failed\n", fname ) );
        return;
    }
    ullTimes[ 0 ] = ullGetHighResolutionTime( );
    for( index = 0; index < WRITE_BUF_COUNT; index++ )
    {
    int rc;
        rc = ff_fread( ucBuffer, 1, WRITE_BUF_SIZE, fp );
        ullTimes[ index + 1 ] = ullGetHighResolutionTime( );
        if( rc <= 0 )
        {
            break;
        }
    }
    ff_fclose( fp );
}

Regards.