Slow Processing and calculations of FF_CountFreeClusters(FF_IOManager_t*, FF_Error_t*)

I’m using the FF_CountFreeClusters function in the FreeRTOS FAT layer implementation, and I’ve noticed that it takes an excessively long time to complete. After some investigation, it seems that a lot of single block reads are being used instead of multi-block reads, which could be contributing to the slow performance.

Is there an ideal way to optimize this process? Could there be a potential issue in the low-level driver?
Any advice on improving the efficiency of FF_CountFreeClusters would be appreciated.

Hi @adithya65, thank you for using +FAT, and also for suggesting an optimisation of counting the free clusters. I know that it can last a long time when checking the free space on a large disk. The whole FAT table must be read, and the vacant entries are counted.

You must have seen the following macros:

/* Set to 1 to determine the disk's free space and the disk's first free
 * cluster when a disk is mounted.
 *
 * Set to 0 to find these two values when they are first needed.
 * Determining the values can take some time. */

    #define ffconfigMOUNT_FIND_FREE    1  /* default 0 */

/* Set to 1 to 'trust' the contents of the 'ulLastFreeCluster' and
 * ulFreeClusterCount fields.
 *
 * Set to 0 not to 'trust' these fields.*/

    #define ffconfigFSINFO_TRUSTED    1  /* default 0 */

In my projects, I have switched them on. Once the free count has been calculated, it is stored in a special place on disk. And after that, the value is updated, and FF_CountFreeClusters() is always fast.

But if you still want to optimise FF_CountFreeClusters(), you can not use the standard 512-byte cache buffers.
You would need a big buffer, eg. 10 x 512 bytes or more, so you can read at least 10 sectors at a time.
You can easily test this by doing two things:

  • Read 10 single sectors in 10 calls to ff_fread()
  • Read 10 sectors in a single call

And if you have lots of RAM, try to read the whole FAT in one go.

( PS when reading large amounts of data, the sector-caching system is by-passed. The user-supplied buffer will be passed directly to the DMA controller. )

2 Likes

Thank you @htibosch ,
The suggested method of setting the macros works for me.

Thanks for reporting back!

PS. I compared reading the FAT in blocks 512 bytes with reading blocks of 20 sectors (10 KB). The latter is about 7 times faster.
So it is worth changing the function FF_CountFreeClusters() one day, to use larger blocks of RAM.

Great to know that as well,
When I traversed through the code, I found that the FF_CountFreeClusters() uses 1 block read at a time.
when I tried to increase the count to 10, it resulted In a freertos abort. I suspect that the buffer allocated is limited to 512Bytes to handle the read data. Is there a suitable way to increase the buffer size?

The most “standard” way of counting the free clusters is shown in prvCountFreeClustersSimple():

    ulFreeClusters = 0U;
    for( ulIndex = 0; ulIndex < xTotalClusters; ulIndex++ )
    {
        ulFATEntry = FF_getFATEntry( pxIOManager, ulIndex, &xError, NULL );

        if( ulFATEntry == 0UL )
        {
            ulFreeClusters++;
        }
    }

When you apply that to a FAT32 partition on large flash disk, you can have lunch while the application is counting clusters.

In FF_CountFreeClusters(), things go a bit faster, because it works with sector buffers: FF_Buffer_t.

A buffer can either be a:

  • write buffer with a single owner
  • read buffer with multiple owners
  • vacant buffer

Currently, the “free clusters” are counted using “FF_Buffer_t” buffers.

A third method:

As we are only counting clusters, without modifying them, it is OK to read the FAT, while other tasks are reading/writing in it. It may happen that the resulting count is a little too high or too low, because the situation has changed in a meanwhile. No problem.

I have written a sketch, what a new function FF_FastCountFreeClusters() could look like. Note the word “Fast”.

uint32_t FF_FastCountFreeClusters(
             FF_IOManager_t * pxIOManager, // The I/O manager.
             FF_Error_t * pxError,         // Will contain an error, if any.
             size_t uxBlockSize )          // The size of the buffer to be used.

It will use this existing read function:

int32_t FF_BlockRead(
            FF_IOManager_t * pxIOManager, // The I/O manager.
            uint32_t ulSectorLBA,         // The logical block address of the sector.
            uint32_t ulNumSectors,        // Number of sectors to read.
            void * pBuffer,               // The RAM buffer.
            BaseType_t aSemLocked );      // True if it already holds a semaphore.

I used this testing code:

for( int index = 0; index < 2; index++ )
{
    size_t uxBlockSize = (index == 0) ? 512 : 102400;
    FF_Error_t xError;
    FreeRTOS_printf( ( "Counting block size %u\n", ( unsigned ) uxBlockSize ) );
    uint32_t count = FF_FastCountFreeClusters( pxIOManager, &xError, uxBlockSize );
    FreeRTOS_printf( ( "Counting count %u\n", ( unsigned ) count ) );
}

When it reads single sectors, it lasts 5 seconds. When it reads blocks of 100 KB each, counting lasts 0.5 second only.

Buffer size 512: 5 seconds

 S.MS
05.791   Counting block size 512
10.974   Break out of 1st loop
10.974   Break out of 2nd loop
10.974   Counting read_count = 15211 SecsPerBlock 1 compares 1946921
10.974   Counting count 1932760

Buffer size 10 KB: 0.5 seconds

 S.MS
00.974   Counting block size 102400
01.481   Reading the last 32 sectors
01.482   Break out of 1st loop
01.482   Break out of 2nd loop
01.483   Counting read_count = 77 SecsPerBlock 200 compares 1946921
01.483   Counting count 1932760

Here is the code that you can try:

FreeRTOS-FAT.free_cluster_count.zip (369.3 KB)

Note: this is a testing version. Also the code for FSinfo still needs some work. I started in ff_ioman.c with function FF_UpdateFSInfo(), which will replace duplicate code.

1 Like

Thanks a lot, @htibosch for the prompt response, This patch seems to work awesome.
I am able to reduce the time taken for the directory creation with FF_FastCountFreeClusters.