FreeRTOS+FAT ff_fseek() non-standard behavior

Libraries
1

Re: ff_fseek()

I’m experimenting with large files on SD cards. Writing multi-GB files is slow (on the order of a GB/hour), and I tried speeding it up by using an old Unix trick to reserve the space up front:

/* Try to reserve some space */
int rc = ff_fseek( pxFile, 0, FF_SEEK_SET ); // Seek to beginning of file
if (-1 == rc) printf("ff_fseek(%s): %s (%d)\n", pathname, strerror(stdioGET_ERRNO()), -stdioGET_ERRNO());
size_t x = size - 1;
while (x > 0) {
    long y = LONG_MAX - x;
    if (y < 0) {
        rc = ff_fseek( pxFile, LONG_MAX, FF_SEEK_CUR ); // Seek as far as we can
        if (-1 == rc) printf("ff_fseek(%s): %s (%d)\n", pathname, strerror(stdioGET_ERRNO()), -stdioGET_ERRNO());            
        x -= LONG_MAX;
    } else {
        rc = ff_fseek( pxFile, x, FF_SEEK_CUR );
        if (-1 == rc) printf("ff_fseek(%s): %s (%d)\n", pathname, strerror(stdioGET_ERRNO()), -stdioGET_ERRNO());                        
        x = 0;
    }       
}
uint8_t byte = 0;
lItems = ff_fwrite( &byte, sizeof(byte), 1, pxFile );
if (lItems < 1) printf("ff_fwrite(%s): %s (%d)\n", pathname, strerror(stdioGET_ERRNO()), -stdioGET_ERRNO());

However, this fails with

ff_fseek(/UserSDCrd/bft250M): Illegal seek (-29)

According to The Open Group Base Specifications, fseek , “The fseek() function shall allow the file-position indicator to be set beyond the end of existing data in the file. If data is later written at this point, subsequent reads of data in the gap shall return bytes with the value 0 until data is actually written into the gap.”

So, the old trick doesn’t work. I think I’m going to have to resort to a special API to reserve space with the SD pre-erase command (ACMD23). According to SD Physical Layer Simplified Specification 6.00 (https://www.sdcard.org/downloads/pls/index.html), “Setting a number of write blocks pre-erased (ACMD23) will make a following Multiple Block Write operation faster compared to the same operation without preceding ACMD23.”

2

Well, I tried that, and it didn’t help much. I think for it to work, the ACMD23 would have to be followed immediately by all of the CMD25s (WRITE MULTIPLE BLOCK) without any “stop transmission” tokens, but I’m only writing a few blocks at a time through +FAT.

Meanwhile, I discovered ff_truncate which does let me specify the file size. Unfortunately, ff_truncate does not behave at all like I would expect. The Open Group Base Specifications says “The truncate () function shall not modify the file offset for any open file descriptions associated with the file.” But ff_truncate seems to permanently move the offset to the end of the file. Even ff_rewind doesn’t seem to get the offset back to the beginning. For example, this little test case:

char pcBuffer[8];            
FF_FILE *pxFile = ff_truncate(pathname, sizeof(pcBuffer));
configASSERT(pxFile);
ff_rewind(pxFile); 
size_t i;
for (i = 0; i < sizeof(pcBuffer); ++i) {
    pcBuffer[i] = i;
}
ff_fwrite( pcBuffer, sizeof(pcBuffer), 1, pxFile );
ff_fclose( pxFile );

pxFile = ff_fopen( pathname, "r" );
char val;
while (1 == ff_fread( &val, sizeof(val), 1, pxFile )) {
    printf("%d ", val);
}
printf("\n");

produces this output:

0 0 0 0 0 0 0 0 0 1 2 3 4 5 6 7

and the CLI dir command says [writable file] [size=16].

3

Carl, thanks again for testing and reporting.

The 2 “issues” that you mention are already known:

  • ff_fseek() pas end-of-file: we decided not to implement this in order to keep the library simple. There has rarely been a request for this.

  • Write to a file in append mode: when a file is opened in append mode (like ff_truncate() does), and ff_write() will always make sure that it writes at the EOF position.

You are right that the behaviour in both cases is not in accordance with the Open Group Base Specifications.

I’m afraid that if we change this now, existing users will complain about the unexpected behaviour.

PS. Doesn’t your device have a 4-bit connection for SD cards?

4

In the beginning of this thread, I said that I have never tried FreeRTOS+FAT on a drive larger than 32 GB. I went to a shop and bought a 128 GB Sandisk micro SD-card.

I tested it on a Xilinx Zynq. I had to upgrade the Xilinx driver, because it would only recognise SDHC cards, and not SDXC/UHS-I cards.

You are right, when the low-level driver recognises the large disk, FreeRTOS+FAT has no problem with it.

I created 3 paritions of 42 GB each and mounted them as /, /part1, and /part2.

More tests will be needed before I can really confirm that it works.

5

I’m still surprised that I can’t seek back to the beginning. I guess I could close and reopen the file. Anyway, ff_truncate() certainly doesn’t save any time.

Fair enough. I was thinking about portability, since I have been considering porting SQLite to FreeRTOS+FAT, but I think it is not too demanding of the filesystem underlying its virtual file system or “VFS”. (Porting SQLite To New Operating Systems) (And, it is open source.)

It doesn’t have an SD Host Controller, if that’s what you’re asking. See PSoC® 6 MCU: PSoC 63 with BLE Datasheet Programmable System-on-Chip (PSoC®).

6

Such a vast amount of space in such a tiny package! And, for not much money, considering.

7

Such a vast amount of space in such a tiny package!
And, for not much money, considering.

Sure!
In a meanwhile I tested it further and created a PR #3 on FreeRTOS Labs.

I made sure that you can use these 2 functions:

FF_Error_t FF_Partition( pxDisk, pxParameters );
FF_Error_t FF_Format( pxDisk, xPartitionNumber, xPreferFAT16, xSmallClusters );

I found a problem in FF_Format() when formatting the second and third partition. That is now repaired in ff_format.c

I was able to use 3 partitions in parallel, each one 42 GB.

8

Excellent!

I didn’t know until now that FreeRTOS-FAT lived here:
FreeRTOS/ Lab-Project-FreeRTOS-FAT

Earlier, I uploaded a sort of demo project for PSoC 63 to:
carlk3/ FreeRTOS-FAT-CLI-for-PSoC-63