FreeRTOS-FAT creates files with same long file name but with different short name

Kernel
1

Hi,

We create a FreeRTOS-FAT test to create/write files using FreeRTOS-FAT on SD card. The test is multi-threading. From test result, we get 4 files with same long file name (fstest_dump_task_0x3971_file_idx_0.bin), while the short file names are different.

The sdcard can mount and read/write in Linux and no error message pop-up

Is it a normal or it means there is an issue?

/mnt/tmp$ ls -l
-rwxr-xr-x 1 root root 10485760 1月 1 2028 fstest_dump_task_0x3971_file_idx_0.bin
-rwxr-xr-x 1 root root 10485760 1月 1 2028 fstest_dump_task_0x3971_file_idx_0.bin
-rwxr-xr-x 1 root root 10485760 1月 1 2028 fstest_dump_task_0x3971_file_idx_0.bin
-rwxr-xr-x 1 root root 10485760 1月 1 2028 fstest_dump_task_0x3971_file_idx_0.bin

Using Linux fatcat, they have different short names:

sudo ./fatcat /dev/sdd -l /
Listing path /

f 1/1/2028 00:00:00 fstest_dump_task_0x3971_file_idx_0.bin (FSTEST~1.BIN) c=371496 s=10485760 (10M)
f 0/1/2028 00:00:00 fstest_dump_task_0x3971_file_idx_0.bin (FSTEST~2.BIN) c=371498 s=10485760 (10M)
f 0/1/2028 00:00:00 fstest_dump_task_0x3971_file_idx_0.bin (FSTEST~3.BIN) c=371497 s=10485760 (10M)
f 0/1/2028 00:00:00 fstest_dump_task_0x3971_file_idx_0.bin (FSTEST~4.BIN) c=371499 s=10485760 (10M)

2

Interesting, and thanks for reporting this.

Did you also run fsck.vfat? And did it complain?

Would it be difficult for me to reproduce this? I guess that I need 4 tasks that run in parallel and created files with the same LFN?
As you know, the SFN is “composed” by the driver, using certain standard rules to translate an LFN to a SFN.

The result it not what you want, but looking at the code, it is as expected:

Task 1 creates “fstest_dump_task_0x3971_file_idx_0.bin”, which results in a SFN “FSTEST~1.BIN”.
Task 2 creates the same LFN, and the driver looks for the next available SFN, which is “FSTEST~2.BIN”.

But this is not what you want. Task 2 should not be able to succeed to create a file with mentioned LFN. It should get an error as long as task-1 has an open write handle to this file.

I will have a closer look.

I would be grateful if you can attach some testing code.

3

This the test code, dump 10M memory content into file

FSTEST_BUF_ADDR is the memory address, can be changed in other environment

fopen/fwrite/fclose just a wrapper to ff_open/etc

#define FSTEST_TASK_NUM 4
#define FSTEST_FILE_NUM 10
#define FSTEST_FILE_SIZE (10 * 1024 * 1024)
#define FSTEST_BUF_ADDR 0x1F700000U
static void fstest_write_file(void *p)
{
    int i=0;
    char fname[256];
    uint32_t *task_ptr = (uint32_t *)p;

    uint32_t task_id = 0x3971;

    for (uint32_t i = 0; i < FSTEST_FILE_NUM; i++) {
        sprintf(fname, "fstest_dump_task_0x%x_file_idx_%d.bin", task_id, i);
        FILE *fp = fopen(fname, "w");
        if (fp == NULL) {
            printf("Open file fail\n");
            break;
        }
        size_t size = fwrite((void *)FSTEST_BUF_ADDR, FSTEST_FILE_SIZE, 1, fp);
        if (size != 1U) {
            printf("Write file fail\n");
            fclose(fp);
            break;
        }
        fclose(fp);
        printf("task 0x%x file %d is written:%s\n", task_id, i, fname);
    }

    *task_ptr = *task_ptr + 1;
    printf("task 0x%x done: write %d files, join number=%d\n", task_id, param_num, *task_ptr);

    vTaskDelete(NULL);
}

static void test_filesystem()
{
    TaskHandle_t fstest_task[FSTEST_TASK_NUM];

    uint32_t join_num = 0U;
    /* Create Tasks */
    for (uint32_t i = 0; i < FSTEST_TASK_NUM; i++) {
        xTaskCreate(fstest_write_file, "fstest", 1024, (void *)(&join_num), 1, &fstest_task[i]);
    }

    while (join_num < FSTEST_TASK_NUM) {
        vTaskDelay(1000U);
    }

    printf("Test done, check filesystem on SDcard\n");
}
4

Task 1 creates “fstest_dump_task_0x3971_file_idx_0.bin”, which results in a SFN “FSTEST~1.BIN”.
Task 2 creates the same LFN, and the driver looks for the next available SFN, which is “FSTEST~2.BIN”.
But this is not what you want. Task 2 should not be able to succeed to create a file with mentioned LFN. It should get an error as long as task-1 has an open write handle to this file.
But this is not what you want. Task 2 should not be able to succeed to create a file with mentioned LFN. It should get an error as long as task-1 has an open write handle to this file.

For above test code, sometime there is “open fail” (see bellow log)
The the first file (idx_0) are all created for the four tasks
And then the following files (idx_1/2/…), the task may return “open fail”

FF_Open may need to be protected by FF_PendSemaphore( pxIOManager->pvSemaphore )

Test log (get “open fail” sometimes)

Conduct test for fs
task 0x3971 file 0 is written:fstest_dump_task_0x3971_file_idx_0.bin
task 0x3971 file 0 is written:fstest_dump_task_0x3971_file_idx_0.bin
task 0x3971 file 0 is written:fstest_dump_task_0x3971_file_idx_0.bin
task 0x3971 file 0 is written:fstest_dump_task_0x3971_file_idx_0.bin
Open file fail
task 0x3971 done: writtae sk1 0 f0xil39e7s1,  jfoiline  n1u imbse wrr=1it
 en:fstest_dump_task_0x3971_file_idx_1.bin
task 0x3971 file 1 is written:fstest_dump_task_0x3971_file_idx_1.bin
task 0x3971 file 1 is written:fstest_dump_task_0x3971_file_idx_1.bin
Open file fail
eask 0x3971 done: writtaes k 100x 3f9il7e1s f,i ljoe i2n  insu mbwreir=t2t
 n:fstest_dump_task_0x3971_file_idx_2.bin
task 0x3971 file 2 is written:fstest_dump_task_0x3971_file_idx_2.bin
task 0x3971 file 3 is written:fstest_dump_task_0x3971_file_idx_3.bin
task 0x3971 file 3 is written:fstest_dump_task_0x3971_file_idx_3.bin
task 0x3971 file 4 is written:fstest_dump_task_0x3971_file_idx_4.bin
task 0x3971 file 4 is written:fstest_dump_task_0x3971_file_idx_4.bin
task 0x3971 file 5 is written:fstest_dump_task_0x3971_file_idx_5.bin
task 0x3971 file 5 is written:fstest_dump_task_0x3971_file_idx_5.bin

dir/ls in freertos

>ls
fstest_dump_task_0x3971_file_idx_0.bin [writable file] [size=4096]
fstest_dump_task_0x3971_file_idx_0.bin [writable file] [size=4096]
fstest_dump_task_0x3971_file_idx_0.bin [writable file] [size=4096]
fstest_dump_task_0x3971_file_idx_0.bin [writable file] [size=4096]
fstest_dump_task_0x3971_file_idx_1.bin [writable file] [size=4096]
fstest_dump_task_0x3971_file_idx_1.bin [writable file] [size=4096]
fstest_dump_task_0x3971_file_idx_1.bin [writable file] [size=4096]
fstest_dump_task_0x3971_file_idx_2.bin [writable file] [size=4096]
fstest_dump_task_0x3971_file_idx_2.bin [writable file] [size=4096]
fstest_dump_task_0x3971_file_idx_3.bin [writable file] [size=4096]
fstest_dump_task_0x3971_file_idx_3.bin [writable file] [size=4096]
fstest_dump_task_0x3971_file_idx_4.bin [writable file] [size=4096]
fstest_dump_task_0x3971_file_idx_4.bin [writable file] [size=4096]
fstest_dump_task_0x3971_file_idx_5.bin [writable file] [size=4096]
fstest_dump_task_0x3971_file_idx_5.bin [writable file] [size=4096]

fsck result

$ sudo fsck.vfat  /dev/sdd
fsck.fat 4.1 (2017-01-24)
0x41: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
1) Remove dirty bit
2) No action
? 1
Perform changes ? (y/n) y
/dev/sdd: 23 files, 24/1893985 clusters

$ sudo mount /dev/sdd /mnt/tmp/
$ ls /mnt/tmp/
fstest_dump_task_0x3971_file_idx_0.bin  fstest_dump_task_0x3971_file_idx_1.bin  fstest_dump_task_0x3971_file_idx_3.bin  fstest_dump_task_0x3971_file_idx_6.bin  fstest_dump_task_0x3971_file_idx_8.bin
fstest_dump_task_0x3971_file_idx_0.bin  fstest_dump_task_0x3971_file_idx_1.bin  fstest_dump_task_0x3971_file_idx_4.bin  fstest_dump_task_0x3971_file_idx_6.bin  fstest_dump_task_0x3971_file_idx_9.bin
fstest_dump_task_0x3971_file_idx_0.bin  fstest_dump_task_0x3971_file_idx_2.bin  fstest_dump_task_0x3971_file_idx_4.bin  fstest_dump_task_0x3971_file_idx_7.bin  fstest_dump_task_0x3971_file_idx_9.bin
fstest_dump_task_0x3971_file_idx_0.bin  fstest_dump_task_0x3971_file_idx_2.bin  fstest_dump_task_0x3971_file_idx_5.bin  fstest_dump_task_0x3971_file_idx_7.bin
fstest_dump_task_0x3971_file_idx_1.bin  fstest_dump_task_0x3971_file_idx_3.bin  fstest_dump_task_0x3971_file_idx_5.bin  fstest_dump_task_0x3971_file_idx_8.bin
5

Hello @Austin,

While I am unfamiliar with the FAT codebase, I see a potential race condition in your code: *task_ptr = *task_ptr + 1;. This line will be executed by all tasks in parallel and if they somehow coincide, it’ll lead to a different value.

I would suggest that you use task notifications in such scenarios. See this example: FreeRTOS task notifications.

6

Ok. Thanks for the comments

This is just make sure the parent task can wait till the four fork-ed tasks. It doesn’t impact the core issue we are discussing.

7

Thank you for this well documented report. It also works for me, I can reproduce the problem.

It looks indeed like a race between equal-priority tasks, which are trying to create the same object. When ff_fopen() returns a creation error, that is actually good.

The code would only close the file in case of an error. And also I would swap the COUNT and SIZE parameters of fwrite(), so you can see how many bytes (not blocks) have actually been written:

-    size_t size = fwrite(pucBuffer, FSTEST_FILE_SIZE, 1, fp);
+    size_t size = fwrite(pucBuffer, 1, FSTEST_FILE_SIZE, fp);
+    fclose(fp);
-    if (size != 1U) {
+    if (size != FSTEST_FILE_SIZE) {
            printf("Write '%s' failed\n", fname);
-           fclose(fp);
            break;
        }

Now I will see if there is an easy solution.

8

@Austin wrote:

FF_Open may need to be protected by FF_PendSemaphore( pxIOManager->pvSemaphore )

That will stop the race indeed!

The standard semaphore can not be used for this though: it would lead to a dead-lock.
There is also a lock of “directory access”, and a lock on the FAT, but those are not-recursive.

What I did is add a new semaphore called pvSemaphoreOpen:

pxIOManager->pvSemaphoreOpen = xSemaphoreCreateRecursiveMutex();

And protect the function FF_Open() with it, but only if a file is to be created.

I tested it and it works perfectly: only a single instance of each LFN (long file name) will be created on disk. Other tasks all receive an error because task-1 still has an open handle.

I would like to make it conditional on some macro.

You can find the new code in this branch.

I only changed 3 source files: ff_file.c, ff_ioman.c, and ff_ioman.h

You you have a try with these changes? You will have to add define to your copy of FreeRTOSFATConfig.h:

#define ffconfigPROTECT_FF_FOPEN_WITH_SEMAPHORE   1

You wrote:

#define FSTEST_FILE_SIZE (10 * 1024 * 1024)

I only wrote 4 KB to each file to see the race.

Hein

9

I just presented my branch as a pull request #26

Could you give it a try?

10

Thanks for the patch. Will verify and provide the feedback ASAP

11

After test, there is only 1 file is created.
Thank you for the quick fix.