FAT acces time

hesimo wrote on Monday, May 22, 2017:

In a loop I open, write and close a file on SD-Card. Each time a file is opened, the access time is increased.
How can I solve the problem?

rtel wrote on Monday, May 22, 2017:

Have you tried a different SD card? There HUGE variations between
cards. It is worth keeping a few premium brand cards to hand for
comparison.

heinbali01 wrote on Monday, May 22, 2017:

Beside the quality of the SD-cards, which indeed varies a lot:

How many files do you create within a single directory?
The FAT system has a known problem with huge directories, I mean directories with thousands of files.
We have tried to optimise the access in huge directories.
There are some options that may increase the efficiency:

    #define	ffconfigHASH_CACHE           1
    /* Set to CRC8 or CRC16 to use 8-bit or 16-bit
     * HASH values respectively. */
    #define	ffconfigHASH_FUNCTION        CRC16
    #define ffconfigHASH_CACHE_DEPTH     64

Postpone flushing to disk :

    #define ffconfigCACHE_WRITE_THROUGH   0

hesimo wrote on Tuesday, May 23, 2017:

I have tested different card.

The controller ist an ATSAM4E @ 120MHz. I use the HSMCI (own driver. not the asf stuff) to write 250 files (96Bytes/file) in a single directory. Hash function brings no improvement and needs too much ram.

Intenso SDHC 4GB class 4
1 16ms
100 36ms
250 74ms

LYNQ SDHC 4GB class 4
1 19ms
100 45ms
250 86ms

Transcend SDHC 4GB class 10
1 14ms
100 36ms
250 69ms

Toshiba SDHC 4GB class 4
1 14ms
100 37ms
250 71ms

I have determined that the time is used in the ff_open function.

heinbali01 wrote on Tuesday, May 23, 2017:

Hi Hendrik, thanks for this overview!
You’re proving that the difference between qualities of the cards tested is not as spectacular as we assumed. Maybe the speed differences will show-up more clearly when writing tons of data.

I assume that the times show ( 16, 36, 74 ms ), are the times you need to create a single file? To me that looks as extremely fast!

How did you measure this? Is that the time it took to call ff_fopen()?

We have spend a lot of time on working with directories that contain like 65K files. The hashing of directory entries should really help to determine if a given short file name already exists or not.

I just took out my SAM4E Xplained board and created 250 files. The time it took to open/write/close was much longer: a minimum of 200 ms per file. The open itself took most of the time.

I attached a graph the actual times on my board. I used a cheap Transcend SDHC 4GB SD-card.

I wonder where our enormous difference in time comes from

heinbali01 wrote on Tuesday, May 23, 2017:

Another question that I should have asked you: do your file names confirm to the 8.3 notation?
Or are those Long File Names ?

hesimo wrote on Wednesday, May 24, 2017:

Hi, the files already exist on card. In a for loop I open overwrite and close each file. The times I measured, are the times for a single file. I simply measure the time by xTaskGetTickCount().
The file names confirm to the 8.3 notation. For all files I need about 10s.

I tested again (Transcend SDHC 4GB class 4)
file 1: open 4ms, write 0ms, close 7ms
file 100: open 25ms, write 2ms, close 6ms
file 250: open 82ms, write 3ms, close 8ms

heinbali01 wrote on Friday, May 26, 2017:

Hi Hendrik, I was measuring the time to create 250 new files, which is a lot more work.

ffconfigHASH_CACHE only optimises the creation of new files. It calculates a hash table for a directory so it knows quickly if a new short file name already exists or not.

On my SAM4E it also takes 10s to re-write 250 files of 96 bytes each.

These were the times that I measured:

   # Open Write/Close
   1   1      9
   2   1      7
   3   0      8

 100   5      8
 101   9      8
 102   5      7

 150  18      9
 151  17      9
 152  17      9

 200  17      9
 201  18      9
 202  17     10

 247  22      9
 248  21     10
 249  21     10
 250  21     17

And I see some strange peeks while writing/closing the files:

  #  Open Write/Close
 130  14      9
 131  14    154
 132  15    305
 133  15    316
 134  15    151
 135  15      9
 136  15      9

I’m not sure what is taking up to 300 ms.
Sometimes an SD-card has some internal clean-up, it will flush cached sectors, which can take time. The card will remain ‘busy’ until the internal work is done.

The files were created from FILE001.TXT up to FILE250.TXT. So I think that when you upgrade them it is normal to see access times increasing along with the index of the file.

Regards.

glenenglish wrote on Friday, May 26, 2017:

Hi Hendrick
I have lots of SD card experience, You should NEVER assume the write performance will be deterministic. Hein is absolutely correct with regard to internal housekeeping.

However- stick to the highest class (and most expensive cards).
AND there are MANY counterfeits . Lots of low quality rubbish with "Sandisk Extreme PRO " printed on them… fake fake fake fake fake . only buy from reputable and listed retailers. Buying on ebay is trouble…

The cheap cards- well my mother does not care if the photo storage takes an extra second more than usual, but your application might !