hesimo wrote on Monday, May 22, 2017:
In a loop I open, write and close a file on SD-Card. Each time a file is opened, the access time is increased.
How can I solve the problem?
rtel wrote on Monday, May 22, 2017:
Have you tried a different SD card? There HUGE variations between
cards. It is worth keeping a few premium brand cards to hand for
comparison.
heinbali01 wrote on Monday, May 22, 2017:
Beside the quality of the SD-cards, which indeed varies a lot:
How many files do you create within a single directory?
The FAT system has a known problem with huge directories, I mean directories with thousands of files.
We have tried to optimise the access in huge directories.
There are some options that may increase the efficiency:
#define ffconfigHASH_CACHE 1
/* Set to CRC8 or CRC16 to use 8-bit or 16-bit
* HASH values respectively. */
#define ffconfigHASH_FUNCTION CRC16
#define ffconfigHASH_CACHE_DEPTH 64
Postpone flushing to disk :
#define ffconfigCACHE_WRITE_THROUGH 0
hesimo wrote on Tuesday, May 23, 2017:
I have tested different card.
The controller ist an ATSAM4E @ 120MHz. I use the HSMCI (own driver. not the asf stuff) to write 250 files (96Bytes/file) in a single directory. Hash function brings no improvement and needs too much ram.
Intenso SDHC 4GB class 4
1 16ms
100 36ms
250 74ms
LYNQ SDHC 4GB class 4
1 19ms
100 45ms
250 86ms
Transcend SDHC 4GB class 10
1 14ms
100 36ms
250 69ms
Toshiba SDHC 4GB class 4
1 14ms
100 37ms
250 71ms
I have determined that the time is used in the ff_open function.
heinbali01 wrote on Tuesday, May 23, 2017:
Hi Hendrik, thanks for this overview!
You’re proving that the difference between qualities of the cards tested is not as spectacular as we assumed. Maybe the speed differences will show-up more clearly when writing tons of data.
I assume that the times show ( 16, 36, 74 ms ), are the times you need to create a single file? To me that looks as extremely fast!
How did you measure this? Is that the time it took to call ff_fopen()?
We have spend a lot of time on working with directories that contain like 65K files. The hashing of directory entries should really help to determine if a given short file name already exists or not.
I just took out my SAM4E Xplained board and created 250 files. The time it took to open/write/close was much longer: a minimum of 200 ms per file. The open itself took most of the time.
I attached a graph the actual times on my board. I used a cheap Transcend SDHC 4GB SD-card.
I wonder where our enormous difference in time comes from
heinbali01 wrote on Tuesday, May 23, 2017:
Another question that I should have asked you: do your file names confirm to the 8.3 notation?
Or are those Long File Names ?
hesimo wrote on Wednesday, May 24, 2017:
Hi, the files already exist on card. In a for loop I open overwrite and close each file. The times I measured, are the times for a single file. I simply measure the time by xTaskGetTickCount().
The file names confirm to the 8.3 notation. For all files I need about 10s.
I tested again (Transcend SDHC 4GB class 4)
file 1: open 4ms, write 0ms, close 7ms
file 100: open 25ms, write 2ms, close 6ms
file 250: open 82ms, write 3ms, close 8ms
heinbali01 wrote on Friday, May 26, 2017:
Hi Hendrik, I was measuring the time to create 250 new files, which is a lot more work.
ffconfigHASH_CACHE only optimises the creation of new files. It calculates a hash table for a directory so it knows quickly if a new short file name already exists or not.
On my SAM4E it also takes 10s to re-write 250 files of 96 bytes each.
These were the times that I measured:
# Open Write/Close
1 1 9
2 1 7
3 0 8
100 5 8
101 9 8
102 5 7
150 18 9
151 17 9
152 17 9
200 17 9
201 18 9
202 17 10
247 22 9
248 21 10
249 21 10
250 21 17
And I see some strange peeks while writing/closing the files:
# Open Write/Close
130 14 9
131 14 154
132 15 305
133 15 316
134 15 151
135 15 9
136 15 9
I’m not sure what is taking up to 300 ms.
Sometimes an SD-card has some internal clean-up, it will flush cached sectors, which can take time. The card will remain ‘busy’ until the internal work is done.
The files were created from FILE001.TXT up to FILE250.TXT. So I think that when you upgrade them it is normal to see access times increasing along with the index of the file.
Regards.
glenenglish wrote on Friday, May 26, 2017:
Hi Hendrick
I have lots of SD card experience, You should NEVER assume the write performance will be deterministic. Hein is absolutely correct with regard to internal housekeeping.
However- stick to the highest class (and most expensive cards).
AND there are MANY counterfeits . Lots of low quality rubbish with "Sandisk Extreme PRO " printed on them… fake fake fake fake fake . only buy from reputable and listed retailers. Buying on ebay is trouble…
The cheap cards- well my mother does not care if the photo storage takes an extra second more than usual, but your application might !