In my project, I am using eMMC as my storage media and have a FAT FS set up on it using the +FAT library. My project requires high-speed data logging to be possible. I have each file to be set to a fixed size of 1MB. Once a file is full, a new one is opened, filled with 0s, flushed, and closed. This file is then opened by the logging task and written to in 1024B chunks and is closed when full and the process is repeated.
I used the 'new file filled with 0s" approach based on some previous posts in this forum. I believe the reason for doing this was to ensure that the dir entry in the FS gets updated with the correct file size before the file is used.
However, I am having an issue where, although fast, the new file process is not fast enough. My logger task has a queue into which other tasks that need to log data push an event. The firmware and eMMC are fast enough to process the events and write them to an open file. But when the file gets full and a new file has to be created, the events start piling up in the queue and don’t get cleared fast enough until another new file has to be created. As you can see, the firmware eventually reaches a point where there is no more room in the queue and events are dropped consistently. The following output from the serial terminal will clear this up:
Data logger: Queue spaces available: 828
Data logger: Queue spaces available: 828
Data logger: Queue spaces available: 828
Data logger: Queue spaces available: 828
Data logger: Queue spaces available: 828
Data logger: Queue spaces available: 828
Data logger: Queue spaces available: 828
Data logger: Queue spaces available: 828
Data logger: Queue spaces available: 828
Data logger: Queue spaces available: 828
Data logger: Queue spaces available: 828
Data logger: Queue spaces available: 828
Data logger: Queue spaces available: 828
Data logger: prv_logger_create_and_open_new_file: Closing file: /logs/20210314/log_20210314_182613.bin.
Data logger: prv_logger_create_and_open_new_file: Opening/Creating file: /logs/20210314/log_20210314_182615.bin.
Data logger: Queue spaces available: 828
Data logger: Queue spaces available: 827
Data logger: Queue spaces available: 826
Data logger: Queue spaces available: 825
Data logger: Queue spaces available: 824
Data logger: Queue spaces available: 823
Data logger: Queue spaces available: 822
Data logger: Queue spaces available: 821
Data logger: Queue spaces available: 820
Data logger: Queue spaces available: 819
Data logger: Queue spaces available: 818
Data logger: Queue spaces available: 817
Data logger: Queue spaces available: 816
Data logger: Queue spaces available: 815
Data logger: Queue spaces available: 814
Data logger: Queue spaces available: 813
Data logger: Queue spaces available: 812
Data logger: Queue spaces available: 811
Data logger: Queue spaces available: 810
Data logger: Queue spaces available: 809
Data logger: Queue spaces available: 808
Data logger: Queue spaces available: 807
Data logger: Queue spaces available: 806
Data logger: Queue spaces available: 805
Data logger: Queue spaces available: 804
Data logger: Queue spaces available: 803
Data logger: Queue spaces available: 802
Data logger: Queue spaces available: 801
Data logger: prv_logger_create_and_open_new_file: Successfully created new log file
As you can see, the number of free spaces in the queue drop while a new file is being created.
So here is my question:
Is it possible to create a new file and assign it a size so that the FAT will allocate the space for it without having to write 0s to it? I believe this will solve my issue as writing 1MB of 0s to the file is proving to be the bottleneck.
Sid, in principle it would be possible to allocate 1 MB for a file, and write that to the directory entry. The contents will be “whatever was written earlier”, maybe old logging data. But, FreeRTOS+FAT as it is now can not do that though.
Have you tried to fill a file with larger buffers? Is that faster?
Or, a bit more advanced: can’t you create a file earlier in time: as soon as you start writing to a new file, you start a task that fills a new file with zero’s. Wouldn’t that be possible?
. I believe the reason for doing this was to ensure that the dir entry in the FS gets updated with the correct file size before the file is used.
That is correct. A new file has three properties:
An entry in a directory that contains its ( short + long ) name and length.
A couple of entries in the FAT table.
Actual data clusters.
So when you pre-allocate the file, the above 3 areas are written to and flushed. When you gather logging, only 3), the clusters need to be updated, the others are correct already.
I tried doing it with larger buffers but it didn’t make a noticeable difference. I’m using a 1MB buffer for a 1MB file so performance-wise, this would be the best way to do it right?
This is the backup plan if I am unable to hack together a new function within FreeRTOS+FAT that lets me allocate 1MB for a file without having to write 0s to it.
Would it be possible for you to guide me about the things that need to happen to make this possible? This, for me, is the ideal solution. I am happy to write this function and share it with you if it isn’t too complicated to implement.
Would it be possible for you to guide me about the things that need to happen to make this possible?
I would rather create the empty files earlier from a separate low-priority task. Not with a 1MB buffer, but a smaller buffer to give other FAT-using tasks a chance to use the disk as well.
That is a clean solution, and you won’t get confused by seeing old data.
But if you prefer, you could change this function:
Now if the buffer is NULL, it should not call FF_WritePartial(), FF_BlockWrite(), or FF_WriteClusters(). And also, it should not increment pucBuffer as in:
pucBuffer += nBytesToWrite;
As you might guess:
FF_WritePartial() does a read/modify/write of a sector FF_BlockWrite()writes to a whole sector FF_WriteClusters()writes to one or more clusters
With this hack, you create a 1MB file and do a write of 1MB from a NULL buffer. All steps will be taken except actually writing to the clusters.
Okay since it doesn’t look like it is too difficult to do the hack, I will make that the backup option and do as you recommend - create a new file in a low priority task.
I agree with you that it is the cleaner solution, however, I have one concern with doing it this way - if the other task that writes data to the files ends up producing a large amount of data in a very short amount of time such that it fills up the current file and needs a new file (it’s a possible scenario), the low-priority task may never get a chance to fully create the new file. If this happens, it will become a problem for me.
I will think about it a bit more and try the implementations and report back.
With some help from Hein, I was able to use the following solution:
The function FF_ExtendFile in ff_file.c is declared as a static function. By removing the static keyword and adding the function to ff_file.h, it can be used to extend a file without having to fill it with 0s.
The only catch here is that the ulFileSize variable does not get updated so if your application depends on the size of the file in any way, it can be set manually if the FF_ExtendFile command succeeded.
Something like this (code provided by Hein Tibosh):
pxFile = ff_fopen( "logfile.txt", "w" );
if( pxFile != NULL )
{
FF_Error_t xError = FF_ExtendFile( pxFile, 1024*1024 );
if( FF_isERR( xError ) == pdFALSE )
{
/* ulFileSize has not been updated yet, because no
* data has been written yet. */
pxFile->ulFileSize += 1024*1024;
}
}
ff_fclose( pxFile );
if( FF_isERR( xError ) == pdTRUE )
{
}
This solution didn’t completely solve my problem of the queue getting backed up so I will likely have to implement a low priority task that creates and extends a new file.
I will report back once I have a system that meets my needs.
Maybe we can leave FF_ExtendFile() a static function and add a new function that also sets the file length.
Normally, FF_ExtendFile() only reserves new clusters. They belong to the file, they are contained in its clusters chain.
The actual file length depends on how much data has been written.
In this solution we skip the actual writing, and just increase the file member ulFileSize, after the clusters have been successfully reserved:
/* Extend the file, without actually writing to the new clusters. */
FF_Error_t FF_ExtendFileSize( FF_FILE * pxFile, uint32_t ulSize )
{
FF_Error_t xError = FF_ExtendFile( pxFile, ulSize );
if( FF_isERR( xError ) == pdFALSE )
{
/* ulFileSize has not been updated yet, because no
* data has been written yet. */
pxFile->ulFileSize += ulSize;
}
return xError;
}
I wonder if your FreeRTOS+FAT Media Driver sends CMD23 Set Block Count for the number of blocks in 1MB, prior to writing the blocks. I think there might be a performance advantage to doing that.
The Zynq SD-card driver uses DMA. It uses CMD25 to write multiple blocks to the disk.
Would it make a difference to use CMD23? I read that it is optional for SD-cards, and required for MMC.
The write-function can be found here. It is called XSdPs_WritePolled(), but it becomes interrupt-driven when ffconfigSDIO_DRIVER_USES_INTERRUPT is defined.
Pre-erase setting prior to a multiple block write operation Setting a number of write blocks to be pre_erased (ACMD23) will make a following Multiple Block Write operation faster compared to the same operation without preceding ACMD23. The host will use this command to define how many write blocks are going to be sent in the next write operation. If the host terminates the write operation (using stop transmission) before all the data blocks are sent to the card, the content of the remaining write blocks is undefined (can be either erased or still have the old data). If the host sends a greater number of write blocks than are defined in ACMD23, the card will erase blocks one by one (as new data is received). This number will be reset to the default (=1) value after Multiple Blocks Write operation.
It is recommended to use this command preceding CMD25, so that SanDisk’s SD Card will be faster for Multiple Write Blocks operation. Note that the host must send ACMD23 just before the WRITE command if the host wants to use the pre-erase feature. If not, pre-erase-count might be cleared automatically when another command (ex: Security Application Commands) is executed.
Thanks Carl for this information. I will keep that in mind.
But in the meanwhile, we found that writing zeros is actually very fast. When the buffer is big enough, it took like 40 ms to clear 1 MB.
Allocating enough clusters to store 1MB is slow. It requires a lot of reading and writing to the FAT.