FreeRTOS+FAT + FTP Server - Ring buffer of files

Libraries
1

Hello,

To start off, here are the details of my project:

Xilinx Zynq-7000 on a custom board
64GB eMMC card for storage
Latest FreeRTOS source from the Git repo
Latest FreeRTOS+TCP and an older version of +FAT

At its core, my project is a data logger that is storing data in binary log files and running an FTP server to send those log files to a base station.

The expectation is that the storage will never get full as log files will be pulled from the device on a regular basis using the FTP server and will then be deleted from the FS. However, I do have to plan for the worst case which means the logs keep on piling up and if the storage is full, I want to delete the oldest file in the FS and use the freed up space for the latest one.

Currently, based on Hein’s recommendations in a different forum post, I am creating a binary log file of a fixed size and filling it with 0s before using it for logging. Each file is configured for a size of 512 bytes. Once it is full, the file is closed and new one is created.

I also have timestamping implemented. My project is using an RTC so I had to modify the ff_time.c file to suit my RTC driver but the FS is now able to correctly tag the files with a valid timestamp.

So my question is that is it possible to somehow search the FS for the “oldest” file so that it can be deleted? And question 2 is that will the FS become slow once the files start piling up? Based on some rough math, I could have as many as ~128,000 files stored on the FS (64GB / 512B).

I am open to any suggestions to make a better implementation of my logging mechanism too.

Thank you for your help,
Sid

2

Does this do what you want https://freertos.org/FreeRTOS-Plus/FreeRTOS_Plus_FAT/stdio_API/ff_stat.html ? If not, look at the implementation of that file to see how it is obtaining the time.

3

Scanning 128,000 files in a FAT file system is indeed a big job. I hope that you can keep the directory sizes small, because FAT is inefficient when handling large folders.
Isn’t it possible to use larger files, like e.g. 1 MB each?

Another approach could be this: scan the entire disk in the background, in a low-priority task. Now if you have a maximum age, it is easy to decide what files should be removed.

4

@ sidmodi: did you get closer to a solution?

5

I have a similar requirement. I’m doing this at the moment, but it is a work in progress:

// If necessary, delete old files to free up space.
static bool pruneSysSDCrd(ssdc_t *this) {
	uint64_t freeMB;
	unsigned freePct;
	
	if (!this->pDisk)
		return false;

	const time_t now = FreeRTOS_time(0);
	if (this->prune_time >= now)
		return false;
	
	if (!this->prune_time) {
		struct tm t;
		localtime_r(&now, &t);
		t.tm_year -= 10; // Go back ten years
		t.tm_mon = 0; // and back to Jan 1st
		t.tm_mday = 0;
		this->prune_time = mktime(&t);
	}    
	
	getFree(this->pDisk, &freeMB, &freePct);
	FF_PRINTF("%s free space: %llu MB, %u %%\n", this->mntpnt, freeMB, freePct);
	struct tm tmbuf; 
	localtime_r(&this->prune_time, &tmbuf); 
	while ((freeMB < BIN_RESERVED_MB || freePct < RESERVED_PCT)) {
		char buf[BUF_LEN];
		size_t n = snprintf(buf, sizeof buf, "%s/%d", 
			this->mntpnt, tmbuf.tm_year);
		if (!ff_chdir(buf)) {
			++tmbuf.tm_year;
			this->prune_time = mktime(&tmbuf); // Renormalizes tm
		} else {        
			n = snprintf(buf, sizeof buf, "%d/%d/%d.bin", 
				tmbuf.tm_mon, tmbuf.tm_mday, tmbuf.tm_hour);
			configASSERT(sizeof buf > n);
			FF_PRINTF("%s: Removing %s.\n", __FUNCTION__, buf);
			int ec = ff_remove(buf);
			if (ec) {
#if defined(DEBUG)
				int error = stdioGET_ERRNO();
				DBG_PRINTF("%s: %s: %s (%d)\n", __FUNCTION__, "ff_remove", strerror(error), error);
#endif            
			} else {
				// See where we stand now
				getFree(this->pDisk, &freeMB, &freePct);
				FF_PRINTF("%s free space: %llu MB, %u %%\n", this->mntpnt, freeMB, freePct);
			}
		}
		++tmbuf.tm_hour;
		this->prune_time = mktime(&tmbuf); // Renormalizes tm
		if (this->prune_time >= now - SECS_24HOUR) { // Caught up to yesterday
			FF_PRINTF("%s: Could not free up space! (Check RTC)", __FUNCTION__);
			return false;
		}
	} // while
	return true;
}

where getFree is

void getFree(FF_Disk_t *pxDisk, uint64_t *pFreeMB, unsigned *pFreePct) {
	FF_Error_t xError;
	uint64_t ullFreeSectors, ulFreeSizeKB;
	int iPercentageFree;

	configASSERT(pxDisk);
	FF_IOManager_t *pxIOManager = pxDisk->pxIOManager;
    
	FF_GetFreeSize(pxIOManager, &xError);

	ullFreeSectors = pxIOManager->xPartition.ulFreeClusterCount * pxIOManager->xPartition.ulSectorsPerCluster;
	if (pxIOManager->xPartition.ulDataSectors == 0) {
		iPercentageFree = 0;
	} else {
		iPercentageFree = (int) (( 100ULL * ullFreeSectors + pxIOManager->xPartition.ulDataSectors / 2)
				/ ((uint64_t) pxIOManager->xPartition.ulDataSectors));
	}

    const int SECTORS_PER_KB = 2;
	ulFreeSizeKB = (uint32_t)(ullFreeSectors / SECTORS_PER_KB);
    
    *pFreeMB = ulFreeSizeKB/1024;
    *pFreePct = iPercentageFree;
}

and pruneSysSDCrd is called from a task when requested by a Software Timer.

6

Hi @htibosch, I got busy with another aspect of the project and didn’t get time to work on this. Will start work on it again this week and will update here when I have something worth sharing.

@carlk3 thanks for sharing your solution! I will take a look at it and see if I can implement it into my project if it fits.

7

Hello,

Just an update for anyone looking at this thread - I spoke with my client and ultimately we decided that there will never be a situation where the 64GB eMMC card will get full so they asked me to not spend time on developing this ring buffer mechanism. In the odd situation that the card does get full, the firmware will keep overwriting the latest file until more room becomes available.

Best,
Sid