FreeRTOS+FAT + FTP Server - Ring buffer of files

Hello,

To start off, here are the details of my project:

Xilinx Zynq-7000 on a custom board
64GB eMMC card for storage
Latest FreeRTOS source from the Git repo
Latest FreeRTOS+TCP and an older version of +FAT

At its core, my project is a data logger that is storing data in binary log files and running an FTP server to send those log files to a base station.

The expectation is that the storage will never get full as log files will be pulled from the device on a regular basis using the FTP server and will then be deleted from the FS. However, I do have to plan for the worst case which means the logs keep on piling up and if the storage is full, I want to delete the oldest file in the FS and use the freed up space for the latest one.

Currently, based on Hein’s recommendations in a different forum post, I am creating a binary log file of a fixed size and filling it with 0s before using it for logging. Each file is configured for a size of 512 bytes. Once it is full, the file is closed and new one is created.

I also have timestamping implemented. My project is using an RTC so I had to modify the ff_time.c file to suit my RTC driver but the FS is now able to correctly tag the files with a valid timestamp.

So my question is that is it possible to somehow search the FS for the “oldest” file so that it can be deleted? And question 2 is that will the FS become slow once the files start piling up? Based on some rough math, I could have as many as ~128,000 files stored on the FS (64GB / 512B).

I am open to any suggestions to make a better implementation of my logging mechanism too.

Thank you for your help,
Sid

Does this do what you want https://freertos.org/FreeRTOS-Plus/FreeRTOS_Plus_FAT/stdio_API/ff_stat.html ? If not, look at the implementation of that file to see how it is obtaining the time.

Scanning 128,000 files in a FAT file system is indeed a big job. I hope that you can keep the directory sizes small, because FAT is inefficient when handling large folders.
Isn’t it possible to use larger files, like e.g. 1 MB each?

Another approach could be this: scan the entire disk in the background, in a low-priority task. Now if you have a maximum age, it is easy to decide what files should be removed.

@ sidmodi: did you get closer to a solution?

I have a similar requirement. I’m doing this at the moment, but it is a work in progress:

// If necessary, delete old files to free up space.
static bool pruneSysSDCrd(ssdc_t *this) {
	uint64_t freeMB;
	unsigned freePct;
	
	if (!this->pDisk)
		return false;

	const time_t now = FreeRTOS_time(0);
	if (this->prune_time >= now)
		return false;
	
	if (!this->prune_time) {
		struct tm t;
		localtime_r(&now, &t);
		t.tm_year -= 10; // Go back ten years
		t.tm_mon = 0; // and back to Jan 1st
		t.tm_mday = 0;
		this->prune_time = mktime(&t);
	}    
	
	getFree(this->pDisk, &freeMB, &freePct);
	FF_PRINTF("%s free space: %llu MB, %u %%\n", this->mntpnt, freeMB, freePct);
	struct tm tmbuf; 
	localtime_r(&this->prune_time, &tmbuf); 
	while ((freeMB < BIN_RESERVED_MB || freePct < RESERVED_PCT)) {
		char buf[BUF_LEN];
		size_t n = snprintf(buf, sizeof buf, "%s/%d", 
			this->mntpnt, tmbuf.tm_year);
		if (!ff_chdir(buf)) {
			++tmbuf.tm_year;
			this->prune_time = mktime(&tmbuf); // Renormalizes tm
		} else {        
			n = snprintf(buf, sizeof buf, "%d/%d/%d.bin", 
				tmbuf.tm_mon, tmbuf.tm_mday, tmbuf.tm_hour);
			configASSERT(sizeof buf > n);
			FF_PRINTF("%s: Removing %s.\n", __FUNCTION__, buf);
			int ec = ff_remove(buf);
			if (ec) {
#if defined(DEBUG)
				int error = stdioGET_ERRNO();
				DBG_PRINTF("%s: %s: %s (%d)\n", __FUNCTION__, "ff_remove", strerror(error), error);
#endif            
			} else {
				// See where we stand now
				getFree(this->pDisk, &freeMB, &freePct);
				FF_PRINTF("%s free space: %llu MB, %u %%\n", this->mntpnt, freeMB, freePct);
			}
		}
		++tmbuf.tm_hour;
		this->prune_time = mktime(&tmbuf); // Renormalizes tm
		if (this->prune_time >= now - SECS_24HOUR) { // Caught up to yesterday
			FF_PRINTF("%s: Could not free up space! (Check RTC)", __FUNCTION__);
			return false;
		}
	} // while
	return true;
}

where getFree is

void getFree(FF_Disk_t *pxDisk, uint64_t *pFreeMB, unsigned *pFreePct) {
	FF_Error_t xError;
	uint64_t ullFreeSectors, ulFreeSizeKB;
	int iPercentageFree;

	configASSERT(pxDisk);
	FF_IOManager_t *pxIOManager = pxDisk->pxIOManager;
    
	FF_GetFreeSize(pxIOManager, &xError);

	ullFreeSectors = pxIOManager->xPartition.ulFreeClusterCount * pxIOManager->xPartition.ulSectorsPerCluster;
	if (pxIOManager->xPartition.ulDataSectors == 0) {
		iPercentageFree = 0;
	} else {
		iPercentageFree = (int) (( 100ULL * ullFreeSectors + pxIOManager->xPartition.ulDataSectors / 2)
				/ ((uint64_t) pxIOManager->xPartition.ulDataSectors));
	}

    const int SECTORS_PER_KB = 2;
	ulFreeSizeKB = (uint32_t)(ullFreeSectors / SECTORS_PER_KB);
    
    *pFreeMB = ulFreeSizeKB/1024;
    *pFreePct = iPercentageFree;
}

and pruneSysSDCrd is called from a task when requested by a Software Timer.

Hi @htibosch, I got busy with another aspect of the project and didn’t get time to work on this. Will start work on it again this week and will update here when I have something worth sharing.

@carlk3 thanks for sharing your solution! I will take a look at it and see if I can implement it into my project if it fits.