we’re using FreeRTOS+FAT for storing sensor data. One FreeRTOS task is collecting data and writing it to a file, while another task is reading from it and uploading data to a server. The data is chunked, and we’re currently writing one line in the file for each chunk. We’re keeping a separate read pointer to the file to be able to switch between read and write mode for the two processes (The often run parralell for “live streaming”). We’re basically using the file as a large FIFO with a binary semaphore.
The above solution works well, but eventuelly the file will become to large as no data is ever removed. Also, if the MCU should restart for some reason (power outage, WDT-reset etc.), it will upload all data ever collected. We could ofcourse empty the file each time the read pointer reaches EOF, but that would not solve the problem where the MCU restarts when only half the data is uploaded. Everything would then be re-uploaded.
So, we need some way of removing data that has allready been uploaded. The ideal solution would be to “truncate” with an offset. That is, if the file is N bytes large, and we upload the first X bytes, it would be perfect to truncate it to (N-X) bytes with X bytes offset. I don’t see any way this is possible with the current API.
Another solution would be to somehow prepend data to the file, and thus use the regular ff_truncate function for removing uploaded data from the end.
We’ve initially tried using one file per datachunk, but the FS became extremely slow after a few thousand files.
Does anyone have a good solution to this problem? Maybe there’s a clever/obvious solution I haven’t though of.