I would like to record all CAN bus data to a SD card. I will be using an ESP32 with a 40Mhz SPI SD card module. Using a 2KB buffer I can write to the SPI SD card at just over 1 MB/s, so the SD write is not a bottle neck.
The CAN bus speed is 500 Kbs and the overall data rate about 20-30 Kb/sec.
From a RTOS perspective what is the best way to do this?
• Task 1: CAN bus data > RTOS Queue
• Task 2: Get data from RTOS Queue and write to SD card
-OR-
• Task 1: CAN bus data > Large (1KB) RTOS Ring buffer
• Task 2: When ring buffer is almost full, write to SD card.
If you handle CAN bus data reception in the ISR a single task for SD card writing would be sufficient using either a custom ring buffer signaled by a task notification or a message buffer or a classic queue.
If you’re using an ISR and a corresponding post-processing task for the CAN bus data I’d use a separate SD card writing task to run in background at a lower priority than the CAN data task also connected via the same mechanism as mentioned before. Just my 2ct…
To help me learn more about RTOS, I am first going to copy the CAN bus data to a normal queue. Then copy that data to a much larger buffer (2.5 KB) and when that buffer is at 2K, write the buffer to the SD card.
It is my understanding this is the quickest way to write data to the SD card.
Once that is working, then I will add a second task to copy the CAN data to the larger buffer and when it exceeds 2KB, write it to the SD card.
Right now I suspect both ways will run the same, and will verify this soon
Well, in your initial statement you claimed that the sampling is much slower than the logging. If that’s the case, you wouldn’t really need a buffer at all. You can and it won’t hurt, but a queue/buffer is like a shock absorber that is used to balance varying loads between producer (CAN bus input) and consumer (logging output); if there is no varying load AND the producer always produces slower than the consumer consumes, you don’t need the shock absorber, right?
Anyways, if you use buffers, I would advise against your second solution because it would imply “CPU intensive” bursts which might impact lower priority tasks (you may argue that there are none, to which I would reply “probably not yet.”). If you have a choice, an even distribution of the CPU between tasks using short I/O bound computations is generally preferrable.
Thanks for the quick reply. I will keep your suggestion of “short I/O bound computations” in mind going forward, sounds like a good “guiding principle”
In this example, my untested guess would be that writing a 2KB buffer with a 240 Mhz ESP32 shouldn’t be taxing on the system.
I was only able to get the 1MB/s SD card write speed, with a 2KB buffer.
With a 1K buffer the write speed dropped to 658 KB/s.
As a learning exercise, I will test with a 2K, 1K, 512B and no buffer (just RTOS queue) and check the micros() timestamp for spikes.
Thanks to all for the input. I greatly appreciate it!
Here is what we went with:
ESP32 CAN bus frame triggers an interrupt and the CAN frame data is copied to a RTOS queue and returns.
Task #1, copies the CAN frame data from the queue, does some processing/formatting and saves it to a larger queue.
Task #2, copies the formatted data and writes it to the SD card. Every 150 CAN frames, flush() is called to write the data to the SD card.
My reason for using 2 queues is to be able to handle the variable CAN frame rate (500 - 2100 frames/sec) and to handle the delay in SD card writing (up to 100ms) when it is doing its housekeeping.
It is still being field tested, but so far it is working very well.
Thanks to hs2 and RAc for their help!
