What would be the best approach to handle images being acquired at a fast rate? I have a freeRTOS task which acquires an image at 6Hz and I relay a message to another task to copy the image, perform some processing, and then try to fill a fixed size buffer (holds 24 images) with pixel information. I am encountering an issue where my my acquisition rate is much faster that my processing so in turn when a new image is ready and available, the message I send to process the new image is dropped. Would using a separate queue and having a task act on that queue for processing aid with this or should I restructure my existing task?
Hi @khuang
Is it possible for you to do some sort of profiling to see what part of the processing task is most time consuming? That will help us focus on the right part to optimize. Also, why do you need to create a copy of the acquired image?
Is using DMA an option or can you employ zero copy, as Rahul suggested?
generally, buffering and queueing will only help you if on the average, you can process the incoming data timely. If that is the case, you need to determine how many buffers to provide to buffer your worst case latency. If you can not serve the throughput either for computing speed or memory restrictions and can not afford to drop frames, your hardware platform is not apt for your platform, and you will need to modify your hardware (faster MCU, more memory, graphics coprocessor…) it is rather straightforward.
In a compatible project, my customer decided on restricting the maximum frame size to allow the existing hardware to serve the throughput restrictions.
I am working specifically with Espressif’s ESP32-S3. My current processing consists of converting flattened unsigned short image of size 1024 to int8_t and then taking four smaller sized flattened images of size 100 from the original image. This currently cause one skip of an image for processing. I create a copy of my image within an image processing task as I have a second acquisition task which only acquires images. The acquisition task relays a message to the image processing task when there is an image available and then copies the image. Later I would like for any other tasks which needs images to accept the “image available” message and use the image when needed.
I am using and ESP32-S3, and it looks to have compatibility with DMA (Memory Types, in their ESP-IDF API). I am not familiar with DMA, what does it provide?
My images are fixed at size 1024 of type unsigned short. My image acquisition task currently runs at 6Hz but later we would like to have it down to 4Hz. My reply to Rahul provides more context on what type of processing happens on the image.
DMA (direct memory access) allows the hardware component (over which interface are the images aquired?) to store data in memory without MCU intervention. Needs to be implemented in the driver, provided the peripheral supports it.
If images are coming in at 6 Hz, and processing an image takes more that 1/6th of a second, there is nothing you can do but drop images. If some take longer but the average is less than 1/6th of a second, or if sometimes you get faster bursts, then a queue can help, and the size is based on the statistics of the timing (how many images to get the average to what you can handle). DMA can sometimes help with processing time, as the DMA can move the data with minimal processor overhead. If by size 1024 you mean 1024x1024, that is a lot of data to be processing (but not out of the question), but you do need to be careful with operations and avoiding unneeded copying.
I am currently using a thermopile sensor and using their propriety code to retrieve images. I am not sure if it supports DMA access. If they do, are you saying this would allow me to save some of processing for image processing?
I should of specified that my image from my sensor is of size 32x32, but flattened it would be 1024 bytes. Could my current messaging protocol of having a copy of the image in each task be a limiting factor? If I can preprocess my image within the task that acquires the image and still retain my capture rate could that be a better approach?
Making many copies of even a somewhat large object like that takes CPU time, and that uses up your CPU bandwidth for processing.
A better solution is normally to have a predefined set of buffers for images, and just pass the buffer address down the chain to avoid all the extra copying. Remember, you are copying TWICE for each transfer, once into the queue, and once out of it.