Multiple tasks and memory slow

[allacmc (Allan Caldas)]

I would like help in knowing how I should divide my program.

If someone with enough experience can help me, I would appreciate it.

This hardware of mine below does several things, including:

It reads 2 sensors, DS18B20 and an AM2301 sensor

In addition to showing these temperatures on the display with LVGL in gauges and texts, it also has routines that execute alarms and send them via MQTT.

I divided my program into the following tasks:

As you can see, the lowest value the memory reaches is 3372, which is very low and runs the risk of crashing.

I’m a little lost as to which tasks I should separate and which I should combine.

Here is a list of things my product does at the same time:

• Date and Time in the Queue
• Read temperature sensor A and B(ds18b20)
• Read temperature sensor AM2301
• Shows the temperatures in the queues and the Date and Time on the Display
• Datalogger (Records temperatures to the file).
• Send data via MQTT
• Alarm to monitor items.

At no time do I use global variables, I always use queues to receive the data.

A MCU usada é uma ESP32-WROM com 4MB de Flash

[richard-damon]

First, each sensor likely doesn’t need its own task, but you could easily have one task that at the desired rate polls each of the sensors to get its current reading.

Sometimes, a global variable is the best way to communicate “current” values of a sensor, with short critical sections used to update/gather the readings.

You have a lot of unused priority levels, which will cost some memory, using the CMSIS wrappers can cause this, and cause some additional memory loss to handle their wrapping. I tend to use the native FreeRTOS API.

A number of your tasks have a lot of slack in their stacks. You might try exercising the system well and see if you can reduce the stack sizes.

A well designed embedded system can use no (or maybe almost none) dynamically allocated memory after startup, so not much free heap afterwords might not be a concern.

[allacmc (Allan Caldas)]

Firstly, I would like to express my gratitude for answering my questions, thank you very much.

@richard-damon

First, each sensor likely doesn’t need its own task, but you could easily have one task that at the desired rate polls each of the sensors to get its current reading.

The problem in this case is that they are different hardware with different reading times? I didn’t want to use millis again, it seems pointless in a freertos environment.

@richard-damon

Sometimes, a global variable is the best way to communicate “current” values of a sensor, with short critical sections used to update/gather the readings.

I don’t use global variables because they say it’s not a good practice in the freertos environment.

Do you think differently?

Do you think queues consume a lot of memory?

@richard-damon

You have a lot of unused priority levels, which will cost some memory, using the CMSIS wrappers can cause this, and cause some additional memory loss to handle their wrapping. I tend to use the native FreeRTOS API.

I don’t know what this is: CMSIS wrappers? How to change this?

@richard-damon

A number of your tasks have a lot of slack in their stacks. You might try exercising the system well and see if you can reduce the stack sizes.

These tasks that have slack are because at a certain point of use it exceeds the memory value, although it is running with a certain allocation, there are moments when there is a spike in memory usage, like for example httpd which is the webserver for example.

@richard-damon

A well designed embedded system can use no (or maybe almost none) dynamically allocated memory after startup, so not much free heap afterwords might not be a concern.

I need dynamic memory, because I send emails, webhook alarms and other tasks that are not defined only at startup.

[richard-damon]

@allacmc

Firstly, I would like to express my gratitude for answering my questions, thank you very much.

@richard-damon

First, each sensor likely doesn’t need its own task, but you could easily have one task that at the desired rate polls each of the sensors to get its current reading.

The problem in this case is that they are different hardware with different reading times? I didn’t want to use millis again, it seems pointless in a freertos environment.

As long as reading from the sensors is fairly fast (even if the sensor takes a while to get the answer) then one task can keep track of when each sensor needs to be read again, and block till then. An alternative would be to setup a small queue that timers callbacks post a code to when a given sensor needs to be read. The overhead of a task stack will be larger than this sort of coding method. It is a bit more complicated, but more memory efficient.

@allacmc

@richard-damon

Sometimes, a global variable is the best way to communicate “current” values of a sensor, with short critical sections used to update/gather the readings.

I don’t use global variables because they say it’s not a good practice in the freertos environment.

Do you think differently?

Do you think queues consume a lot of memory?

The overhead of a queue isn’t that high, but can add up if you have a lot of variables. It also forces every value to exist in three places (but at different times), there is the source variable to push into the queue, the space on the queue, and the destination variable to read from the queue.

@allacmc

@richard-damon

You have a lot of unused priority levels, which will cost some memory, using the CMSIS wrappers can cause this, and cause some additional memory loss to handle their wrapping. I tend to use the native FreeRTOS API.

I don’t know what this is: CMSIS wrappers? How to change this?

Is you code calling things like vTaskCreate (which would be the direct API) or functions that begin with os (the CMSIS API). If the former, you can change the number of task priorities in the FreeRTOSConfig.h file. IF the latter, recoding to use the direct API may save you some memory. There is a cost to the generic nature of the CMSIS interface.

@allacmc

@richard-damon

A number of your tasks have a lot of slack in their stacks. You might try exercising the system well and see if you can reduce the stack sizes.

These tasks that have slack are because at a certain point of use it exceeds the memory value, although it is running with a certain allocation, there are moments when there is a spike in memory usage, like for example httpd which is the webserver for example.

Note, the stack usage report is the HIGHEST usage of the task, which is why I suggested exercising the system to get everything to use the most it is apt to use, and then check the slack, and reduce what is excessive. To my knowledge, httpd doesn’t need a lot of “stack” usage, but buffers, which are normally statically allocated. If it is using the heap to allocate more buffers, you want more free HEAP, not stack, for it to use.

@allacmc

@richard-damon

A well designed embedded system can use no (or maybe almost none) dynamically allocated memory after startup, so not much free heap afterwords might not be a concern.

I need dynamic memory, because I send emails, webhook alarms and other tasks that are not defined only at startup.

Robust embedded systems tend to NOT create new tasks in response to things, but pre-create everything at startup and have them wait for the need. That way you don’t need to worry about what to do if an allocation fails when the task is needed.

[allacmc (Allan Caldas)]

@richard-damon

As long as reading from the sensors is fairly fast (even if the sensor takes a while to get the answer) then one task can keep track of when each sensor needs to be read again, and block till then. An alternative would be to setup a small queue that timers callbacks post a code to when a given sensor needs to be read. The overhead of a task stack will be larger than this sort of coding method. It is a bit more complicated, but more memory efficient.

I understand the idea, but I think that in this context, it doesn’t interfere so much… For now I’ll try to keep it.

@richard-damon

Note, the stack usage report is the HIGHEST usage of the task, which is why I suggested exercising the system to get everything to use the most it is apt to use, and then check the slack, and reduce what is excessive. To my knowledge, httpd doesn’t need a lot of “stack” usage, but buffers, which are normally statically allocated. If it is using the heap to allocate more buffers, you want more free HEAP, not stack, for it to use.

I just learned something new. And it makes perfect sense. I have to review this code.

@richard-damon

Robust embedded systems tend to NOT create new tasks in response to things, but pre-create everything at startup and have them wait for the need. That way you don’t need to worry about what to do if an allocation fails when the task is needed.

I would love to be able to do this, but I really don’t have the memory to run so many things at the same time.

Unfortunately, I have to stop some tasks and perform others to achieve my goal.

[richard-damon]

@allacmc

I understand the idea, but I think that in this context, it doesn’t interfere so much… For now I’ll try to keep it.

The point is you are talking about being short on memory, and then say that you don’t mind wasting it to have all the seperate gathering tasks. All the stack memory being used by the various tasks that aren’t collecting data at that point of time is “wasted”, and could be reused by combining your intermittantly running collection tasks into one.

@allacmc

I would love to be able to do this, but I really don’t have the memory to run so many things at the same time.

Unfortunately, I have to stop some tasks and perform others to achieve my goal.

This is where, like the above for the sensors, partitioning your “tasks” into mutually exclusive operations that are put within one task that return to a core selector that waits for one of the operations to be needed. Once you get into the mode of creating and destroying tasks as you run, you need to figure out for EVERY case what happens if that creation fails.

[allacmc (Allan Caldas)]

The code above is sensor read:

QueueHandle_t qSensor_Display_A;
QueueHandle_t qSensor_Display_B;
QueueHandle_t qSensor_Display_C;
QueueHandle_t qSensor_Display_D;
QueueHandle_t qSensor_MQTT_A;
QueueHandle_t qSensor_MQTT_B;
QueueHandle_t qSensor_MQTT_D;
QueueHandle_t qSensor_MQTT_C;
QueueHandle_t qSensor_Datalogger_A;
QueueHandle_t qSensor_Datalogger_B;
QueueHandle_t qSensor_Datalogger_C;
QueueHandle_t qSensor_Datalogger_D;
QueueHandle_t qSensor_Alarme_A;
QueueHandle_t qSensor_Alarme_B;
QueueHandle_t qSensor_Alarme_C;
QueueHandle_t qSensor_Alarme_D;

TaskHandle_t hDisplayTempDesc = NULL;
TaskHandle_t hLerSensor = NULL;

//extern void f_DisplayTempDesc(void *parameters);
#define TEMP_BUFFER_SIZE 10
#define DS18B20_PIN 22
#define AM2301_PIN 21
#define SENSOR_DISCONNECTED_TEMP -127.00
#define TAG "SensorTask"

//static ds18b20_device_handle_t ds18b20_handle;

#define MAX_SENSORS 2  // Número máximo de sensores DS18B20

static ds18b20_device_handle_t ds18b20_handles[MAX_SENSORS];
static int sensor_count = 0;

// char* f_temperatura() {
//         float tempReceived;
//         if (xQueueReceive(qSensor_Display, &tempReceived, pdMS_TO_TICKS(100))) {
//             static char tempStr[TEMP_BUFFER_SIZE];
//             snprintf(tempStr, TEMP_BUFFER_SIZE, "%.2f", tempReceived);
//             return tempStr;
//         }
//         return NULL;
// }

void f_startLerSensor(){
    xTaskCreatePinnedToCore(v_LerSensor, "v_LerSensor", 2400, NULL, tskIDLE_PRIORITY + 1, &hLerSensor, PRO_CPU_NUM);
    xTaskCreatePinnedToCore(f_LerSensorAM2301, "f_LerSensorAM2301", 2400, NULL, tskIDLE_PRIORITY + 1, NULL, PRO_CPU_NUM);
}

void f_startStopLerSensor(){
    xTaskCreate(f_stopLerSensor, "f_stopLerSensor", 2400, NULL, tskIDLE_PRIORITY + 10, &hLerSensor);
}

void f_stopLerSensor(){
    ESP_LOGI(TAG, "Stop ler sensor");
    while(1){
        if(hLerSensor!=NULL){
            vTaskDelete(hLerSensor);
            vTaskDelete(NULL);
        }
    }
}

#define MAX_SENSORS 2 

void v_LerSensor(void *parameters) {
    // Criação das filas para cada sensor
    qSensor_Display_A = xQueueCreate(1, sizeof(float));
    qSensor_Display_B = xQueueCreate(1, sizeof(float));
    qSensor_MQTT_A = xQueueCreate(1, sizeof(float));
    qSensor_MQTT_B = xQueueCreate(1, sizeof(float));
    qSensor_Datalogger_A = xQueueCreate(1, sizeof(float));
    qSensor_Datalogger_B = xQueueCreate(1, sizeof(float));
    qSensor_Alarme_A = xQueueCreate(1, sizeof(float));
    qSensor_Alarme_B = xQueueCreate(1, sizeof(float));

    // Verificação das filas
    if (!qSensor_Display_A || !qSensor_Display_B || 
        !qSensor_MQTT_A || !qSensor_MQTT_B || 
        !qSensor_Datalogger_A || !qSensor_Datalogger_B || 
        !qSensor_Alarme_A || !qSensor_Alarme_B) {
        ESP_LOGE(TAG, "Erro ao criar as filas");
        vTaskDelete(NULL);
        return;
    }

    // Configuração do barramento 1-Wire
    onewire_bus_handle_t bus = NULL;
    onewire_bus_config_t bus_config = {.bus_gpio_num = DS18B20_PIN};
    onewire_bus_rmt_config_t rmt_config = {.max_rx_bytes = 10};
    ESP_ERROR_CHECK(onewire_new_bus_rmt(&bus_config, &rmt_config, &bus));

    onewire_device_iter_handle_t iter;
    onewire_device_t device;
    ESP_ERROR_CHECK(onewire_new_device_iter(bus, &iter));

    // Detectar e armazenar sensores encontrados
    int sensor_count = 0;
    static ds18b20_device_handle_t ds18b20_handles[MAX_SENSORS];
    while (sensor_count < MAX_SENSORS && onewire_device_iter_get_next(iter, &device) == ESP_OK) {
            ds18b20_config_t ds_cfg = {};
            if (ds18b20_new_device(&device, &ds_cfg, &ds18b20_handles[sensor_count]) == ESP_OK) {
                ESP_LOGI(TAG, "DS18B20 encontrado: %016llX", device.address);
                sensor_count++;
            } else {
                ESP_LOGW(TAG, "Dispositivo desconhecido: %016llX", device.address);
            }
    }
    ESP_ERROR_CHECK(onewire_del_device_iter(iter));

    if (sensor_count < MAX_SENSORS) {
        ESP_LOGE(TAG, "Sensores DS18B20 insuficientes detectados");
        vTaskDelete(NULL);
        return;
    }

    float temperatures[MAX_SENSORS] = {0};

    // Loop principal de leitura dos sensores
    while (1) {
        for (int i = 0; i < sensor_count; i++) {
            // Inicia a conversão de temperatura e realiza a leitura
            if (ds18b20_trigger_temperature_conversion(ds18b20_handles[i]) == ESP_OK &&
                ds18b20_get_temperature(ds18b20_handles[i], &temperatures[i]) == ESP_OK) {

                if (i == 0) {
                        xQueueOverwrite(qSensor_Display_A, &temperatures[i]);
                        xQueueOverwrite(qSensor_MQTT_A, &temperatures[i]);
                        xQueueOverwrite(qSensor_Datalogger_A, &temperatures[i]);
                        xQueueOverwrite(qSensor_Alarme_A, &temperatures[i]);
                        //ESP_LOGI(TAG, "Sensor %d - Temperatura: %.2f °C", i, temperatures[i]);
                } else if (i == 1) {
                        xQueueOverwrite(qSensor_Display_B, &temperatures[i]);
                        xQueueOverwrite(qSensor_MQTT_B, &temperatures[i]);
                        xQueueOverwrite(qSensor_Datalogger_B, &temperatures[i]);
                        xQueueOverwrite(qSensor_Alarme_B, &temperatures[i]);
                        //ESP_LOGI(TAG, "Sensor %d - Temperatura: %.2f °C", i, temperatures[i]);
                }
            } else {
                ESP_LOGE(TAG, "Erro ao ler o sensor %d", i);
            }
        }
        vTaskDelay(pdMS_TO_TICKS(50));
    }
}


void f_LerSensorAM2301(void *args) {
        ESP_LOGW(TAG, "Lendo Sensor AM2301 com sucesso");
        
        qSensor_Display_C = xQueueCreate(1, sizeof(float));
        qSensor_MQTT_C = xQueueCreate(1, sizeof(float));
        qSensor_Datalogger_C = xQueueCreate(1, sizeof(float));
        qSensor_Alarme_C = xQueueCreate(1, sizeof(float));

        qSensor_Display_D = xQueueCreate(1, sizeof(float));
        qSensor_MQTT_D = xQueueCreate(1, sizeof(float));
        qSensor_Datalogger_D = xQueueCreate(1, sizeof(float));
        qSensor_Alarme_D = xQueueCreate(1, sizeof(float));

        float temp = 0.0;
        float humid = 0.0;

        while(true){
            if (dht_read_float_data(DHT_TYPE_AM2301, AM2301_PIN, &humid, &temp) == ESP_OK) {
                    //ESP_LOGW(TAG, "Temp(%f), Humid(%f)", temp, humid);
                    xQueueOverwrite(qSensor_Display_C, &temp);
                    xQueueOverwrite(qSensor_MQTT_C, &temp);
                    xQueueOverwrite(qSensor_Datalogger_C, &temp);
                    xQueueOverwrite(qSensor_Alarme_C, &temp);
                    xQueueOverwrite(qSensor_Display_D, &humid);
                    xQueueOverwrite(qSensor_MQTT_D, &humid);
                    xQueueOverwrite(qSensor_Datalogger_D, &humid);
                    xQueueOverwrite(qSensor_Alarme_D, &humid);
                } else {
                    vTaskDelay(pdMS_TO_TICKS(500));
                    //ESP_LOGE(TAG, "Falha ao ler o sensor");
                }
                vTaskDelay(pdMS_TO_TICKS(1000));
        }
        vTaskDelete(NULL);
}

[RAc]

@allacmc

  while (sensor_count < MAX_SENSORS && onewire_device_iter_get_next(iter, &device) == ESP_OK) {
            ds18b20_config_t ds_cfg = {};
            if (ds18b20_new_device(&device, &ds_cfg, &ds18b20_handles[sensor_count]) == ESP_OK) {
                ESP_LOGI(TAG, "DS18B20 encontrado: %016llX", device.address);
                sensor_count++;
            } else {
                ESP_LOGW(TAG, "Dispositivo desconhecido: %016llX", device.address);
            }
    }

That code looks dangerous, but whether it causes harm or not depends on what ds18b20_new_device() does with its second parameter.

I will not comment on your memory usage as Richard has already pointed out everything important.

[allacmc (Allan Caldas)]

@RAc

That code looks dangerous, but whether it causes harm or not depends on what ds18b20_new_device() does with its second parameter.

What would be the error, if you could help me I would be grateful.

[RAc]

ds_cfg is a local variable. If you pass its address to somewhere in the system and this somewhere attempts to access the structure, the behavior is unpredictable and undefined once the scope is left. Also, since you re-use the variable for each iteration in the loop, all devices will try to access the same instance. Is that what you want?

[allacmc (Allan Caldas)]

Thanks for your help, I’ll review this code.