Collect sensor data from different task

Kernel
1

hi everyone ,

i am using FreeRTOS in esp32 framework esp idf c language , i have 8 task max to read 8 sensor i2c software (with https://github.com/UncleRus/esp-idf-lib library )run at same time with in local value (task variable) so i want to collect those data variabe with one fonction in real time without block any task from them.

i dont know how i can manage that,

so the number of task runing depend of number of sensor is connected to esp 32 and variable in each task depend too of type of this sensor

what is better queue or use extern variable for all sensor and i can read an struct them in read task file

Capture
Capture1765×720 315 KB

2

You should be able to read the sensors in interrupts, with very little task interaction. If you do have eight values in eight different tasks then the simplest way, assuming no security concerns on the data, is to simply have eight global variables. As long as the variables can be written to and read from atomically (by which I mean, in one go, so reading a 32-bit value on a 32-bit architecture would be atomic, but reading a 32-bit value on a 16-bit architecture would not be atomic as it would take two separate reads to get all 32-bit values), and there is no more than one task or interrupt writing to the variables, then it will be safe.

3

I first question for me is that you say you don’t want to block the various tasks that are using the sensor value but do those tasks actually have anything productive that they can do while waiting for the next sensor value?

A second question is about the I2C Bus bandwidth, is it ample or are you going to need to be careful about scheduling sensor reading to meet your requirements?

If it does make sense to create a Sensor Reading task to get all the sensor values, it likely makes sense to broadcast the values with globals. Queues might make more sense if you needed a central sensor task but wanted the other tasks to block for a new value, then they could wait on the Queue for the value, but even then a global and a direct-to-task notification might make more sense.

4

@rtel @richard-damon thanks for ur replay,

for me i want detect automatically sensor connected with i2cscanner bcs i know the sensor used (i have 5 or 6 type for now :bme280,BH1750,MCP3428,PFC8574…) to my device and start task for each sensor to send data with file json to server mqtt.

for that i want add all my library sensor want use and create for each library 8 task with 8 BUS i2c

like :

bme280_task1 (bus1) -bme280_task2(bus2)… bme280_task8(bus8)
BH1750_task1 (bus1) -BH1750_task2(bus2)…BH1750_task8(bus8)
MCP3428_task1(bus1) -MCP3428_task2(bus2)…MCP3428_task8(bus8)

i dont know if there is better method, the purpose is i can connect dynamically my sensor (like only 8 bme280 or only 2 bme280 or 3 bme280 and 2 adc3428 and 3 pcf …) and send data with one JSON file or maybe its easy to send data witch each task with MQTT TOPIC …

i want use one tak for each sensor to make sensor independent , if one sensor is broked i send notification to server and other sensor (task) work normally

5

Your MCU has 8 I2C controllers ? Or are the devices connected to the same bus ?
However, I’d probe for connected I2C devices in a main task and setup a table of the devices found. That’s an optimization to avoid useless tries to read from devices not connected later on.
If you take the approach of collecting all available sensor data and publish them by 1 JSON file this main task can walk through the table fetching data from each sensor, send the file when done and start over. Timing / update frequency can be easily adjusted e.g.by simple vTaskDelay. I’d even use most simple, non-interrupt driven I2C accesses.
Beware that I2C bus is stateful and doesn’t support hot-plugging. So dynamically un/plug sensors at runtime would be a serious problem if you have this in mind.
You could decouple the collector task from MQTT communication task if needed.
The signaling scheme depends on your requirements. Maybe a simple notification ‘JSON file ready’ is sufficient. The collector task should then wait for a ‘continue collecting’ notification/ acknowledge to proceed to avoid overwriting the JSON data being sent.

6

@hs2,

yes iam using 8 soft i2c not same bus using i2cdev controller https://github.com/UncleRus/esp-idf-lib that work well for now

i dont understand well how i can get value with main task (walk through the table fetching data from each sensor) if i have 8 same sensor connected like 8 bme280 , i read name of sensor in table of the devices found and i call library and i store data

i want something look like this code for JSON file and mqtt publish :

and i have 2 mode
mode 1 : send data each 10sec
mode2 : send data if sensor change value (+/- 5 for exemple )

7

Sorry, I’ve no idea about the MQTT lib etc. you’re using.
But what about just periodically reading sensor data one by another, waiting a bit using vTaskDelay in between and storing them in a struct/array ? Simply check for a sensor change after reading with the previously stored value and set a changed flag.
Check the time expired e.g. by xTaskGetTickCount and publish an update when appropriate if the changed flag was set (clear it afterwards) or after 10 sec depending on the mode.

8

One big issue you will run into is that ‘soft’ I2C controllers will be somewhat slow, and very demanding on CPU time. It will also tend to require that the devices support arbitrarily slow access (unless you make the i2c task the highest priority) as if it gets pre-empted the transaction will pause. (And if you can have the i2c task as highest priority, that almost shows you don’t really need a RTOS but the program could as easily be just a one big loop program.

@hs2, yes, technically I2C isn’t hot swappable in theory, but in practice it can be. If devices get power before they connect to the bus, it practically eliminates the problems of corrupting the cycle in progress. If the plugged in device is just a slave, it will get itself synced on the next Start signal. Even if you don’t get the power connection right, each device being on its own bus, means that a connection/disconnection won’t disrupt any other device.

9

@richard-damon,

i use same priority 5 for each one :

xTaskCreatePinnedToCore(bmp280_socket1, "bmp280_socket1", configMINIMAL_STACK_SIZE * 8, NULL, 5, NULL, APP_CPU_NUM);
xTaskCreatePinnedToCore(bmp280_socket2, "bmp280_socket2", configMINIMAL_STACK_SIZE * 8, NULL, 5, NULL, APP_CPU_NUM);
xTaskCreatePinnedToCore(bmp280_socket3, "bmp280_socket3", configMINIMAL_STACK_SIZE * 8, NULL, 5, NULL, APP_CPU_NUM);
xTaskCreatePinnedToCore(bmp280_socket4, "bmp280_socket4", configMINIMAL_STACK_SIZE * 8, NULL, 5, NULL, APP_CPU_NUM);

//xTaskCreatePinnedToCore(bmp280_socket5, "bmp280_socket5", configMINIMAL_STACK_SIZE * 8, NULL, 5, NULL, APP_CPU_NUM);

//xTaskCreatePinnedToCore(bh1750_socket6, "bh1750_socket6", configMINIMAL_STACK_SIZE * 5, NULL, 5, NULL, APP_CPU_NUM);
//xTaskCreatePinnedToCore(bh1750_socket7, "bh1750_socket7", configMINIMAL_STACK_SIZE * 5, NULL, 5, NULL, APP_CPU_NUM);
//xTaskCreatePinnedToCore(bh1750_socket8, "bh1750_socket8", configMINIMAL_STACK_SIZE * 5, NULL, 5, NULL, APP_CPU_NUM);

and that work well, i print data with task consumer with extern globale variable
i need just to manage those variable dynamically in terms of nomber and type of sensor connected

if i have 8 sensor file json look like :

Sensor1
var x
var y
var time

sensor1
var x
var y
var z
var time
.
.
.
sensor8
var x
var y
var z
var f
var time

10

You’re right Richard and a bus lock can be handled pretty straight forward if the slaves are power controlled or if you have a dedicated bus to each device. Unfortunately the HW I had to deal didn’t support those desirable features :frowning:
It gets worse if they are not power managed and a transfer is interrupted/stuck and you need to unlock the bus by software. However, this is doable as well, but more complicated…

11

The simplest why I have found to unblock the bus is to simply drop and then a bit later raise the SDA line with the SCL line high, that issues a START (or RESTART) then a STOP state, which will clear the state of the bus. If the SDA line is current low, that means that some slave thinks it is being read from, so you just cycle SCL low than high until the SDA line goes high (which shouldn’t take more than 8 cycles). No slave should hold SCL low indefinitely, so you should always be able to do one of those in a reasonable time frame. All you need to do this are two GPIO on the processor, and often you can switch the pins of the I2C controller into GPIOs when needed. Normally, you won’t need to do this, as normally the worse that happens is that plugging the device on the bus generates a Start state, that make your master think the bus is busy, and a simple controller reset will get you back in control, and the first Start state by the controller gets all the slaves back in sync.

1 Like
12

Good idea - thanks Richard !
I’m aware about reassigning I2C lines as GPIOs to bit-bang the bus out of lock. I’ve only found the ‘9 SCL/bit’ workaround last time I searched for it.
The better I2C devices indeed have such a bus timeout which helps a lot. Unfortunately some (rather dumb) devices don’t acc. to their data sheets.

13

Capture
Capture1765×720 322 KB

Application for Fire Alarm Projec

please, some one can can help me to find the best method to create json file depend table of device found , how i can change json in terms of number and type of sensor connect and i want send data each 10s and when the value change with 5 unity (dont wait 10s to send data i need send it immediately )

maybe this can help : https://www.freertos.org/FreeRTOS_Support_Forum_Archive/September_2010/freertos_adding_task_dynamically_3836101.html

14

The only devices that need to have a bus timeout are those that can hold the clock signal to hold the bus (most slaves don’t need it). Any device that holds SCL indefinitely is inherently broken, as the only reason to hold the clock is to wait to have data ready, which should take a finite time. The only times I have seen the issue is a ‘smart’ device with a program bug that has ‘forgotten’ about the I2C request.

15

My first thought is WHY do you need JSON? JSON is a data format designed to communicate data structures across a text stream. For a ‘small’ machine like what you seem to be describing, I would just use some in-memory structures, and not bother trying for some high level device independent format to record it in. Think about your REAL problem, what is it that you ACTUALLY need to do (and not how you think you might want to solve it).

Some of the items to think about:
How do you detect that a device has been added?
How do you detect what type of device that is?

16

hi yes i know i want json bcs i want send data with mqtt and is not that my problem,

for detect device i use scanne each socket (bus) i2c and i get adresse of sensor connect and with this adress i know the type of sensor (bcs i dont have many sensor so i can give for each sensor one adress fixe ) that work i can creat table of device found like in the last pictures i have 8 case (case1= adress sensor in socket 1,case2= adress sensor in socket 2…case8= adress sensor in socket 8)

my problem before collect data is how i can read data (if we change composition sensor we need to reboot esp32 to make update for table and start to get new sensor data )

i want something like this https://microcontrollerslab.com/arduino-freertos-structure-queue-receive-data-multiple-resources/

but with sensor detected (table device found)

17

I afraid I don’t exactly understand the problem you want to solve…
Maybe I’m wrong but referring to the Arduino example you could of course send sensor data as structs to another (MQTT handler) task perhaps with a special end marker struct sent by the last sensor task in the table to tell the MQTT task to finally create the JSON file or whatever of the collected sensor data and send it.
Define a struct that covers all possible sensor data (e.g. using a union) and kind of message ID to distinguish between sensor data and end marker struct to implement a simple communication protocol between those tasks and you’re done :wink:
Beware that the queue might get full if the MQTT task runs into networking trouble.
Hence I’d use kind of acknowledge/sync mechanism in a way that the sensor tasks wait on e.g. a task notification or doing a self-suspend after enqueuing its data and let the MQTT task restart/resume the sensor tasks after processing/flushing all sensor data items in the queue and sending the JSON data successfully.

18

@hs2 yes sorry my requirements are not clear. but before that , i want my code adapt for each restart with table of device found ( if i have one sensor or 2 or 5 or 8 max )

in my application i have 4 sensor (maybe i will add another type ) (bme280,bh1750,mcp3428,and pcf8574 ) using software i2c and the customer can plugin what he want in 8 socket (same type or different type) and my code can adapt automatic with device connecting and initialize and start read those device detected (if one or more are Unplug the code send error in topic and stop to read )

19

Even when watching/reading all sockets supporting plug and play wouldn’t change the approach described. You might extend the sensor struct/messaging accordingly.
Once the main/collector task receives an unplug message it might just resume the sensor tasks already processed and start-over waiting for incoming sensor report messages. Maybe sending an appropriate MQTT error before…
Or just mark the entry of the unplugged/failed sensor of the current sequence, complete the sequence and send a final report…
Using a simple state machine (in this case probably just a flag) in the sensor tasks helps to track the state transitions between plugged - unplugged to generate the corresponding sensor message.

20

First let me comment that what exactly you are trying to do seems to be coming off a bit muddled, with a number of different goals getting mixed into the mess. Also, most of this issue isn’t so much a FreeRTOS question, but a question on general program organization, so not really appropriate here.

The first thing that needs to be done is to prepare a requirements listing. What do you need this unit to do, with some clearly spelled out specifications. You seem to have several different sets of these requirements, detecting which and how many of different types of sensors, and send that data out over a communications link.

Note that these operations are at very different ‘layers’ of system description, and thus are likely dealt with at very different parts of the program. The piece of the program that is figuring out what devices are present on the I2C busses shouldn’t also be worrying about sending that data to something in the outside world but should be just setting up information to let the program itself understand what is there, and a communication module then does the formatting of the data (like into JSON) to tell it to some other device. What data it needs to gather will come out of your systems requirement analysis.

You are then setting up your system in a way that may be causing you some additional issues. You are using a ‘Real-Time’ OS, but then hobbling it by using a number of software-based I2C channels. Software-based I2C tends to imply low-performance requirements, but a RTOS implies tighter requirements. You are using a multiprocessor chip, but an OS that was really designed for a single processor system, and then using an offshoot branch of the OS that was cobbled to handle that chip, so much of the expertise here isn’t as useful for that system.