IoT Thread Network error.

FreeOurToes wrote on July 24, 2019:

Hi,

I am using newest FreeRTOS version, always following up the master branch. ESP32 DevKit C.

My application is simple I am sending data every 15 seconds. I am using the same handles as from OTA demo app. So bascially, in the main.c I a create couple more tasks to run in parallel and once in a while I send data to the queue and in OTA app while loop, I checked the queue and send the payload. I runs smoothly and OTA updates work. But then It stops sending data and I get these messages from the log.

When payload is published succesfully.


9008 363174 [iot_thread] [INFO ][MQTT][3631740] (MQTT connection 0x3ffd5a44) MQTT PUBLISH operation queued.
9009 363174 [iot_thread] [INFO ][MQTT][3631740] (MQTT connection 0x3ffd5a44, PUBLISH operation 0x3ffd5b60) Waiting for operation completion.
9010 363178 [iot_thread] [INFO ][MQTT][3631780] (MQTT connection 0x3ffd5a44, PUBLISH operation 0x3ffd5b60) Wait complete with result SUCCESS.
9011 363178 [iot_thread] FreeHeapSize 21824. Heap low watermark: 8716. Stack high watermark: 3796.
9012 363278 [iot_thread] State: Ready  Received: 2   Queued: 2   Processed: 2   Dropped: 0
9013 363278 [iot_thread] FreeHeapSize 21828. Heap low watermark: 8716. Stack high watermark: 3796.
9014 363378 [iot_thread] State: Ready  Received: 2   Queued: 2   Processed: 2   Dropped: 

Then after a while


9522 384105 [iot_thread] [INFO ][MQTT][3841050] (MQTT connection 0x3ffd5a44) MQTT PUBLISH operation queued.
9523 384105 [iot_thread] [INFO ][MQTT][3841050] (MQTT connection 0x3ffd5a44, PUBLISH operation 0x3ffd5b60) Waiting for operation completion.
9524 384106 [iot_thread] [INFO ][MQTT][3841060] (MQTT connection 0x3ffd5a44, PUBLISH operation 0x3ffd5b60) Wait complete with result NETWORK ERROR.
9525 384106 [iot_thread] FreeHeapSize 55428. Heap low watermark: 8716. Stack high watermark: 3796.
9526 384206 [iot_thread] State: Ready  Received: 2   Queued: 2   Processed: 2   Dropped: 0
9527 384206 [iot_thread] FreeHeapSize 56020. Heap low watermark: 8716. Stack high watermark: 3796.
9528 384306 [iot_thread] State: Ready  Received: 2   Queued: 2   Processed: 2   Dropped: 0

And eventually


11962 483119 [iot_thread] [INFO ][MQTT][4831190] (MQTT connection 0x3ffd5a44) MQTT PUBLISH operation queued.
11963 483119 [iot_thread] [INFO ][MQTT][4831190] (MQTT connection 0x3ffd5a44, PUBLISH operation 0x3ffd5b60) Waiting for operation completion.
11964 483121 [iot_thread] [INFO ][MQTT][4831210] (MQTT connection 0x3ffd5a44, PUBLISH operation 0x3ffd5b60) Wait complete with result NETWORK ERROR.
11965 483121 [iot_thread] FreeHeapSize 55624. Heap low watermark: 8716. Stack high watermark: 3796.
11966 483221 [iot_thread] State: Ready  Received: 2   Queued: 2   Processed: 2   Dropped: 0
11967 483221 [iot_thread] FreeHeapSize 56020. Heap low watermark: 8716. Stack high watermark: 3796.
11968 483321 [iot_thread] State: Ready  Received: 2   Queued: 2   Processed: 2   Dropped: 0
11969 483321 [iot_thread] FreeHeapSize 56020. Heap low watermark: 8716. Stack high watermark: 3796.
11970 483421 [iot_thread] State: Ready  Received: 2   Queued: 2   Processed: 2   Dropped: 0
11971 483421 [iot_thread] FreeHeapSize 56020. Heap low watermark: 8716. Stack high watermark: 3796.
11972 483521 [iot_thread] State: Ready  Received: 2   Queued: 2   Processed: 2   Dropped: 0
11973 483521 [iot_thread] FreeHeapSize 56020. Heap low watermark: 8716. Stack high watermark: 3796.
11974 483621 [iot_thread] State: Ready  Received: 2   Queued: 2   Processed: 2   Dropped: 0
11975 483621 [iot_thread] FreeHeapSize 56020. Heap low watermark: 8716. Stack high watermark: 3796.
11976 483721 [iot_thread] State: Ready  Received: 2   Queued: 2   Processed: 2   Dropped: 0
11977 483721 [iot_thread] FreeHeapSize 56020. Heap low watermark: 8716. Stack high watermark: 3796.
11978 483821 [iot_thread] State: Ready  Received: 2   Queued: 2   Processed: 2   Dropped: 0
11979 483821 [iot_thread] FreeHeapSize 56020. Heap low watermark: 8716. Stack high watermark: 3796.
11980 483853 [iot_thread] [ERROR][MQTT][4838530] (MQTT connection 0x3ffd5a44) Failed to send PINGREQ.
11981 483853 [iot_thread] [INFO ][MQTT][4838530] (MQTT connection 0x3ffd5a44) Network connection closed.
11982 483921 [iot_thread] State: Ready  Received: 2   Queued: 2   Processed: 2   Dropped: 0
11983 483921 [iot_thread] FreeHeapSize 56048. Heap low watermark: 8716. Stack high watermark: 3796.
11984 484021 [iot_thread] State: Ready  Received: 2   Queued: 2   Processed: 2   Dropped: 0
11985 484021 [iot_thread] FreeHeapSize 56048. Heap low watermark: 8716. Stack high watermark: 3796.
11986 484121 [iot_thread] State: Ready  Received: 2   Queued: 2   Processed: 2   Dropped: 0
11987 484121 [iot_thread] FreeHeapSize 56048. Heap low watermark: 8716. Stack high watermark: 3796.
11988 484221 [iot_thread] State: Ready  Received: 2   Queued: 2   Processed: 2   Dropped: 0
11989 484221 [iot_thread] FreeHeapSize 56048. Heap low watermark: 8716. Stack high watermark: 3796.
11990 484321 [iot_thread] State: Ready  Received: 2   Queued: 2   Processed: 2   Dropped: 0
11991 484321 [iot_thread] FreeHeapSize 56048. Heap low watermark: 8716. Stack high watermark: 3796.
11992 484421 [iot_thread] State: Ready  Received: 2   Queued: 2   Processed: 2   Dropped: 0
11993 484421 [iot_thread] FreeHeapSize 56048. Heap low watermark: 8716. Stack high watermark: 3796.
11994 484521 [iot_thread] State: Ready  Received: 2   Queued: 2   Processed: 2   Dropped: 0
11995 484521 [iot_thread] FreeHeapSize 56048. Heap low watermark: 8716. Stack high watermark: 3796.
11996 484584 [i2c_app] GF: 23.375000, NODE: 28.625000, X: 0.842965, Y: -0.957915, Z: -9.540829, POS: 26520
11997 484584 [i2c_app] FreeHeapSize 56244. Heap low watermark: 8716. Stack high watermark: 1848.
11998 484584 [i2c_app] "node_temp":28.625000,"gf_temp":23.375000,"x":0.842965,"y":-0.957915,"z":-9.540829,"position":26520,"load_1":-469532,"load_2":875496
11999 484621 [iot_thread] State: Ready  Received: 2   Queued: 2   Processed: 2   Dropped: 0
12000 484621 [iot_thread] queue found "node_temp":28.625000,"gf_temp":23.375000,"x":0.842965,"y":-0.957915,"z":-9.540829,"position":26520,"load_1":-469532,"load_2":875496 
12001 484622 [iot_thread] [WARN ][MQTT][4846220] (MQTT connection 0x3ffd5a44) Attempt to use closed connection.
12002 484622 [iot_thread] [ERROR][MQTT][4846220] (MQTT connection 0x3ffd5a44) New operation record cannot be created for a closed connection
12003 484622 [iot_thread] FreeHeapSize 55656. Heap 


And then it only repeats


12037 486122 [iot_thread] [WARN ][MQTT][4861220] (MQTT connection 0x3ffd5a44) Attempt to use closed connection.
12038 486122 [iot_thread] [ERROR][MQTT][4861220] (MQTT connection 0x3ffd5a44) New operation record cannot be created for a closed connection
12039 486122 [iot_thread] FreeHeapSize 55460. Heap low watermark: 8716. Stack high watermark: 3796.
12040 486222 [iot_thread] State: Ready  Received: 2   Queued: 2   Processed: 2   Dropped: 0

In between of these messages I did not received any other debugging out put as wifi was dropped or anything else. Besides the errors you can notice the FreeHeapSize changes from 21828 to 56020 But it just happens without any other message in between.

Thanks

abhijit-AWS wrote on July 31, 2019:

Hello FreeOurToes ,

Apologies for the late response.
Is it possible for you to get wire trace when this condition happens? You have mentioned that you are creating couple of more tasks. Can you please share what these tasks are doing ? I’m trying to determine if remote connection got closed for some reason.
Thanks.