FreeRTOS+TCP V2.3.3: FreeRTOS_send does not return after RST on a slow link (follow-up to GitHub issue #1373)

Libraries
1

Hi Hein,

This continues [BUG] TCP close stuck: FIN suppressed because the stored FIN sequence does not match the segment being sent · Issue #1373 · FreeRTOS/FreeRTOS-Plus-TCP · GitHub.

GitHub has been delivering this issue as email, and I have been answering those messages. I guess GitHub was requoting the previous response, which is why the earlier notes quoted the thread. I have created an account on the FreeRTOS forum so that I can post the follow-ups here, as you asked.

These are the points already answered on 14 October:

  • Azure IoT Hub does not answer ICMP, so a ping of `iot-prod1.azure-devices.net` does not produce a result.
  • There is one MQTT client (azure-iot-sdk-c 1.3.8). A new TCP port is opened only after the previous socket is already dead.
  • Local MSS stays 1160 (`ipconfigNETWORK_MTU` 1200). `MSS change 1160 → 1460` is the value Azure advertises.
  • `ipconfigTCP_KEEP_ALIVE` is 1, interval 20 seconds.
  • `shutdown_socket()` from your 10 October note is already in both the application and the bootloader. It checks the socket before `FreeRTOS_shutdown()`, then calls `FreeRTOS_closesocket()`.

These are the new questions:

  • The CPU is a Microchip ATSAME70Q21B (Cortex-M7), on our own board, using the GMAC.
  • `vPrintResourceStats()` was not being called. The MAC task will call it on each wake. `ipconfigCHECK_IP_QUEUE_SPACE` is 1.
  • There is no hub in the path.

This is a failure at a client site whose network is running poorly. Capturing logs from that site is not possible. The slow network in the lab is how I reproduce the failure I am seeing there. It is the reproduction of the field problem, and it is the reason I am running it.

These are the changes going into the next run. I will send you the failure log from it:

  • TCP lines will be prefixed with `xTaskGetTickCount()`.
  • `shutdown_socket()` will use your later copy: `xSocketValid()`, the `shutdown_socket: start` and `shutdown_socket: done` lines, a wait of up to 5 seconds, then `FreeRTOS_closesocket()`.
  • The MAC task will call `vPrintResourceStats()`.

The call that does not return is `FreeRTOS_send`. The log shows `FreeRTOS_send: … no space`, then the hub RST, then `eESTABLISHED → eCLOSED`, and the caller does not come back.

Steve

2

Hi @SteveCarlisle , thank you joining the forum.

You wrote:

Azure IoT Hub does not answer ICMP, so a ping of `iot-prod1.azure-devices.net` does not produce a result.

This I see on Windows, it finds a CNAME:

ping -n 300 -w 1000 iot-prod1.azure-devices.net

Pinging gateway-prod-gw-eastus-5-g2.eastus.cloudapp.azure.com [40.79.156.141] with 32 bytes of data:
Reply from 40.79.156.141: bytes=32 time=255ms TTL=103
Reply from 40.79.156.141: bytes=32 time=256ms TTL=103
…

There is one MQTT client (azure-iot-sdk-c 1.3.8). A new TCP port is opened only after the previous socket is already dead.

Why would a socket go dead*? You only need to connect once to your MQTT broker and use the same socket for ever. Just make sure that it is also connected at the MQTT level.

*See below: FreeRTOS_send() gets an error.

Local MSS stays 1160 (`ipconfigNETWORK_MTU` 1200). `MSS change 1160 → 1460` is the value Azure advertises.

MSS: Better too low than too high :slight_smile:

`shutdown_socket()` from your 10 October note is already in both the application and the bootloader. It checks the socket before `FreeRTOS_shutdown()`, then calls `FreeRTOS_closesocket()`

Be careful with FreeRTOS_closesocket(), it must be called 1 time only (just like calling free()).

The CPU is a Microchip ATSAME70Q21B (Cortex-M7), on our own board, using the GMAC.

Nice one! As soon as you have logging with timestamps, I am curious to see how long “TLS” takes.

The slow network in the lab is how I reproduce the failure I am seeing there. It is the reproduction of the field problem, and it is the reason I am running it.

Yes, I would like to propose two steps:

  1. Try to get a MQTT client that works perfectly, day and night.
  2. Run the device in the designated location.

Q: what is so bad about your internet? That’s why I asked you to run with ping. Does your Internet have a slow throughput? Does it go up and down?

vPrintResourceStats()

That function logs when any of the resources is getting low. I use it in all my projects.

The call that does not return is `FreeRTOS_send`. The log shows `FreeRTOS_send: … no space`, then the hub RST, then `eESTABLISHED → eCLOSED`, and the caller does not come back

That’s interesting.

I think that you have plenty of RAM? You did work with FREERTOS_SO_WIN_PROPERTIES, didn’t you?

It allows you to set the buffer sizes for both Rx and Tx:

    WinProperties_t xWinProperties;
    const size_t MSS = 1460;
    xWinProperties.lTxBufSize = 6 * MSS; /* Units of bytes. */
    xWinProperties.lTxWinSize = 3;       /* Size in units of MSS */
    xWinProperties.lRxBufSize = 6 * MSS; /* Units of bytes. */
    xWinProperties.lRxWinSize = 3;       /* Size in units of MSS */
    FreeRTOS_setsockopt(
        xSocket,
        0,
        FREERTOS_SO_WIN_PROPERTIES,
        ( void * ) &xWinProperties,
        sizeof( xWinProperties ) );

This setup is very detailed. It helps to tune the TCP performance. When it is too much, use lower values.

One final question, why is the Internet so bad?

I installed a managed switch TL-SG108 to see the conversation between my DUT and the MQTT server. It is a real switch, but it can also mirror ports and see all packets.

Cheers

3

Hein —

We’ve tracked down the root cause of the Hub / network lockups we’d been seeing on the Device (including the customer units and throttled-link cases).

When we moved from lwIP to FreeRTOS-Plus-TCP, we carried over a socket-error check from the old stack. That check was written against lwIP’s / BSD-style error polarity. FreeRTOS-Plus-TCP returns the errno **negated** on the API (e.g. closed socket → `-pdFREERTOS_ERRNO_ENOTCONN`), whereas the inherited logic was still looking for the opposite sense of those constants.

Because of that mismatch we were **never actually detecting** the fatal condition (socket gone / not connected). Code paths that should have broken out of drain / DoWork receive loops and torn down the Hub connection simply kept going. The device stayed marked “connected,” MQTT/TLS stalled, and we saw the multi-minute DoWork hangs, Expired SAS, and reconnect thrash in the field logs.

We’ve corrected the polarity on those checks so a closed socket is recognized and we exit those loops and recover properly. That’s now in the recent Device builds we’ve been validating.

Steve

4

Hi Steve,

Congratulations and thank you for reporting back. These kind of reports can be very useful for other FreeRTOS developers too.

So the shutdown procedure is now running smoothly for you?

My attitude was : let’s first get this working on a healthy network, and once that looks good (as in my STM32) we can do other experiments.

A good side-effect for me: I finally created a working setup with TLS for HTTPS and MQTT.

Never hesitate to ask more here on this Forum. It’s got a much bigger “audience” than Github.

Hein