Freertos+tcp issue when client perform many tcp connect/disconnect back to back on the same port

Libraries
1

Hi,

I am running a python script that performs http requests, and the script open then close the connection on each request. The goal is to transfer a firmware update, so I transfer about 900+ tcp packets with a http body size of about 1024 bytes.

I use freertos+tcp version v4.2.2 and mongoose webserver 7.19.

The issue I have is that after a few packets, the mongoose library no longer get notified from select that it can accept a connection, and forcing mongoose to call FreeRTOS_accept also does not work. I see the usual tcp handshake from freertos+tcp in wirewshark (SYN, SYN+ACK,ACK), but there is just no way to get a client socket from that point on from FreeRTOS_accept.

I have the following log with some of my comments:
run python script, timeout from python script on offset=48124
I see on following log that my handler is getting called for offset=47104, so the previous call was OK (47104 + 1024 == 48124)

comProtocol_updateHandler: write mcuUpdateFw offset=47104
Socket 80 → [192.168.40.2]:10174 State eESTABLISHED->eLAST_ACK
Socket 80 → [192.168.40.2]:10174 State eLAST_ACK->eCLOSE_WAIT
Lost: Socket 80 now has 1 / 128 child
TCP: passive 80 => 2.40.168.192 port 10176 set ESTAB (scaling 0)
Socket 80 → [192.168.40.2]:10176 State eESTABLISHED->eLAST_ACK
Socket 80 → [192.168.40.2]:10176 State eLAST_ACK->eCLOSE_WAIT
Socket 80 → [192.168.40.2]:10175 State eESTABLISHED->eLAST_ACK
Socket 80 → [192.168.40.2]:10175 State eLAST_ACK->eCLOSE_WAIT
Lost: Socket 80 now has 1 / 128 child
FreeRTOS_closesocket[0ip port 80 to c0a82802ip port 10175]: buffers 16 socks 4
mongoose nbr of connections is now 2

run curl (GET request on http port 80), I see SYN,SYN+ACK,ACK in wireshark but cannot accept connection

Gain: Socket 80 now has 2 / 128 children me: 0x2200d590 parent: 0x2200c6e8 peer: 0x2200d388
prvSocketSetMSS: 1460 bytes for c0a82802ip port 10481
Socket 80 → [192.168.40.2]:10481 State eCLOSED->eSYN_FIRST
prvWinScaleFactor: uxRxWinSize 2 MSS 1460 Factor 0
Socket 80 → [192.168.40.2]:10481 State eSYN_FIRST->eSYN_RECEIVED
TCP: passive 80 => 2.40.168.192 port 10481 set ESTAB (scaling 1)
Socket 80 → [192.168.40.2]:10481 State eSYN_RECEIVED->eESTABLISHED

hit ctrl+c after a while because we are getting no response

Have any of you seen anything like this? Is there any config parameter that could prevent this?
Something I could do?

I managed to avoid the issue by not closing the connection on each http request in my script, but I would prefer for our device to be as robust as possible, because once I get the issue I can no longer get any connections on the affected port.

Thanks!

2

@nicolasb565

Is it possible to share your FreeRTOSIPConfig.h and the pcap capture?

3

Hi Tony,

Had to put spaces for it to work because I am a new user.

https: //limewire .com/d/43uhS#ZRRvi6wxY4

Thanks!

4

Hello @nicolasb565 , thank you for contacting FreeRTOS.

I can not debug the “mongoose webserver”, because I have never worked with it.

Some questions though:

  • Can you configure mongoose to reply with ‘Connection: keep-alive’? That will cause that socket stay open, to be re-used.

( keep-alive is a smart feature: it checks constantly the pool of sockets, and it will close sockets that are not used for a while ).

  • Why do you send the firmware in packets of 1 KB? Why not post 1 MB in one go?
    TCP is made to transport small and big blocks without worrying about the integrity of the data.

  • Monitoring resources: most +TCP network interfaces call the function vPrintResourceStats() regularly to warn about shrinking resources: free network buffers, free heap size, IP-buffer overflow (only when ipconfigCHECK_IP_QUEUE_SPACE is defined).

This logging looks good:

"FreeRTOS_closesocket[0.0.0.0 port 80 to 192.168.40.2 port 10175]: buffers 16 socks 4"

How is FreeRTOS_accept() called? Could you change it to the following:

    TickType_t xTimeoutTime = pdMS_TO_TICKS( 1000 );

    /* RCVTIMEO is the timeout for both recv(), recvfrom(), and accept(). */
    ( void ) FreeRTOS_setsockopt( xServerSocket, 0, FREERTOS_SO_RCVTIMEO, &( xTimeoutTime ), sizeof( TickType_t ) );

    for( ;; )
    {
        Socket_t xNew = FreeRTOS_accept( xServerSocket,
                                         &xAddress,
                                         &xAddressLength );
        if( xSocketValid( xNew ) == pdTRUE )
        {
            /* Make sure that `xNew` is served. */
        }
    }
5

Here is the ZIP file which I downloaded from the limewire:

socket_issue_many_open_close_random_data.pcapng.zip (120.4 KB)

I wondered what CPU/platform you are using?

And also, is there mongoose “port” that allows to use FreeRTOS+TCP?

Cheers

6

I modified my python script to use a Session so that if does not close the connection and it works properly in that case. I will try adding Connection: keep-alive to the header, but normally I should only add Connection: keep-alive if it was present in the request? On server side I keep the connection open unless the client close it or hang protection detect there is no activity.

I send packets in chunk of 1KB because I do not have enough RAM to store the entire firmware, so I need to write to flash in small chunks. Note I created a file with random content for the wireshark capture to avoid leaking our firmware.

Mongoose use the sockets in non-blocking mode. They use FreeRTOS_select to know when to call recv, send, accept, etc.

#define setsockopt(a, b, c, d, e) FreeRTOS_setsockopt((a), (b), (c), (d), (e))

static void mg_set_non_blocking_mode(MG_SOCKET_TYPE fd) {
#if defined(MG_CUSTOM_NONBLOCK)
  MG_CUSTOM_NONBLOCK(fd);
#elif MG_ARCH == MG_ARCH_WIN32 && MG_ENABLE_WINSOCK
  unsigned long on = 1;
  ioctlsocket(fd, FIONBIO, &on);
#elif MG_ENABLE_RL
  unsigned long on = 1;
  ioctlsocket(fd, FIONBIO, &on);
#elif MG_ENABLE_FREERTOS_TCP
  const BaseType_t off = 0;
  if (setsockopt(fd, 0, FREERTOS_SO_RCVTIMEO, &off, sizeof(off)) != 0) (void) 0;
  if (setsockopt(fd, 0, FREERTOS_SO_SNDTIMEO, &off, sizeof(off)) != 0) (void) 0;

static void mg_iotest(struct mg_mgr *mgr, int ms) {
#if MG_ENABLE_FREERTOS_TCP
  struct mg_connection *c;
  for (c = mgr->conns; c != NULL; c = c->next) {
    c->is_readable = c->is_writable = 0;
    if (skip_iotest(c)) continue;
    if (can_read(c))
      FreeRTOS_FD_SET(c->fd, mgr->ss, eSELECT_READ | eSELECT_EXCEPT);
    if (can_write(c)) FreeRTOS_FD_SET(c->fd, mgr->ss, eSELECT_WRITE);
    if (c->is_closing) ms = 1;
  }
  FreeRTOS_select(mgr->ss, pdMS_TO_TICKS(ms));
  for (c = mgr->conns; c != NULL; c = c->next) {
    EventBits_t bits = FreeRTOS_FD_ISSET(c->fd, mgr->ss);
    c->is_readable = bits & (eSELECT_READ | eSELECT_EXCEPT) ? 1U : 0;
    c->is_writable = bits & eSELECT_WRITE ? 1U : 0;
    if (c->fd != MG_INVALID_SOCKET)
      FreeRTOS_FD_CLR(c->fd, mgr->ss,
                      eSELECT_READ | eSELECT_EXCEPT | eSELECT_WRITE);
  }

void mg_mgr_poll(struct mg_mgr *mgr, int ms) {
  struct mg_connection *c, *tmp;
  uint64_t now;

  mg_iotest(mgr, ms);
  now = mg_millis();
  mg_timer_poll(&mgr->timers, now);

  for (c = mgr->conns; c != NULL; c = tmp) {
    bool is_resp = c->is_resp;
    tmp = c->next;
    mg_call(c, MG_EV_POLL, &now);
    if (is_resp && !c->is_resp) {
      long n = 0;
      mg_call(c, MG_EV_READ, &n);
    }
    MG_VERBOSE(("%lu %c%c %c%c%c%c%c %lu %lu", c->id,
                c->is_readable ? 'r' : '-', c->is_writable ? 'w' : '-',
                c->is_tls ? 'T' : 't', c->is_connecting ? 'C' : 'c',
                c->is_tls_hs ? 'H' : 'h', c->is_resolving ? 'R' : 'r',
                c->is_closing ? 'C' : 'c', mg_tls_pending(c), c->rtls.len));
    if (c->is_resolving || c->is_closing) {
      // Do nothing
    } else if (c->is_listening && c->is_udp == 0) {
      if (c->is_readable) accept_conn(mgr, c);
    } else if (c->is_connecting) {
      if (c->is_readable || c->is_writable) connect_conn(c);
      //} else if (c->is_tls_hs) {
      //  if ((c->is_readable || c->is_writable)) mg_tls_handshake(c);
    } else {
      if (c->is_readable) read_conn(c);
      if (c->is_writable) write_conn(c);
    }

    if (c->is_draining && c->send.len == 0) c->is_closing = 1;
    if (c->is_closing) close_conn(c);
  }
}
7

I use a renesas RA8M1 which is a cortex M-85 core at 480Mhz with 1MB RAM and 2MB flash.

Mongoose is configured to use FreeRTOS+TCP stack by header defines. It also uses FreeRTOS heap for memory allocation.

8

I send packets in chunk of 1KB because I do not have enough RAM to store the entire firmware, so I need to write to flash in small chunks.

Although you don’t have abundant RAM, you can receive the firmware in a continuous stream. You can still use the special markers (offset…data…) and parse them.

You can call FreeRTOS_recv() at your pace and the TCP should be patient. You will see warnings in the PCAP, informing that the “window is full”, which is normal.

Thank you for the information about Mongoose. interesting to see how a library brings many operating systems and TCP/IP implementations together.

Mongoose use the sockets in non-blocking mode. They use FreeRTOS_select to know when to call recv, send, accept, etc.

That is also my favourite way of handling sockets. Let a server task sleep in a central call to select().

It should be noted that a successful FreeRTOS_connect() will set eSELECT_WRITE, and a successful FreeRTOS_accept() will set a eSELECT_READ event.

And the eSELECT_WRITE should only be selected when you have data to send. Otherwise it will prevent select() from sleeping.

I will try adding Connection: keep-alive to the header, but normally I should only add Connection: keep-alive if it was present in the request?

Yes I think to too. A browser will ask/propose to keep the sockets open, the server can confirm this by sending:

Connection: keep-alive

On server side I keep the connection open unless the client close it or hang protection detect there is no activity.

And when looking at a PCAP, can you see that sockets have a longer life, i.e. used multiple times?

9

You should now be able to post links.

10

Yes, when the connection is kept open I see that the socket is reused for subsequent rest calls.

About not sending using chunks, it’s just because mongoose is designed to parse an entire http request, doing otherwise mean I have to handle the whole tcp stream. I tried doing that and decided it was not worth the trouble.