FreeRTOS+TCP: How to use the FREERTOS_SO_REUSE_LISTEN_SOCKET option

edeviser wrote on Monday, March 19, 2018:

Hello,

I am running FreeRTOS+TCP and I have some trouble on setting up my server. The server goes like this:

void socketTask(void * pvParameters) {
  // Create Listen Socket
  c->listen_socket = FreeRTOS_socket(FREERTOS_AF_INET, FREERTOS_SOCK_STREAM, FREERTOS_IPPROTO_TCP);
  if ( c->listen_socket == FREERTOS_INVALID_SOCKET)
    return -1;

  // Bind listen Socket
  if (FreeRTOS_bind(c->listen_socket, &c->listen_addr, sizeof(c->listen_addr)))
    return -1;

  // Set Timeout
  setRxTimeout(c, portMAX_DELAY);
  setTxTimeout(c, portMAX_DELAY);

  // Set Reuse socket option
  const BaseType_t reuseSocket = pdTRUE;
  if (FreeRTOS_setsockopt(c->listen_socket, 0, FREERTOS_SO_REUSE_LISTEN_SOCKET,
      &reuseSocket, sizeof(reuseSocket)))
    return -1;
  c->remote_sock = c->listen_socket;  // Already the same
  
for( ;; ) {
    ret = FreeRTOS_listen(c->listen_socket, 1);
    if (ret) {
      DEBUG("Error (%d) listen...\n", ret);
      vTaskDelay(100);
      continue;
    }

     c->remote_sock = FreeRTOS_accept(c->listen_socket, &c->remote_addr, &l);
    if (c->remote_sock == NULL) {
      DEBUG("Cannot Accept Error Code %d\n", (int)c->remote_sock);
      vTaskDelay(100);
      continue;
    }

    /* Reading and Writing is done by some other service tasks */
    
    // Wait for disconnect
    SocketSet_t waitDisc = FreeRTOS_CreateSocketSet();
    FreeRTOS_FD_SET(c->remote_sock, waitDisc, eSELECT_EXCEPT);
    FreeRTOS_select(waitDisc, portMAX_DELAY);
    DEBUG("connection terminated to %lu.\n", c->remote_addr.sin_addr);

    FreeRTOS_closesocket(c->remote_sock);

    taskYIELD();
  }

To test my server I implemented a little client on my ubuntu host machine:

int main(int argc, char *argv[]) {
    char recvBuff[1024];
    struct sockaddr_in serv_addr; 

    if(argc != 4) {
        printf("\n Usage: %s <ip of server> <port> <count>\n",argv[0]);
        return 1;
    } 

	for (int i = 0; i < atoi(argv[3]); i++) { 
		int sockfd = 0, n = 0;
		struct timeval  tv1, tv2;

		memset(recvBuff, '0',sizeof(recvBuff));
		if((sockfd = socket(AF_INET, SOCK_STREAM, 0)) < 0)
		{
		    printf("\n Error : Could not create socket \n");
		    return 1;
		} 

		memset(&serv_addr, '0', sizeof(serv_addr)); 
		serv_addr.sin_family = AF_INET;
		serv_addr.sin_port = htons(atoi(argv[2])); 

		if(inet_pton(AF_INET, argv[1], &serv_addr.sin_addr)<=0)
		{
		    printf("\n inet_pton error occured\n");
		    return 1;
		} 

		gettimeofday(&tv1, NULL);
		if( connect(sockfd, (struct sockaddr *)&serv_addr, sizeof(serv_addr)) < 0)
		{
		   printf("\n Error : Connect Failed \n");
		   return 1;
		} 
		    
         // Shutdown and close
		if (shutdown(sockfd, SHUT_RDWR) != 0)
          return -1;
        if (close(sockfd) !=0)
          return -1;

		gettimeofday(&tv2, NULL);
		 
		printf ("Total time = %f ms\n",
	         (double) (tv2.tv_usec - tv1.tv_usec) / 1000000 +
	         (double) (tv2.tv_sec - tv1.tv_sec));
	}

Lets take a look to the output of the client started with count = 3

Total time = 0.000359
Total time = 1.024391
Total time = 1.020205

Well, the really first connection is pretty fast. The fallowing connections need more than a second. :frowning:

I next I Tested my client against a netcat server, which was launched by nc -vkl localhost $PORT. Lets take a look to that outout:

Total time = 0.000127
Total time = 0.000049
Total time = 0.000050

This is really awesome. Every connection is pretty fast. So I think there is an issue in my Server code above. In the wireshark log you can see, that there is a [RST, ACK] Message and an timeout of about 1000ms.

Does anybody know how to solve this issue?

rtel wrote on Friday, March 23, 2018:

Sorry you didn’t get a reply sooner, there were a couple of posts I missed due to email fumbling on my end. I’ve not had a chance to read through your code in detail but wanted to quickly note there is an example of using this option in FreeRTOS-Plus\demo\common\FreeRTOS_Plus_CLI_Demos\TCPCommandConsole.c in the labs download.

heinbali01 wrote on Saturday, March 24, 2018:

The client uses this code:

    if (shutdown(sockfd, SHUT_RDWR) != 0)
        return -1;
    if (close(sockfd) !=0)
        return -1;

This looks like a gracious shutdown, but I think that it misses one important step: waiting for an acknowledgement of the shutdown.

Could you call recv() on that socket until it returns an error code?

    int rc;
    shutdown( sockfd, SHUT_RDWR );
    for( ;; )
    {
        rc = recv( sockfd, recvBuff, sizeof( recvBuff ), 0 );
        if( rc < 0 )
        {
            break;
        }
    } 
    close(sockfd);

heinbali01 wrote on Saturday, March 24, 2018:

There is a second bigger problem with your embedded code.
You set the option FREERTOS_SO_REUSE_LISTEN_SOCKET for the server socket.
Re-using a socket means that the server socket wil turn into a connected socket.
After closing it:

    FreeRTOS_closesocket(c->remote_sock);

you can not use it anymore: a new listing socket must be created.

So you can either stop using FREERTOS_SO_REUSE_LISTEN_SOCKET, or you create a new socket after every connection.

Re-using sockets is useful in two situations: when you have very litte RAM, or when a server socket will only accept 1 client.

edeviser wrote on Monday, March 26, 2018:

Hello Hein Tibosch,
thank you for your response. It sounds very plausible for me. I’m going to implement your suggestions tomorrow.

But I have on more question. You said:

After closing it:

FreeRTOS_closesocket(c->remote_sock);

you can not use it anymore: a new listing socket must be created.

So you can either stop using FREERTOS_SO_REUSE_LISTEN_SOCKET, or you create a new socket after every connection.

Re-using sockets is useful in two situations: when you have very litte RAM, or when a server socket will only accept 1 client.

In fact I have little RAM and my socket server shall accept only one client at the moment. So I would like to use the FREERTOS_SO_REUSE_LISTEN_SOCKET option.

In the FreeRTOS documentation it says:

At the end of the connection, the socket must be closed by calling FreeRTOS_closesocket(). After that a new socket can be created and bound to the same port.

So does this mean if the client has disconnected from my socket server which uses the FREERTOS_SO_REUSE_LISTEN_SOCKET option ,

A: the server must close the connection and the server must create a new socket for the next client?

B: the server can just go on using the previous socket to reenter the listen mode for the next client?

What is truth? A, B or A&B?

heinbali01 wrote on Monday, March 26, 2018:

So does this mean if the client has disconnected from my socket server which
uses the FREERTOS_SO_REUSE_LISTEN_SOCKET option,

A: the server must close the connection and the server must create a
new socket for the next client?

Correct

B: the server can just go on using the previous socket to reenter the
listen mode for the next client?

False

Note the distinction between closing a TCP connection and closing a socket:
Closing a connection is normally initiated by the TCP client. But if you want, a server can also decide to close the connection by calling shutdown(). The shutdown procedure takes a few ms (or up to a second on the Internet). You must wait for its result.
A TCP application will notice that a connection was closed as soon as API’s return an error code not being equal to EWOULDBLOCK (== EAGAIN).

Closing a socket means freeing the memory space used by the socket, and also freeing up a port number. If the connection is not closed before a socket is closed, you will see a non-gracious closure and possibly a hanging socket.
Please have a look at ipconfigTCP_HANG_PROTECTION. When enabled, the +TCP socket will send an “Are You Alive message” to the other party every ipconfigTCP_HANG_PROTECTION_TIME seconds. This will avoid hanging sockets. It is worth using it.

One last word about FREERTOS_SO_REUSE_LISTEN_SOCKET: reusing means that the same socket pointer is being used for both the listening socket as well as the connected socket.

After the following statement:

    Socket_t xChildSocket = accept( xParentSocket, &xAddress, &xAddressSize );

xChildSocket will either be equal to:

NULL in case the function timed-out
FREERTOS_INVALID_SOCKET in case xParentSocket is invalid
xParentSocket in case a new connection was made

So if the function succeeds, do not use xParentSocket any more, and do not close it because xChildSocket == xParentSocket.

For non-REUSE sockets, xParentSocket will be used again and again as a parameter of accept(). In that case also you can put a limit on the maximum number of connected clients:

    /* Allow 1 simultaneous connection only: */
    FreeRTOS_listen( xParentSocket, 1 );

I hope that this text is all clear.

edeviser wrote on Wednesday, March 28, 2018:

Hello Hein Tibosch,

thank you for the detailed explanation. For me its all clear now. I fixed my implementation now on the client and the server side. I moved thefor statement in the server to the very beginning. So the socket is being created and set up for every conenction.

Furthermore I added the ‘read-loop wating for a netgative return value’ to the client’s code between the shutdown and the close. Nevertheless the recv never returns with a negative value. I replaced the ‘read-loop wating for a netgative return value’ with a simple usleepp(50000) //50ms. Thus the duration (previous about one second) goes down to 50ms and there are no error frames in wireshark any more. So I think we are on the right way. The client must wait after doing a shutdown until the connection is really downed. After that the client can close the socket and can go on establishing a new connection. Waiting a constant amount of millliseconds is not a good workaround.

Maybe you have got an other idea to wait until the connection is down?

edeviser wrote on Tuesday, April 24, 2018:

Hello.

I worked really hard at this problem, but it is still not solved. I think I respected all recommendations. Nevertheless the problem resists.

I attached for you:

  • server code snippet
  • client code (compile with ‘gcc linuxClient.c’ and run with ‘./a.out 192.168.42.1 3102 2’)
  • client code (compile with ‘javac linuxClient.java’ and run with ‘java linuxClient.class’)
  • Screenshots of wireshark

In the screenshot you can see very clearly, that there is an error frame and a retransmission which causes a delay of 1000ms. I would really like to avoid this delay!

heinbali01 wrote on Tuesday, April 24, 2018:

Hi, when you want to show the traffic, please save the captured data in a compressed format and attach it to your post. That’s a lot easier than a screen-shot.

I’m a bit puzzled by 2 things in the capture:

  1. Why is there a RST packet?
  2. Why does it say “TCP port numbers reused”?

Did you implement a function for ipconfigRAND32() ? It must return a 32-bit random number.
And if possible, every time after a reboot, it should start with a different random number. Maybe you can use the RND peripheral for this.

This randomness is very important.

In linuxClient.c you comment that : “Wait for shutdown does not work”
What happens then? Does recv() ever return? With what value?

When using Winsock this is easier: when the FIN status has been reached, select() will return a positive value for READ, while recv() returns 0. That is a sign that the connection was shut down, and you can safely close() the socket.

I’m less sure about the internals of Linux. I always call select() before calling recv(), so that recv() never has to block. When the socket has become invalid or the connection got closed, select()will return witth an error.

Basically you wrote this client code:

    for (int counter = - conuter < N; counter++) {
        connect(sockfd, (struct sockaddr *)&serv_addr, sizeof(serv_addr));
        shutdown(sockfd, SHUT_RDWR);
        close(sockfd);
    }

What is the purpose of connecting to a TCP server without exchanging any data? Is this just a test?

What happens if you do send/recv a bit of data?

In the STM server program you wrote:

    while(FreeRTOS_recv(c->socket, buf, 1, 0) >= 0) {
      vTaskDelay(5);
    }

Officially you should check for -EWOULDBLOCK ( == -EGAIN ) :

    for( ;; )
    {
    int rc;

        rc = FreeRTOS_recv(c->socket, buf, 1, 0);
        if( ( rc < 0 ) && ( rc != -pdFREERTOS_ERRNO_EWOULDBLOCK ) )
        {
            break;
        }
        vTaskDelay(5);
    }

But as you socket is used in blocking mode, I do not think that EWOULDBLOCK will be returned.

When I find time, I will start-up my STM32Fx and do the same tests.

heinbali01 wrote on Tuesday, April 24, 2018:

No need to answer the above questions: I’m running your code on my STM32Fx board and on Ubuntu.

The first thing that I found is that a short (10 ms) delay between shutdown() and close() makes a big difference. I implemented this delay with nanosleep().

Now I will try to find out what happens if the delay is not present.

edeviser wrote on Tuesday, April 24, 2018:

Okay, that sound good. Here is my explanation for the error frames:

Take a look to the screen shot attached. You can produce it by importing flow.pcapng of my first port. Then choose Statistics->Flowgraph and select ‘TCP Flows’ as Flow type.

A normal communication flow is marked by one color. You can see very clearly, that the first flow, (green) is being interrupted by a second flow (orange). This seems to be no problem for the server at 10.169.7.217 which is a simple server which used netcat (nc -l).

But the server at 10.169.7.49 which is the stm32Server does have a problem withi this interruption which occures when the second flow (beige) starts ere the first flow (red) had been terminated by fin,ack|ACK.

This could be caused by the reuse socket option which was used by the stm32server. The resuse socket option prevents the stm32server to handle the second flow ere the first flow is being terminated. Waiting about 10ms is enough for the stm32 to handle the last fin,ack and the ACK.

The netcat server can handle multiple connections at the same time. Thus it has no problem with this.

heinbali01 wrote on Tuesday, April 24, 2018:

The problem that you encountered is the following:

The client connects to the server-socket, sends a shutdown and closes its socket. It think this is OK, the OS will handle the shutdown and postpone the actual freeing of the socket until the connection has really finished.

Right after the connection, the client sends a SYN packet, while FreeRTOS+TCP has not yet created a new server-socket. And therefore it replies to the SYN packet with a RST packet.

After a time-out of 3 seconds, the client will send a new SYN, which will be acknowledged. Wireshark warns that my STM32Fx takes the same port number ( TCP port numbers reused ). The warning may be ignored in this case.

I tried a trick: I created a new server socket right after accept() succeeded. I also put it into listen mode.

That helped: the client doesn’t have to wait again for the new server socket to be ready.

When this is the normal TCP conversation:

Client SYN
Server SYN + ACK
Client ACK

Client FIN
Server FIN + ACK
Client ACK

I observed the following if the client is too fast:

Client_1 SYN
Server   SYN + ACK
Client_1 ACK

Client_1 FIN
Client_2 SYN       // A SYN is sent for a new connection
Server   RST       // Sorry, there is no server socket yet
Server   FIN + ACK // to Client_1
Client_1 ACK

The problem does not exist if you do not use the re-use option. Note that a server socket is not as expensive as a connected socket because it does not allocate TCP stream buffers.

If you insist on using the re-use option I could think of creating a patch. I’m not yet happy about such a patch because it is a bit tricky.

heinbali01 wrote on Tuesday, April 24, 2018:

The netcat server can handle multiple connections at the same
time. Thus it has no problem with this

Of course FreeRTOS+TCP can also handle multiple connections at the same time if you omit the re-use option.
The second parameter of FreeRTOS_listen() determines the maximum number of clients that are allowed at any moment.

heinbali01 wrote on Tuesday, April 24, 2018:

For you info: I attached a PCAP file in which the FreeRTOS+TCP server socket is permanent, the re-use option is not used. It recevies 1,000 TCP connections per second.

If you use a permanent socket that allows for only 1 client, you will meet the same problem as with the re-use option: a second connect from Linux might come wjile the first connection isn’t closed yet.

edeviser wrote on Tuesday, April 24, 2018:

What is a ‘permanent socket’? Does it mean to use two sockets instead of using the reuse option and to limit the clients by passing a 1 to the FreeRTOS_listen(...) command?

heinbali01 wrote on Tuesday, April 24, 2018:

Right!

    for (;;) {
        Socket_t xSocket = FreeRTOS_accept( xServerSocket, &remote_addr, &l );
        if (xSocket == FREERTOS_INVALID_SOCKET) {
            continue;
        }
        for( ;; )
        {
            int rc = FreeRTOS_recv(xSocket, buf, 1, 0);
            if( ( rc < 0 ) && ( rc != -pdFREERTOS_ERRNO_EWOULDBLOCK ) )
            {
                break;
            }
        }
        FreeRTOS_closesocket(xSocket);
    }

In the above example, xServerSocket continues to exist.

But if you allow only 1 client :

    FreeRTOS_listen( xServerSocket, 1 )

you will still see the problem occurring. I’m afraid that you have to use at least 3, although you only help one client at a time.

Within Linux, after close(socket), the kernel will take over the socket and make sure that it’s connection is closed (because shutdown() was called).

While testing, I saw in a PCAP that at some point there were 3 concurrent connections: 2 were in a closing status, one was just starting.