FreeRTOS+TCP stops replying and recovers after a long time

[michaelv]

Hi.
My application is an ethernet to UART converter with 3 ports.
It runs 4 TCP listening tasks (the 4th task is for configuration) that send received data to UART ports (there are 4 additional UART listening tasks that send received data to connected TCP ports).
All TCP tasks run with the same code (the difference is the parameter which defines to which port to listen), here it is:

void TcpTask(void *param)
{
    int pn = (int) param;
    int port;
    struct freertos_sockaddr xClient, xBindAddress;
    socklen_t xSize = sizeof( xClient );
    Socket_t xConnectedSocket = 0, xListeningSocket = 0;
    BaseType_t lBytesReceived;
    BaseType_t xSocketReuse = pdTRUE;
    Buffer *pbufUartTx, *pbufTcpRx;
    
    port=pn;   
        
    pbufUartTx = &bufUartTX[port];
    pbufTcpRx = &bufTcpRX[port];

    while(1)
    {
  
        if(xListeningSocket == 0)
        {
            xListeningSocket = FreeRTOS_socket( FREERTOS_AF_INET4, FREERTOS_SOCK_STREAM, FREERTOS_IPPROTO_TCP );
            if( xListeningSocket == FREERTOS_INVALID_SOCKET )
            {
                xListeningSocket = 0;
                vTaskDelay(100);
                continue;
            }
            
            FreeRTOS_setsockopt(xListeningSocket, 0, FREERTOS_SO_REUSE_LISTEN_SOCKET, &xSocketReuse, sizeof(xSocketReuse));
                
            memset( &xBindAddress, 0, sizeof(xBindAddress) );
            xBindAddress.sin_port = ( uint16_t ) TCP_PORT_BASE + pn;
            xBindAddress.sin_port = FreeRTOS_htons( xBindAddress.sin_port );
            xBindAddress.sin_family = FREERTOS_AF_INET4;
            FreeRTOS_bind( xListeningSocket, &xBindAddress, sizeof( xBindAddress ) );
            FreeRTOS_listen( xListeningSocket, 1 );
        }
        
        xConnectedSocket = FreeRTOS_accept( xListeningSocket, &xClient, &xSize );
        if((xConnectedSocket != NULL) && (xConnectedSocket != FREERTOS_INVALID_SOCKET))
        {
            xServerSocket[port] = xConnectedSocket;
            
            while(xConnectedSocket)
            {
                lBytesReceived = FreeRTOS_recv( xConnectedSocket, pbufTcpRx->buf, BUFFER_SIZE, 0 );
                if( lBytesReceived > 0 )
                {
                    /* Data was received, process it here. */
                    
                    /* Check whether the previous packet was sent */
                    if(pbufUartTx->count == 0)
                    {
                        pbufUartTx->count = lBytesReceived;
                        memcpy(pbufUartTx->buf, pbufTcpRx->buf, lBytesReceived);
                        
                        if(port != PORT_3) UartSendData(port, pbufUartTx->buf, lBytesReceived);
                        else
                        {
                            PerformGetSetConfig(pbufUartTx->buf, PORT_3, 0, lBytesReceived);
                            Led3 = LED_OFF;
                            pbufUartTx->count = 0;
                        }
                    }
                }
                else if( lBytesReceived == 0 )
                {
                    /* No data was received, but FreeRTOS\_recv() did not return an error.
                       Timeout? */
                }
                else
                {
                    /* Error (maybe the connected socket already shut down the socket?).
                       Attempt graceful shutdown. */
                    FreeRTOS_shutdown( xConnectedSocket, FREERTOS_SHUT_RDWR );
                    FreeRTOS_closesocket( xConnectedSocket );
                    xConnectedSocket = 0;
                    xListeningSocket = 0;
                    xServerSocket[port] = 0;
                    break;
                }
            }
        }     

    }
}

My problem is as following:
when the program runs, an outside application connects to port #2 and transfers data to and from UART #2. Then I use Putty and connect to port #3 (which is a configuration port, it is not related to any UART). As long as I send “wrong” data through this port (the port doesn’t reply to wrong data), nothing happens. But the moment I send a recognized command (like - get version), I get the right reply, but then the whole device stops replying to ping and to the 2 previously connected ports.
Here is what I see at +TCP debug printout:

10:02:13.599 --> .FreeRTOS_AddEndPoint: MAC: 48-5c IPv4: ac1601e9ip
10:02:17.224 --> FreeRTOS_NetworkDown is called
10:02:17.224 --> xPhyReset: phyBMCR_RESET 0 ready
10:02:17.308 --> +TCP: advertise: 0101 config 3100
10:02:17.308 --> prvEthernetUpdateConfig: LS mask 00 Force 1
10:02:17.308 --> xSTM32F_NetworkInterfaceInitialise returns 0
10:02:18.774 --> xPhyCheckLinkStatus: PHY LS now 01
10:02:18.774 --> prvEMACHandlerTask LS has changed
10:02:18.774 --> prvEthernetUpdateConfig: LS mask 01 Force 0
10:02:18.843 --> Network buffers: 59 lowest 59
10:02:18.843 --> Heap: current 6464 lowest 6464
10:02:20.331 --> FreeRTOS_NetworkDown is called
10:02:20.331 --> Link Status is high
10:02:20.331 --> xSTM32F_NetworkInterfaceInitialise returns 1
10:02:20.331 --> Heap: current 4912 lowest 4912
10:02:20.394 --> Heap: current 3280 lowest 3280
10:02:21.518 --> Heap: current 4160 lowest 2608
10:02:23.625 --> Heap: current 1056 lowest 1056
10:02:41.531 --> Network buffers: 57 lowest 57
10:02:41.531 --> Network buffers: 57 lowest 56

/* 30 seconds after stop replying to ping */
10:03:50.015 --> vTCPStateChange: Closing (Queued 0, Accept 0 Reuse 1)
10:03:50.015 --> vTCPStateChange: me 2000edd8 parent 2000edd8 peer 00000000 clear 0
10:03:50.031 --> vTCPStateChange: xHasCleared = 0

/* 3 minutes after  */
10:06:34.875 --> vTCPStateChange: Closing (Queued 0, Accept 0 Reuse 1)
10:06:34.875 --> vTCPStateChange: me 2000ef48 parent 2000ef48 peer 00000000 clear 0
10:06:34.906 --> vTCPStateChange: xHasCleared = 0

It takes the +TCP 3 minutes to start replying again (after it notifies of closing the second port).

Can anyone help solve this issue?
Thanks.

[RAc]

your socket usage is wrong. You need to create the listening socket just once and then do the accept-process-close accepted socket loop in your while(1) loop. There are numerous examples for a correct tcp server on the net.

EDIT: Looking closer at your code, it appears that you do that part right (the control flow is a little bit unusual). I will examine your case closer, apologies.

[aggarg (Gaurav Aggarwal)]

@RAc

There are numerous examples for a correct tcp server on the net.

Here is one - FreeRTOS/FreeRTOS-Plus/Demo/FreeRTOS_Plus_TCP_Minimal_Windows_Simulator/DemoTasks/SimpleTCPEchoServer.c at main · FreeRTOS/FreeRTOS · GitHub.

[RAc]

ok, here is a second attempt. You write that your problem does not occur until valid communication shows up on your “control port 3.” Thus it would help to see the code that serves the port. Also, what are the priorities of all of your tasks? If the “control port task” has a pro higher than your worker tasks AND your tcp task, a lengthy cpu bound computation in that task may starve your system.

[michaelv]

@RAc and @aggarg thank you for your replies.

I’ve changed my code to the following:

void TcpTask(void *param)
{
    int pn = (int) param;
    int port;
    struct freertos_sockaddr xClient, xBindAddress;
    socklen_t xSize = sizeof( xClient );
    Socket_t xConnectedSocket = 0, xListeningSocket = 0;
    BaseType_t lBytesReceived;
    BaseType_t xSocketReuse = pdTRUE;
    Buffer *pbufUartTx, *pbufTcpRx;
    
    port=pn;   
        
    pbufUartTx = &bufUartTX[port];
    pbufTcpRx = &bufTcpRX[port];
    
    xListeningSocket = FreeRTOS_socket( FREERTOS_AF_INET4, FREERTOS_SOCK_STREAM, FREERTOS_IPPROTO_TCP );
 
    memset( &xBindAddress, 0, sizeof(xBindAddress) );
    xBindAddress.sin_port = ( uint16_t ) TCP_PORT_BASE + pn;
    xBindAddress.sin_port = FreeRTOS_htons( xBindAddress.sin_port );
    xBindAddress.sin_family = FREERTOS_AF_INET4;
    FreeRTOS_bind( xListeningSocket, &xBindAddress, sizeof( xBindAddress ) );
    FreeRTOS_listen( xListeningSocket, 1 );

    while(1)
    {
       
        xConnectedSocket = FreeRTOS_accept( xListeningSocket, &xClient, &xSize );
        if((xConnectedSocket != NULL) && (xConnectedSocket != FREERTOS_INVALID_SOCKET))
        {
            xServerSocket[port] = xConnectedSocket;
            
            while(xConnectedSocket)
            {
                lBytesReceived = FreeRTOS_recv( xConnectedSocket, pbufTcpRx->buf, BUFFER_SIZE, 0 );
                if( lBytesReceived > 0 )
                {
                    /* Data was received, process it here. */
                    
                    /* Check whether the previous packet was sent */
                    if(pbufUartTx->count == 0)
                    {
                        pbufUartTx->count = lBytesReceived;
                        memcpy(pbufUartTx->buf, pbufTcpRx->buf, lBytesReceived);

                        if(port != PORT_3) UartSendData(port, pbufUartTx->buf, lBytesReceived);
                        else
                        {
                            PerformGetSetConfig(pbufUartTx->buf, PORT_3, 0, lBytesReceived);
                            Led3 = LED_OFF;
                            pbufUartTx->count = 0;
                        }
                    }
                }
                else if( lBytesReceived == 0 )
                {
                    /* No data was received, but FreeRTOS\_recv() did not return an error.
                       Timeout? */
                }
                else
                {
                    /* Error (maybe the connected socket already shut down the socket?).
                       Attempt graceful shutdown. */
                    FreeRTOS_shutdown( xConnectedSocket, FREERTOS_SHUT_RDWR );
                    FreeRTOS_closesocket( xConnectedSocket );
                    xConnectedSocket = 0;
                    xServerSocket[port] = 0;
                    break;
                }
            }
        }     

    }
}

Now I still get the issue I’ve described, but the recovery time is much shorter.

@RAc

You write that your problem does not occur until valid communication shows up on your “control port 3.” Thus it would help to see the code that serves the port.

The control port task runs the same code as above with task parameter = 3. This makes the task send the received data to a function PerformGetSetConfig() instead to a UART port. This function parses the command and sends back a reply through the same port using FreeRTOS_send().

@RAc

Also, what are the priorities of all of your tasks? If the “control port task” has a pro higher than your worker tasks AND your tcp task, a lengthy cpu bound computation in that task may starve your system.

All tasks have the same priority - 1. No computation is made in the control port task, it just sends back a version number string (the reply is instantaneous).

[RAc]

what does your PerformGetSetConfig do exactly? Also, do you have time slicing enabled, and what is your tcp task priority? Also 1?

[michaelv]

@RAc

what does your PerformGetSetConfig do exactly?

it compares the input string to a constant, and sends a reply according to the sent command (for example - “getver” command string gets a reply string with the name of device version)

@RAc

do you have time slicing enabled, and what is your tcp task priority?

Time slicing is enabled.
TCP task priority is 3.

[RAc]

You say that your system stops responding to icmp (ping) requests in the error scenario. If your tcp task has highest priority, then either it is suspended (possibly due to something holding the critical section) or interrupts from the network stop coming in.

Next thing I would do is try to determine if your system runs at all when your problem occurs. Can you try to break into the sys tick interrupt at that time? If not, you may want to add an led blinky task at medium pri and see if the leds still blink when your connectivity is gone.

[michaelv]

@RAc

you may want to add an led blinky task at medium pri and see if the leds still blink when your connectivity is gone.

I’ve added a LED blinking task with the same priority as others (1) and the LED is blinking all the time, even when connectivity is gone.

[RAc]

@michaelv

                else if( lBytesReceived == 0 )
                {
                    /* No data was received, but FreeRTOS\_recv() did not return an error.
                       Timeout? */
                }

Do you ever reach that branch?

[michaelv]

@RAc

Do you ever reach that branch?

yes occasionally, during both normal operation and when connectivity is gone.

[michaelv]

Any suggestions would be appreciated

[htibosch (Hein Tibosch)]

Some return values are not fatal:

     else if( lBytesReceived == 0 )
     {
         /* No data was received, but FreeRTOS\_recv() did not return an error.
            Timeout? */
     }
-    else
+    else if( ( lBytesReceived != -pdFREERTOS_ERRNO_EAGAIN ) &&
+             ( lBytesReceived != -pdFREERTOS_ERRNO_EINTR ) )
     {

like EAGAIN and EINTR.

Have you checked if any of the UART-related functions is blocking? Do they always return quickly?

Have you looked at a PCAP recording of the TCP conversation?

xListeningSocket = 0

PS. In +TCP, a socket is a pointer, not a number. It would be more “correct” to assign and compare with NULL
It won’t change the flow of the program of course.

FreeRTOS_shutdown( xConnectedSocket, FREERTOS_SHUT_RDWR );
FreeRTOS_closesocket( xConnectedSocket );

It has no use to call shutdown() here. It is used when the connection is still alive and you want to terminate it.

But to be sure, you can call shutdown. In that case you should call FreeRTOS_recv() in a loop and wait for a fatal return code:

static void shutdown_socket( Socket_t sck )
{
    TickType_t start_time;

    FreeRTOS_shutdown( sck, FREERTOS_SHUT_RDWR );
    start_time = xTaskGetTickCount( );

    do {
        int rc = FreeRTOS_recv( sck, buffer, sizeof buffer, 0 );
        if( ( rc < 0 ) &&
            ( rc != -pdFREERTOS_ERRNO_EAGAIN ) &&
            ( rc != -pdFREERTOS_ERRNO_EINTR ) )
        {
            break;
        }
        if( rc > 0 )
        {
          /* Use the received data. */
        }
    } while( ( xTaskGetTickCount( ) - start_time ) < pdMS_TO_TICKS( 1000 ) );

    FreeRTOS_closesocket( sck );
}

Also, I would recommend to issue some logging from all tasks, to know what they’re up to.

10:02:21.518 → Heap: current 4160 lowest 2608
10:02:23.625 → Heap: current 1056 lowest 1056
10:02:41.531 → Network buffers: 57 lowest 57
10:02:41.531 → Network buffers: 57 lowest 56

I see that you have little heap. I think that 56 network buffers is very much. Not sure if you use BufferAllocation_1 or BufferAllocation_2?

Also, another way to save RAM is to lower the MTU size. I don’t know how much data you receive on each socket?
One more thing: ipconfigTCP_WIN_SEG_COUNT is 256 by default. When you have 4 sockets, you won’t need more than 20 TCP segments. So add to your FreeRTOSIPConfig.h:

    #define ipconfigTCP_WIN_SEG_COUNT    20

I’ve changed my code to the following:

I think that your first attempt was also OK. The reviewers did not notice that you were using so-called re-use sockets (FREERTOS_SO_REUSE_LISTEN_SOCKET). Reusable socket can only be used one time, as a server, turning into a client, and then it should be closed ( I was also wondering why each connection needs a new server socket ).

#define ipconfigSOCK_DEFAULT_RECEIVE_BLOCK_TIME portMAX_DELAY

I never like portMAX_DELAY.

I recommend to change the receive blocking/sending time. You only have to do this for the parent socket, the child sockets will inherit all properties of the parent.
Please give it a regular value like eg. 3 seconds (pdMS_TO_TICKS( 3000U )), and see if it makes any difference.
Good luck

[michaelv]

@htibosch

I see that you have little heap.

My heap definition configTOTAL_HEAP_SIZE in FreeRTOSConfig was 10240. I doubled it to 20240 and now the problem disappeared.
Heap status still gets pretty low (Heap: current 1336 lowest 1336). I hope it won’t be a problem.
Thank you @htibosch

[htibosch (Hein Tibosch)]

I doubled it to 20240 and now the problem disappeared.

Very good!

Heap status still gets pretty low (Heap: current 1336 lowest 1336). I hope it won’t be a problem.

Well, as you know, life is just more relaxed with heaps of heap.

Have you already checked ipconfigTCP_WIN_SEG_COUNT?

And also you could check the buffers size of the socket, and set it using the socket option FREERTOS_SO_WIN_PROPERTIES.

Or play with the value of ipconfigNETWORK_MTU, depending on the sizes of your packets?

PS Why don’t you use heap_5.c and make a heap that uses all remaining RAM?
I use special linker symbols in this example.

I assume you are using an STM32F4x?

[michaelv]

@htibosch

Have you already checked ipconfigTCP_WIN_SEG_COUNT?

I have defined

#define ipconfigUSE_TCP_WIN			( 0 )

So I guess ipconfigTCP_WIN_SEG_COUNT value is not relevant (?)

@htibosch

Or play with the value of ipconfigNETWORK_MTU, depending on the sizes of your packets?

My board application is a general Ethernet to UART converter. I don’t know what is the size of the packets. ipconfigNETWORK_MTU is defined as 1200.

@htibosch

Why don’t you use heap_5.c and make a heap that uses all remaining RAM?

Right now this application runs on a dedicated board (which has only one purpose). In the future it is supposed to run on a multipurpose board, where it’ll be one task of a multi tasking platform. This is why I want to use as little RAM as possible.

@htibosch

I assume you are using an STM32F4x?

Right now I’m using STM32F2.