FreeRTOS+TCP on STM32H753 Buffer Exhaustion after client disconnects+reconnects

@tony-josi-aws

Hello!

I am using version 4.3.1. Sorry for the lack of code in the original post, I had to clean it up a bit. I will put the two server tasks below:

The sockets are all stored in a global array initialized to 0.

    for( int i = 0; i < MAX_TM_SOCKETS; i++ )
    {
        xTMClients.sockets[ i ] = 0x0;
    }

Here is the code for the task that handles the connection of clients to the server:

void vTMEthernetSocketHandlerTask(void *pvParameters) {


	struct freertos_sockaddr xClient, xBindAddress;
	Socket_t xTMListeningSocket;
	Socket_t xNewSocket;
	socklen_t xSize = sizeof( xClient );
	const BaseType_t xBacklog = 20;
	BaseType_t xResult;
	TickType_t xAcceptTimeout = pdMS_TO_TICKS(5000);

	xTMListeningSocket = FreeRTOS_socket( FREERTOS_AF_INET4,
										FREERTOS_SOCK_STREAM,
										FREERTOS_IPPROTO_TCP );

	configASSERT( xTMListeningSocket != FREERTOS_INVALID_SOCKET);

	memset( &xBindAddress, 0, sizeof(xBindAddress) );
	xBindAddress.sin_port = ( uint16_t ) 5000;
	xBindAddress.sin_port = FreeRTOS_htons( xBindAddress.sin_port );
	xBindAddress.sin_family = FREERTOS_AF_INET4;


	xResult = FreeRTOS_bind( xTMListeningSocket, &xBindAddress, sizeof( xBindAddress ) );
	configASSERT(xResult == 0);


	xResult = FreeRTOS_listen( xTMListeningSocket, xBacklog );
	FreeRTOS_setsockopt(xTMListeningSocket,
	                    0,
	                    FREERTOS_SO_RCVTIMEO,
	                    &xAcceptTimeout,
	                    sizeof(xAcceptTimeout));
	configASSERT(xResult == 0);
	F_TCP_UDP_Handler_t Handler_OnConnectDisconnect = {0};
	Handler_OnConnectDisconnect.pxOnTCPConnected = Callback_OnConnectDisconnect;
	Handler_OnConnectDisconnect.pxOnTCPReceive = NULL;
	Handler_OnConnectDisconnect.pxOnTCPSent = NULL;

	for( ;; )
	{

		UBaseType_t ustackHighWaterMark = uxTaskGetStackHighWaterMark(xTMEthernetSocketHandlerHandle);
		configASSERT(ustackHighWaterMark > 20);
		FreeRTOS_netstat();

		xNewSocket = FreeRTOS_accept(xTMListeningSocket, &xClient, &xSize);

		/* Check if socket value has been set */
		if (xNewSocket != 0x0) {
		    /* Set up disconnect callback */
		    Handler_OnConnectDisconnect.pxOnTCPConnected = Callback_OnConnectDisconnect;
		    FreeRTOS_setsockopt(xNewSocket,
		                       0,
		                       FREERTOS_SO_TCP_CONN_HANDLER,
		                       (void *)&Handler_OnConnectDisconnect,
		                       sizeof(Handler_OnConnectDisconnect));

		    /* Add to client array */
		    if (xSemaphoreTake(xTMClientsArrayMutex, portMAX_DELAY) == pdTRUE) {
		        BaseType_t xSpaceFound = pdFALSE;
		        for (int i = 0; i < MAX_TM_SOCKETS; i++) {
		            if (xTMClients.sockets[i] == 0) {
		                xTMClients.sockets[i] = xNewSocket;
		                xSpaceFound = pdTRUE;
		                break;
		            }
		        }

		        xSemaphoreGive(xTMClientsArrayMutex);

		        /* If no space available, close socket */
		        if (xSpaceFound == pdFALSE) {
		            FreeRTOS_shutdown(xNewSocket, FREERTOS_SHUT_RDWR);
		            vTaskDelay(pdMS_TO_TICKS(500));
		            FreeRTOS_closesocket(xNewSocket);
		        }
		    } else {
		        /* If array mutex unavailable, close socket */
		        FreeRTOS_shutdown(xNewSocket, FREERTOS_SHUT_RDWR);
		        vTaskDelay(pdMS_TO_TICKS(500));
		        FreeRTOS_closesocket(xNewSocket);
		    }
		} else {
		    vTaskDelay(pdMS_TO_TICKS(100));
		}
}}

The idea is that it will accept a connection, check for an open slot in the array, and store the socket there if there is space. This can store 5 sockets. I am aware the backlog of the listening socket is 20, I just haven’t gotten around to changing it as for now I have only been testing 1 client. This has worked fine to connect a client. The callback to handle disconnects from the client is this:

static void Callback_OnConnectDisconnect( Socket_t xSocket, BaseType_t ulConnected )
{
	FreeRTOS_debug_printf(("Callback_OnConnectDisconnect called with ulConnected = %d\n", ulConnected));
	char pvdummy[2000];

    if( ulConnected == pdFALSE )
    {
    	FreeRTOS_debug_printf(("Disconnect detected\n"));
    	FreeRTOS_shutdown(xSocket, FREERTOS_SHUT_RDWR);
    	FreeRTOS_debug_printf(("After shutdown\n"));
        vTaskDelay( pdMS_TO_TICKS( 100 ) );
        FreeRTOS_debug_printf(("About to take mutex\n"));

        if( xSemaphoreTake( xTMClientsArrayMutex, portMAX_DELAY ) == pdTRUE )
        {
        	FreeRTOS_debug_printf(("Mutex taken successfully\n"));

            for( int i = 0; i < MAX_TM_SOCKETS; i++ )
            {
                if( xTMClients.sockets[ i ] == xSocket )
                {
                    xTMClients.sockets[ i ] = 0x0;

                    while( FreeRTOS_recv( xSocket, pvdummy, sizeof(pvdummy), 0 ) >= 0 )
                        {
                            vTaskDelay( pdMS_TO_TICKS( 250 ) );
                        }
    				UBaseType_t uxCount = uxGetNumberOfFreeNetworkBuffers();
    				FreeRTOS_debug_printf(("Free buffers pre close: %u\r\n", uxCount));
                    FreeRTOS_closesocket( xSocket );
                    uxCount = uxGetNumberOfFreeNetworkBuffers();
                    FreeRTOS_debug_printf(("Free buffers post close: %u\r\n", uxCount));
                    break;
                }
            }
            xSemaphoreGive( xTMClientsArrayMutex );
            FreeRTOS_debug_printf(("closing mutex returned\n"));
        }
        else
                {
                    /* Add debug print if semaphore take fails */
                    FreeRTOS_debug_printf(("Failed to take mutex in Callback_OnConnectDisconnect\n"));
                }
    }
}

Once connected, I have another task which calls my zero-copy receive function to receive a packet and then echoes it back to the client with the zero-copy broadcast function. This is just for testing. This part works fine when the socket is initially connected. The zero-copy broadcast function cycles through each socket in the global array and sends the data to each.

	void vEthernetTxTask(void *pvParameters){
		/**
		 * @brief Adds two integers.
		 *
		 * This function takes two integers, adds them together, and returns the result.
		 *
		 * @param[in] a First integer.
		 * @param[in] b Second integer.
		 *
		 * @return The sum of a and b.
		 */

		CLTUStruct xRxedPacket = {0};
		xRxedPacket.valid = 2; //Just so it doesnt send stuff right away

		for(; ;){
			if (xIsSocketConnected(xTMClients.sockets[0]) == pdTRUE){
				if( xSemaphoreTake(xTMClientsArrayMutex, portMAX_DELAY) == pdTRUE)
				{
					xRxedPacket = xTCReceptionZeroCopy(xTMClients.sockets[0]);
				}
				xSemaphoreGive(xTMClientsArrayMutex);
				if (xRxedPacket.valid == 0)
				{
					char *cltubytes = (char *) &xRxedPacket.cltuFrame;
					xSemaphoreTake(xTMClientsArrayMutex, portMAX_DELAY);
					{
						vTMSocketBroadcastZeroCopy(&xTMClients, cltubytes, xRxedPacket.frameLength);
					}
					xSemaphoreGive(xTMClientsArrayMutex);
					vTaskDelay(pdMS_TO_TICKS(100));
				}
				vTaskDelay(pdMS_TO_TICKS(50));

		}
	}}

I know the receive only calls from the first socket of the array, but I have only been testing with one socket so it hasn’t caused any problems. This is the zero-copy broadcast function:

void vTMSocketBroadcastZeroCopy(SocketArray_t * xClientSockets, char *pcBufferToTransmit , const size_t xTotalLengthToSend){
	BaseType_t xAlreadyTransmitted = 0, xBytesSent = 0;
    for (uint8_t i = 0; i < MAX_TM_SOCKETS; i++){
    	Socket_t xTMSocket = xClientSockets->sockets[i];
		if(xTMSocket == FREERTOS_INVALID_SOCKET || xTMSocket == 0x0 || FreeRTOS_issocketconnected(xTMSocket)!= pdTRUE){
			i++;
			continue;
		}
		while( xAlreadyTransmitted < xTotalLengthToSend )
				{
					BaseType_t xAvlSpace = 0;
					BaseType_t xBytesToSend = 0;
					uint8_t *pucTCPZeroCopyStrmBuffer;
					pucTCPZeroCopyStrmBuffer = FreeRTOS_get_tx_head( xTMSocket, &xAvlSpace );

					if(pucTCPZeroCopyStrmBuffer)
					{
						if((xTotalLengthToSend - xAlreadyTransmitted) > xAvlSpace)
						{
							xBytesToSend = xAvlSpace;
						}
						else
						{
							xBytesToSend = (xTotalLengthToSend - xAlreadyTransmitted);
						}
						memcpy( pucTCPZeroCopyStrmBuffer,
								( void * ) (( (uint8_t *) pcBufferToTransmit ) + xAlreadyTransmitted),
								xBytesToSend);
					}
					else
					{
						/* Error - break out of the loop for graceful socket close. */
						break;
					}
						xBytesSent = FreeRTOS_send(
													xTMSocket,
													NULL,
													xBytesToSend,
													/* ulFlags. */
													0 );

					if( xBytesSent >= 0 )
					{
						/* Data was sent successfully. */
						xAlreadyTransmitted += xBytesSent;
					}
					else
					{
						break;
					}
				}
	    }
}

The zero-copy receive function just receives from one socket. For the most part it is copied from the example in the FreeRTOS tutorial.

CLTUStruct xTCReceptionZeroCopy(Socket_t xTCSocket) {
    static char cRxedData[NETWORK_BUFFER_SIZE];
    BaseType_t lBytesReceived;
    CLTUStruct xCLTUStruct = {0};
    uint8_t *pucZeroCopyRxBuffPtr = NULL;

    // Make sure socket has a reasonable timeout set
    const TickType_t xReceiveTimeout = pdMS_TO_TICKS(100);
    FreeRTOS_setsockopt(xTCSocket, 0, FREERTOS_SO_RCVTIMEO, &xReceiveTimeout, sizeof(xReceiveTimeout));

    // Receive data with zero copy flag
    lBytesReceived = FreeRTOS_recv(xTCSocket,
                                 &pucZeroCopyRxBuffPtr,
                                 ipconfigTCP_MSS,
                                 FREERTOS_ZERO_COPY);

    // Process received data if any
    if(lBytesReceived > 0 && pucZeroCopyRxBuffPtr != NULL) {
        // Copy data to application buffer
        memcpy(cRxedData, pucZeroCopyRxBuffPtr, lBytesReceived);

        // Release the TCP payload buffer
        FreeRTOS_ReleaseTCPPayloadBuffer(xTCSocket, pucZeroCopyRxBuffPtr, lBytesReceived);

        // Process the received data
        xCLTUStruct.frameLength = lBytesReceived;
        memcpy(xCLTUStruct.cltuFrame, cRxedData, lBytesReceived); // Only copy what we received
        xCLTUStruct.valid = 0; // Mark as valid
    }
    else if(lBytesReceived == 0) {
        // Timeout occurred - no data received
        xCLTUStruct.valid = 1;
        FreeRTOS_debug_printf(("TCP receive timeout\n"));
    }
    else {
        // Error occurred
        xCLTUStruct.valid = 2;
        FreeRTOS_debug_printf(("TCP receive error: %ld\n", lBytesReceived));

        // Don't try to shut down the socket here - let the callback handle it
        // Just return error status
    }
    return xCLTUStruct;
}

The task priority goes: sending/recieving task → socket handling → Freertos IP task → MAC task

I am connecting to the MCU from my desktop PC using the PuTTY telnet functionality and just sending small words back and forth. This works fine, and when I disconnect the PuTTY client everything seems to close gracefully and continue on. Wireshark shows the correct handshake happening to close the connection. When attempting to reconnect however, the syn packets timeout in wireshark and nothing is received from the MCU.

Here is a typical log from the situation:

FreeRTOS_AddEndPoint: MAC: 00-80-e1-00-00-00 IPv4: c0a8f638ip

prvIPTask started

PHY ID 7C130

xPhyReset: phyBMCR_RESET 0 ready

+TCP: advertise: 01E1 config 3100

Autonego ready: 00000004: full duplex 100 mbit high status

Socket 5000 -> [0.0.0.0]:0 State eCLOSED->eTCP_LISTEN

Prot Port IP-Remote       : Port  R/T Status       Alive  tmout Child

TCP  5000 0ip                   :    0 0/0 eTCP_LISTEN       12      0 0/20

FreeRTOS_netstat: 1 sockets 26 < 26 < 40 buffers free

pxEasyFit: ARP c0a8f674ip -> c0a8f673ip

ipARP_REQUEST from c0a8f674ip to c0a8f673ip end-point c0a8f638ip

pxEasyFit: ARP c0a8f674ip -> c0a8f673ip

ipARP_REQUEST from c0a8f674ip to c0a8f673ip end-point c0a8f638ip

pxEasyFit: ARP c0a8f674ip -> c0a8f673ip

ipARP_REQUEST from c0a8f674ip to c0a8f673ip end-point c0a8f638ip

pxEasyFit: ARP c0a8f653ip -> c0a8f601ip

ipARP_REQUEST from c0a8f653ip to c0a8f601ip end-point c0a8f638ip

pxEasyFit: ARP c0a8f638ip -> c0a8f662ip

ipARP_REPLY from c0a8f662ip to c0a8f638ip end-point c0a8f638ip

Gain: Socket 5000 now has 1 / 20 child me: 0x2404d700 parent: 0x2404d4f0 peer: 0x2404d700

prvSocketSetMSS: 1460 bytes for c0a8f662ip port 56737

Socket 5000 -> [192.168.246.98]:56737 State eCLOSED->eSYN_FIRST

prvWinScaleFactor: uxRxWinSize 1 MSS ip
                                       0 Factor 0

Socket 5000 -> [192.168.246.98]:56737 State eSYN_FIRST->eSYN_RECEIVED

ipARP_REQUEST from c0a8f674ip to c0a8f673ip end-point c0a8f638ip

TCP: passive 5000 => 98.246.168.192 port 56737 set ESTAB (scaling 1)

Socket 5000 -> [192.168.246.98]:56737 State eSYN_RECEIVED->eESTABLISHED

prvAcceptWaitClient: client 0x2404d700 parent 0x2404d4f0

Prot Port IP-Remote       : Port  R/T Status       Alive  tmout Child

TCP  5000 0ip                   :    0 0/0 eTCP_LISTEN    33257      0 1/20

TCP  5000 c0a8f662ip      :56737 1/0 eESTABLISHED      20  19934

FreeRTOS_netstat: 2 sockets 23 < 26 < 40 buffers free

pxEasyFit: ARP c0a8f674ip -> c0a8f673ip

ipARP_REQUEST from c0a8f674ip to c0a8f673ip end-point c0a8f638ip

pxEasyFit: ARP c0a8f674ip -> c0a8f673ip

ipARP_REQUEST from c0a8f674ip to c0a8f673ip end-point c0a8f638ip

pxEasyFit: ARP c0a8f674ip -> c0a8f673ip

ipARP_REQUEST from c0a8f674ip to c0a8f673ip end-point c0a8f638ip

pxEasyFit: ARP c0a8f674ip -> c0a8f673ip

ipARP_REQUEST from c0a8f674ip to c0a8f673ip end-point c0a8f638ip

Socket 5000 -> [192.168.246.98]:56737 State eESTABLISHED->eLAST_ACK

vTCPStateChange: Closing (Queued 0, Accept 0 Reuse 0)

vTCPStateChange: me 0x2404d700 parent 0x2404d700 peer 0 clear 0

vTCPStateChange: xHasCleared = 0

Socket 5000 -> [192.168.246.98]:56737 State eLAST_ACK->eCLOSE_WAIT

Callback_OnConnectDisconnect called with ulConnected = 0

Disconnect detected

After shutdown

About to take mutex

Mutex taken successfully

Free buffers pre close: 25

closing mutex returned

Lost: Socket 5000 now has 0 / 20 children

FreeRTOS_closesocket[0ip port 5000 to c0a8f662ip port 56737]: buffers 26 socks 1

Prot Port IP-Remote       : Port  R/T Status       Alive  tmout Child

TCP  5000 0ip                             :    0 0/0 eTCP_LISTEN    37995      0 0/20

FreeRTOS_netstat: 1 sockets 21 < 26 < 40 buffers free

pxEasyFit: ARP c0a8f674ip -> c0a8f673ip

ipARP_REQUEST from c0a8f674ip to c0a8f673ip end-point c0a8f638ip

pxEasyFit: ARP c0a8f674ip -> c0a8f673ip

ipARP_REQUEST from c0a8f674ip to c0a8f673ip end-point c0a8f638ip

pxEasyFit: ARP c0a8f674ip -> c0a8f673ip

ipARP_REQUEST from c0a8f674ip to c0a8f673ip end-point c0a8f638ip

pxEasyFit: ARP c0a8f674ip -> c0a8f673ip

ipARP_REQUEST from c0a8f674ip to c0a8f673ip end-point c0a8f638ip

pxEasyFit: ARP c0a8f674ip -> c0a8f673ip

ipARP_REQUEST from c0a8f674ip to c0a8f673ip end-point c0a8f638ip

pxEasyFit: ARP c0a8f674ip -> c0a8f673ip

ipARP_REQUEST from c0a8f674ip to c0a8f673ip end-point c0a8f638ip

pxEasyFit: ARP c0a8f674ip -> c0a8f673ip

ipARP_REQUEST from c0a8f674ip to c0a8f673ip end-point c0a8f638ip

pxEasyFit: ARP c0a80078ip -> c0a80078ip

ipARP_REPLY from c0a80078ip to c0a80078ip end-point c0a8f638ip

pxEasyFit: ARP c0a8f647ip -> c0a8f601ip

pxEasyFit: ARP c0a8f674ip -> c0a8f673ip

pxEasyFit: ARP c0a8f674ip -> c0a8f673ip

pxEasyFit: ARP c0a8f674ip -> c0a8f673ip

pxEasyFit: ARP c0a8f662ip -> c0a8f638ip

pxEasyFit: ARP c0a8f674ip -> c0a8f673ip

pxEasyFit: ARP c0a8f60aip -> c0a8f673ip

pxEasyFit: ARP c0a8f662ip -> c0a8f638ip

pxEasyFit: ARP c0a8f674ip -> c0a8f673ip

pxEasyFit: ARP c0a8f60aip -> c0a8f673ip

pxEasyFit: ARP c0a8f662ip -> c0a8f638ip

pxEasyFit: ARP c0a8f674ip -> c0a8f673ip

pxEasyFit: ARP c0a8f60aip -> c0a8f673ip

pxEasyFit: ARP c0a8f60aip -> c0a8f673ip

pxEasyFit: ARP c0a8f60aip -> c0a8f673ip

pxEasyFit: ARP c0a8f60aip -> c0a8f673ip

*** Warning *** only 2 buffers left

pxEasyFit: ARP c0a8f662ip -> c0a8f638ip

pxEasyFit: ARP c0a8f662ip -> c0a8f638ip

pxEasyFit: ARP c0a8f674ip -> c0a8f673ip

HAL_ETH_RxAllocateCallback: failed

pxEasyFit: ARP c0a8f662ip -> c0a8f638ip

HAL_ETH_RxAllocateCallback: failed

HAL_ETH_RxAllocateCallback: failed

The many unresponded to pxEasyFit calls come right after trying to reconnect the client. In wireshark, there are 4 timed out syn packets sent by the client and not responded to by the MCU server. Wireshark does however show a decent amount of ARP traffic. This is my first time working with FreeRTOS+TCP and FreeRTOS in general, so it’s quite possible I’m just making a dumb error.