ReleaseUDPPayloadBuffer but for TCP?

I’m using FreeRTOS_recv() with FREERTOS_ZERO_COPY which returns a pointer to available data and the length. The problem is that when I call FreeRTOS_recv() again it returns the same data because the circular buffer pointer is not advanced. I agree it shouldn’t be advanced until the application actually consumes the data but I can’t find the function that does it nor a FREERTOS_ZERO_COPY TCP example. Below is the function that I added that does what I need and my application code, but what am I missing?

void FreeRTOS_ReleaseTCPPayload( Socket_t xSocket,
                                 BaseType_t xByteCount )
{
    FreeRTOS_Socket_t * pxSocket = ( FreeRTOS_Socket_t * ) xSocket;
    size_t uxNextTail = pxSocket->u.xTCP.rxStream->uxTail + xByteCount;
    if( uxNextTail >= pxSocket->u.xTCP.rxStream->LENGTH )
    {
        uxNextTail -= pxSocket->u.xTCP.rxStream->LENGTH;
    }
    pxSocket->u.xTCP.rxStream->uxTail = uxNextTail;
}
         /* Receive data on the socket. */
         lBytes = FreeRTOS_recv( xSocket, &pucRxBuffer, 0, FREERTOS_ZERO_COPY );

         /* If data was received then process it. */
         if( lBytes >= 0 )
         {
            ProcessBuffer(pucRxBuffer, lBytes);
            extern void FreeRTOS_ReleaseTCPPayload( Socket_t xSocket, BaseType_t xByteCount );
            FreeRTOS_ReleaseTCPPayload( xSocket, lBytes );
         }

I will answer this question within an hour. Hein

UDP

when calling FreeRTOS_recvfrom() with the zero-copy flag FREERTOS_ZERO_COPY, the ownership of a network buffer is transferred from the socket to the application. The contents will be used and the network buffer will be released.

TCP: unlike UDP, TCP data is treated as a stream. All TCP data is stored in the socket in stream buffers, one for transmission and one for reception.

Receive TCP data without copying:

Here is an example taken from the FTP-server project that shows zero-copy reception:

    char * pcBuffer = NULL;
    const size_t uxMaxCount = 10240U;

    /* The "zero-copy" method, pass the address of a pointer: */
    xRc = FreeRTOS_recv( xTransferSocket, ( void * ) &( pcBuffer ),
                         uxMaxCount, FREERTOS_ZERO_COPY | FREERTOS_MSG_DONTWAIT );

    if( xRc > 0 )
    {
        ulRecvBytes += xRc;
        xWritten = ff_fwrite( pcBuffer, 1, xRc, pxWriteHandle );
        /* In the following call the received data is discarded: */
        FreeRTOS_recv( xTransferSocket, ( void * ) NULL, xRc, 0 );
        /* The above line shows the answer to your question. */
    |

As you see, FreeRTOS_recv() is called two times:

  1. With a pointer to a buffer pointer and the maximum number of bytes to receive.
  2. With NULL and the actual number of bytes received.

Note that in this example the streambuffer is passed directly to the disk driver when calling ff_fwrite().

This method works well, but you really have to know what you are doing, it is non-standard and meant to be used for advanced users. That is why there was no formal documentation about it, except in some posts on the forum.

Sending TCP data without copying:

FreeRTOS_send() can also be used in a zero-copy way, and it is a bit more complex:

    #define SECTOR_LENGTH    512U

    BaseType_t xBufferLength;

    /* FreeRTOS_get_tx_head() returns a pointer to the TX stream and
     * sets xBufferLength to know how much space there is left for writing. */
    pcBuffer = ( char * ) FreeRTOS_get_tx_head( pxClient->xTransferSocket, &xBufferLength );

    if( ( pcBuffer != NULL ) && ( xBufferLength >= SECTOR_LENGTH ) )
    {
        /* Will read disk data directly to the TX stream of the socket. */
        uxCount = FreeRTOS_min_uint32( uxCount, ( uint32_t ) xBufferLength );
    }
    else
    {
        /* Fall-back to the normal file i/o buffer. */
        pcBuffer = pcFILE_BUFFER;
        uxCount = sizeof pcFILE_BUFFER;
    }

    if( uxCount > 0U )
    {
        /* Read at most 'uxCount' bytes from disk. */
        uxItemsRead = ff_fread( pcBuffer, 1, uxCount, pxClient->pxReadHandle );

        if( pcBuffer != pcFILE_BUFFER )
        {
            /* When FreeRTOS_send gets a NULL pointer, it assumes that the
             * transmission stream buffer already contains the data to be sent,
             * in other words, a memcopy() is skipped. */
            pcBuffer = NULL;
        }
        /* When zero-copy is used, pcBuffer will be NULL. The call is necessary
         * to advance the head pointer in the TX stream buffer. */
        xRc = FreeRTOS_send( pxClient->xTransferSocket, pcBuffer, uxCount, 0 );
    }

In other words, for sending zero-copy there are two comparable steps:

  1. FreeRTOS_get_tx_head() will return a pointer to the TX stream buffer, tell how many bytes can be written at that address.
  2. FreeRTOS_send() is called with NULL and the number of bytes that have been written to the stream buffer.

I don’t mind to add documentation about “Using TCP with zero-copy methods”, but the text will start with a warning: please understand what you are doing.

I hope it is all clear now.
Hein

Hi Matt, I read your post on my smartphone and I missed the most important thing: the proposal to add a function FreeRTOS_ReleaseTCPPayload(). Yes that is a great idea, thank you.

If you want you can prepare a PR for this ( we can help you when necessary ), or we can create it.

I find is useful because because it makes more compatible with the UDP approach. Also it has some sanity checks.

Only still don’t know how to handle zero-copy TCP transmissions in simple way…

Thanks,

Hi Hein,

Thanks for the info and example code for using TCP with zero copy. I haven’t tried sending TCP with zero copy yet - my application just lent itself to receiving with zero copy and based on the UDP examples I just looked for the TCP counterpart to release the data but couldn’t find it. The FTP is the one example I didn’t check - I don’t think the HTTP example uses zero copy. In any case yeah I can make a PR for my suggested solution. And I will take a look at sending. Thanks again.

For the sake of completeness my suggested function didn’t fully work. When sending a large burst of data to the socket it would get stuck. I tested without the zero copy and it worked, and I tested with the recv(NULL) method and it worked so apparently the call to recv() updates the socket state and does more than just advance the pointer. So I replaced my function with the call to recv(NULL):

void FreeRTOS_ReleaseTCPPayload( Socket_t xSocket,
                                 BaseType_t xByteCount )
{
    /* Call recv with NULL to advance the stream buffer. */ 
    FreeRTOS_recv( xSocket, NULL, xByteCount, 0 );
}

Hein - you suggested that maybe a dedicated function could be used because additional checks could be performed so maybe a dedicated function still makes sense, but maybe a just a macro would suffice.

In any case I didn’t want anyone reading this thread to be mislead by my previous solution. I will still do a PR for this feature but I wanted to first test more and glad I did, but also wanted to look at zero copy send - I think the HTTP example function prvSendFile() should be easy to modify for zero copy.

Matt

Thank you for reporting back.

The call to FreeRTOS_recv( xSocket, NULL, xCount, 0 ) will have side effects because the socket will get more space in the RX stream buffer. It may wake-up the IP-task, which may send out a WIN update to the TCP peer.

I have used this zero-copy method extensively in the FTP server, and without problems.
Not sue if the DONTWAIT flag has any influence:

    xRc = FreeRTOS_recv( pxClient->xTransferSocket,
                         ( void * ) pcBuffer,
                         sizeof( pcFILE_BUFFER ), FREERTOS_MSG_DONTWAIT );

If you want you can show me some of your TCP code?

Hi Hein,

This looks a bit suspect because in the zero copy case there is no application buffer and I assume the size should be the number previously obtained.

    xRc = FreeRTOS_recv( pxClient->xTransferSocket,
                         ( void * ) pcBuffer,
                         sizeof( pcFILE_BUFFER ), FREERTOS_MSG_DONTWAIT );

I didn’t notice a difference when adding the flag but I will have to trace through the code to see what it does.

I am testing using a modified version of the simple TCP echo example - modified to use a single task using select. But I am still having trouble when I blast it with a chunk of data. I connect to port 7 with putty in raw mode. If I just type or paste small blocks of data it seems okay but if I paste a big block of text (for example copy/paste the complete SimpleTCPEchoServer.c file) then it may work a few times but eventually it gets stuck. It appears to get stuck on the recv side because if I add additional test code to send other data it is sent okay. When it gets in this mode I see messages like this in my serial console

SACK[7,59737]: optlen 12 sending 227155 - 228158
SACK[7,59737]: optlen 12 sending 228158 - 228310

I don’t think the problem is in the network interface or below because other sockets and ping still work, I can open a second connection to echo and it works, and I don’t see any signs of trouble in my serial console. I will try to attach the code here.

I have tried the FTP sample application and it seems to work but performance is dismal. The performance of the HTTP example seems dismal as well. However I am using fatfs by chang not plusFAT so I had to modify for the different file API but I don’t think the slowness is in the file io it is in the networking but that is a topic for another thread.

EDIT: I don’t know what I did or what changed but all of a sudden the HTTP server performance is zippy so scratch those comments for now. I don’t have the FTP server included at the moment so I will have to retest that.

Thanks,

Matt

SimpleTCPEchoServer.c (13.1 KB)

Thanks for reporting Matt.

You wrote:

This looks a bit suspect because in the zero copy case there is no application buffer
I’m sorry, I copy/pasted the wrong call to recv(), it should have been this one:

    xRc = FreeRTOS_recv( pxClient->xTransferSocket,
                         ( void * ) &pcBuffer,
                         0x20000u,
                         FREERTOS_ZERO_COPY | FREERTOS_MSG_DONTWAIT );

You can stop the investigation for now because I will first do more more testing.

I have tried the FTP sample application and it seems to work but performance is dismal

Of course, the speed of the FTP server depends much on how much RAM can be made available, on the CPU, and mostly: the speed of the MMC/SD card interface. Today I am using a fast Xilinx Zynq. Writing is OK, and reading is super fast.

There was one mistake in the code:

    FreeRTOS_closesocket( xSockets[ xInst ] );
    FreeRTOS_FD_SET( xSockets[ xInst ], xSocketSet, eSELECT_ALL );
    xSockets[ xInst ] = NULL;

After calling closesocket() the handle becomes invalid, it is freed from the heap. The correct order should be:

    /* Disconnect the socket from the socket-set. */
    FreeRTOS_FD_CLR( xSockets[ xInst ], xSocketSet, eSELECT_ALL );
	/* Free the memory in the heap. */
    FreeRTOS_closesocket( xSockets[ xInst ] );
	/* Forget the pointer. */
    xSockets[ xInst ] = NULL;

and also FreeRTOS_FD_SET should have been FreeRTOS_FD_CLR. It will disconnect the socket from the socket-set.

But that was not causing your problem.

I played for a long time with the module, changed it here and there, but beside the above, I could not find a real error.

Here is my version so you can try it also: SimpleTCPEchoServer2.c (9.3 KB)

I both tested with puTTY as well as with a tool called packetsender. It is easy to use it as a TCP client. It will connect - wait for reply - close the connection, all within a second. Note it has a button “Load File”.
If you are working on Linux you can also use hping or any other tool.

When you still see a problem when USE_ZERO_COPY is enabled, can you please make a complete PCAP. Before saving the PCAP you can filter the packets with “tcp.port==7”. You can attach it in a ZIP file.
It is also interesting to see the application logging, either through the serial connection or UDP logging.

Here a picture of packetsender:

Hi Hein,

Thanks for the code review and the correction. It is 2AM here and I was about to go to bed when I received email notification of your reply.

As I said the HTTP demo performance drastically improved for some reason so I did retest the FTP demo and it is much better also. When I tested it previously it took 2 minutes to transfer I think a 2MByte file and now it is down to seconds. I’m not sure what changed. I did update to V2.3.3-Patch-1 last night from V2.3.3 but don’t recall a difference immediately afterwards. But earlier today the HTTP performance was so good I thought my web browser had cached it but no it was from the server.

In any case I gave your echo server a quick try but the behavior is the same. I will look at it tomorrow and upload a capture from wireshark.

I am using primarily windows but I will test with linux to see if there is a difference. Ultimately it will be used with linux. FYI my network interface is USB/RNDIS.

Thanks again,

Matt

Hi Hein,

The problem actually seems to be with putty. When I was preparing to capture with wireshark I noticed that when it got into the bad mode that putty stopped sending. I tested with PacketSender and netcat and didn’t experience the issue. However with netcat and my version I see frequent stalls and a retransmission in wireshark but with your version I don’t see that. So I will have to investigate that further and compare our versions and will let you know what I find.

Thanks for all your help,

Matt

Glad to hear that you are making progress on this. I have kept on testing and comparing the 2 versions of SimpleTCPEchoServer, with or without USE_ZERO_COPY.

I found that packet-sender has an option called “Persistent TCP Connections”. Activate it, press send, and a new window will open for the current connection. Just like you did with puTTY: a continuous connection.

Also I tried puTTY and ncat but I have not been able yet to replicate the problem yet with any of the versions.

Upgrading my library to the latest release did change anything.

I assume that you have checked that the network buffers or the heap are not exhausted?

Hi Hein,

Your version was configured to not use zero-copy and mine was which apparently was the difference. When I configure yours to use zero-copy it also has the stalls/retransmits and when I configure mine to not use the zero-copy it does not have the stalls.

I suspect this is because with zero-copy the receive buffer is not getting freed immediately. As an experiment I tried making the non-zero-copy use the zero-copy recv() call:

	#define USE_ZERO_COPY    0
								#if USE_ZERO_COPY
									lBytes = FreeRTOS_recv( xSockets[ xInst ], &( pucRxBuffer ), 0, FREERTOS_ZERO_COPY );
								#else
									static uint8_t ucRxBuffer[ 2048 ];
									//pucRxBuffer = ucRxBuffer;
									lBytes = FreeRTOS_recv( xSockets[ xInst ], &pucRxBuffer, sizeof( ucRxBuffer ), FREERTOS_ZERO_COPY );
									if( lBytes > 0 )
									{
										memcpy( ucRxBuffer, pucRxBuffer, lBytes );
										FreeRTOS_ReleaseTCPPayload( pucRxBuffer, xSockets[ xInst ], lBytes );
									}
									pucRxBuffer = ucRxBuffer;
								#endif

This also works with no stalls as you would expect since it should be essentially the same as the true non-zero-copy case. One subtle difference though is that when the circular buffer wraps this will result in two blocks being received while the true non-zero copy would receive it as one block.

I do not see any messages in my serial console of data loss (malloc, buffers, etc). I have these defined FreeRTOS_printf and FreeRTOS_debug_printf, and enabled these iptraceFAILED_TO_OBTAIN_NETWORK_BUFFER, iptraceETHERNET_RX_EVENT_LOST, and iptraceSTACK_TX_EVENT_LOST.

I suspected that because of the zero-copy the buffer is not freed immediately and data was being dropped at the input of the circular buffer, but I thin I see in the IP code where that would be and there are printf’s there but as I said I don’t see any messages in my console - this is all I see:

Gain: Socket 7 now has 1 / 2 child
prvSocketSetMSS: 1460 bytes for c0a88a01ip:65391
Socket 7 -> c0a88a01ip:65391 State eCLOSED->eSYN_FIRST
prvWinScaleFactor: uxRxWinSize 2 MSS 1460 Factor 0
Socket 7 -> c0a88a01ip:65391 State eSYN_FIRST->eSYN_RECEIVED
TCP: passive 7 => c0a88a01ip:65391 set ESTAB (scaling 1)
Socket 7 -> c0a88a01ip:65391 State eSYN_RECEIVED->eESTABLISHED

Once it gets into this ‘mode’ it seems to persist to the end of the file - after it completes and I send again it seems okay (although the behavior can reoccur during that send).

Attached is a wireshark capture of a send after it is in this mode. This is with your version with only one change to enable zero-copy.

The PacketSender log seems odd when in this mode too - I see only the transfers from the echo server, not to. When it is working ‘normally’ I see tranfers both ways.

image

My echo server task priority is pretty low (below the freertos timer task, the IP task, and the USB stack event handler task). But the application is otherwise lightly loaded so the echo server shouldn’t be blocked by any other application tasks).

So I’m not sure what to make of it all but maybe something here will make senses to you.

Thanks,

Matt
echo.zip (38.9 KB)

Thanks again for your investigations.
One correction: I did try both versions, with and without ZERO_COPY.
Would you mind to share your FreeRTOSIPConfig.h and your FreeRTOSConfig.h?
Later today I’ll study your findings.
Thank so far

Hi Matt,

Some remarks about the progress we’ve made:

  1. There were unnecessary retransmissions because the IP-stack is impatient.

Recently I created a patch that should avoid this behaviour: “TCP minimum time for retransmissions (PR #387)”. See also this post.

For now you can try this change in FreeRTOS_TCP_WIN.c

-#define winSRTT_CAP_mS    50 /**< Cap in milliseconds. */
+#define winSRTT_CAP_mS  1000 /**< Cap in milliseconds. */

This avoids unnecessary retransmissions. But it doesn’t solve you problem.

  1. You found that the communication sometimes stalls for 5 seconds.

Could it be that FreeRTOS_send() can not deliver its data, and it will block for the maximum of 5 seconds?

Note that:

    static const TickType_t xSendTimeOut = pdMS_TO_TICKS( 5000 );

I think that FreeRTOS_send() blocks because of invalid settings, described in the next item 3).

  1. I noticed that the socket in your application has very little buffer space.
xWinProps.lTxBufSize = ipconfigTCP_TX_BUFFER_LENGTH;     // 1000 bytes
xWinProps.lTxWinSize = configECHO_SERVER_TX_WINDOW_SIZE; // 2 segments
xWinProps.lRxBufSize = ipconfigTCP_RX_BUFFER_LENGTH;     // 1000 bytes
xWinProps.lRxWinSize = configECHO_SERVER_RX_WINDOW_SIZE; // 2 segments;

The sizes of the buffers are far too small, less than a TCP segment.

These settings are passed to a socket with the socket option FREERTOS_SO_WIN_PROPERTIES.

Maybe we should adapt the handling in FreeRTOS_setsockopt() of the FREERTOS_SO_WIN_PROPERTIES option and refuse wrong settings.

It should be true that:

lTxBufSize >= lTxWinSize * ipconfigTCP_MSS
lRxBufSize >= lRxWinSize * ipconfigTCP_MSS

The help page shows this example:

    /* Fill in the required buffer and window sizes. */
    /* Unit: bytes */
    xWinProps.lTxBufSize = 4 * ipconfigTCP_MSS;
    /* Unit: MSS */
    xWinProps.lTxWinSize = 2;
    /* Unit: bytes */
    xWinProps.lRxBufSize = 4 * ipconfigTCP_MSS;
    /* Unit: MSS */
    xWinProps.lRxWinSize = 2;

It defines:

  • lTxBufSize: The size of the internal TX buffer (bytes)
  • lTxWinSize: The maximum TCP WIN size for transmissions ( unit MSS, e.g. 1460 bytes )
  • lRxBufSize: The size of the internal TX buffer (bytes)
  • lRxWinSize: The maximum TCP WIN size for reception ( unit MSS, e.g. 1460 bytes )

Transmission buffers are allocated as soon as they are needed: on reception of the first byte, or when sending the first packet. The configured buffer and WIN sizes are fixed for the life time of the socket.

Thanks,