FreeRTOS+TCP - issue getting echo server to reply

Dave · August 5, 2020, 7:51am

For background, I have been working on getting FreeRTOS+TCP (10.3.1) ported to the iMX.RT1064 EVK and have been working through some issues with the Ethernet functionality. For background I have the kernel outputting the gratuitous ARP periodically as expected. The target also works reliable in terms of replying to pings sent to it. I am now working on verifying TCP and UDP operation. A simple UDP send loop on the target shows up on Wireshark as expected with the expected data in the payload. TCP has been giving me issues though.

I have copied the simpleTCPEchoServer for the demo in the release dnload.
Using putty I can open a telnet connection (telnet negotiation off) to the target (port 7).
When characters are typed from putty (‘1’, CRLF), I see them transmitted as 2 packets in Wireshark and the target immediately sends 2 acks back for them.
The echo server task gets the 3 characters from FreeRTOS_recv and then calls FreeRTOS_send. Send payload is verified to to be the 3 characters received ('1’CRLF)
FreeRTOS_send puts the 3 characters in the txStream for the socket. Proper content is verified in the stream buffer memory, stream head and front incremented to 3.
FreeRTOS_send signals IPTask with an eTCPTimerEvent which I assume is supposed to trigger the stream output at some point (code below).
/* Send a message to the IP-task so it can work on this
socket. Data is sent, let the IP-task work on it. */
pxSocket->u.xTCP.usTimeout = 1u;
```
 			if( xIsCallingFromIPTask() == pdFALSE )
 			{
 				/* Only send a TCP timer event when not called from the
 				IP-task. */
 				xSendEventToIPTask( eTCPTimerEvent );
 			}
```
At this point the data never goes out and a breakpoint in xNetworkInterfaceOutput never gets to send the TCP reply (only to send ARP broadcasts). Additional characters sent are not Acked so it appears the TCP stack is no longer processing data.
Breaking app and looking at the socket descriptor still shows the data in the txStream.

I would appreciate any suggestions on what might be causing this or areas to focus on in terms of tracing this out and determining the issue. I am new to FreeRTOS+TCP so this has been a slow process walking through the TCP/IP code so any help would be appreciated.

Wireshark trace of a typical example is available.

Thank you,

Dave

htibosch · August 5, 2020, 11:15am

Dave, thanks for this report. Although it is very precise, would it be possible to create a PCAP from it. If you zip it, you can attach it to your post.

Also it might be useful to see the two config files that you are using: FreeRTOSConfig.h and FreeRTOSIPConfig.h.

Thanks

Dave · August 5, 2020, 4:42pm

Hein,

Files are attached. Sorry they weren’t there originally but when I did the OP I couldn’t locate the upload button.

Thanks,

Dave

FreeRTOSConfig.h (7.0 KB) FreeRTOSIPConfig.h (16.1 KB) Echo_ackbutnoreply_080520.zip (3.0 KB)

htibosch · August 5, 2020, 5:09pm

Reading your PCAP:

Device x.200 connects to x.100. The SYN and SYN/ACK are OK.
Device x.200 sends 2<CRLF>, and device x100 says: thank you.
There is no echo, just an acknowledgement of the data received.

This makes me doubt if the echo server is still running. The IP-task takes care of :

Receiving the connection
Acknowledging the bytes that are received.

But the server task has to do the recv() and the send().
Is there source code (simpleTCPEchoServer) that I can try out?

Dave · August 6, 2020, 5:03am

Hein,

Thanks for helping me with this. I really thought this would be straight forward to get the canned echo server demo running. Bullets 3-6 were from me tracing through the task in the echo server where it receives the 3 bytes (1) and calls FreeRTOS_send with the reply so I thought the server task was OK. I have since confirmed by pausing after characters received by target (192.168.0.100) and supposedly sent that the echo server tasks are still running and the socket status shows it is still active. Source for the echo server is included as well as a few screen shots after pausing code after echo should have gone out.

Please let me know any other information or files that might help.

Dave
SimpleTCPEchoServer.c (11.8 KB)

htibosch · August 6, 2020, 5:13am

Not sure if you have checked it already, but is there enough heap space? You can check this by setting vApplicationMallocFailedHook().
This week a TCP problem was reported that was caused by a failing call to pvPortMalloc().

I will (re)check the source file that you’ve sent me.

In a meanwhile, if you want, check out this simple telnet server: telnet.zip (2.8 KB)

It is very simple to poll it from any task.

Make sure it is started after the IP-stack is up and running.

htibosch · August 6, 2020, 6:49am

I really thought this would be straight forward to
get the canned echo server demo running

For me the module SimpleTCPEchoServer.c runs well, with one exception: I had estimated the necessary stack space too small ( 400 too small, 640 seems enough ).

Dave · August 6, 2020, 8:08am

Hein,

Thanks for the suggestion. I will verify but pretty sure the code checks for valid returned ptrs for all malloc calls. I will try your telnet app as well. Odd thing is that when I set up to output UDP messages originally they went out fine. I just tried a simple TCP output task that would send a dummy string once putty established a connection on the socket and pretty much the same as with the echoserver, no TCP packets go when FreeRTOS_send is called. I will confirm the UDP still works so at least we can be sure it is related to TCP only.

Thanks,

Dave

Dave · August 7, 2020, 5:01am

Hein,

I have increased my stacks to 900 with no change in behavior. Ack only is sent in reply to first 2 received packets (3 characters total). The received data is shown in the socket rxStream and expected outgoing data is in the socket txStream but doesn’t appear to ever have been sent (see IDE screen show of both structures). I am not as well versed in the inner workings of the TCP/IP stack implementation but I am wondering what is supposed to trigger the outgoing data in the txStream to be sent once FreeRTOS_send put it there? As mentioned in the OP, it appears that the code sets and eTCPTimerEvent to cause someone downstream to check and send data that is ready. Ping, ARP and UDP output all are working from my target so I don’t think it is an issue with the low level send drivers.

Although I know it isn’t 100% foolproof, I have had turned on the stack overflow checks in the kernel. IDE indicates I am not out of heap space as well.

Did you test with the same version, 10.3.1? Did you use my same FreeRTOSconfig and FreeRTOSIPConfig files? Wondering if they have an incorrect setting on my end??

I will focus now and integrating the telnet example you sent.

Thanks,

Dave

htibosch · August 7, 2020, 6:06am

IDE indicates I am not out of heap space as well

I am still not convinced that the amount of heap is enough:

#define configTOTAL_HEAP_SIZE   ((size_t)(20480))

Yes it is possible to run FreeRTOS+TCP with 20 KB of heap, but you will have to fine-tune it ( see below ).

Have you set vApplicationMallocFailedHook() and enabled it by setting
#define configUSE_MALLOC_FAILED_HOOK 1 ?

You have all logging disabled:

#define ipconfigHAS_DEBUG_PRINTF    0
#define ipconfigHAS_PRINTF          0

If you have a way to send logging ( USB ? ) , that could be very helpful.

You will need a function like this:

void vLoggingPrintf( const char *pcFormatString, ... );

Not sure if ipconfigIP_TASK_STACK_SIZE_WORDS is being used in you driver:

#define ipconfigIP_TASK_STACK_SIZE_WORDS ( configMINIMAL_STACK_SIZE * 5 )

But this stack might turn out small.

#define ipconfigNUM_NETWORK_BUFFER_DESCRIPTORS  60

This is a lot for a small application. For a small application 16 network buffers should be enough.

#define ipconfigNETWORK_MTU		1200

If later, you have heavy local TCP traffic, it might be profitable to use a larger MTU, like 1500 bytes to get a better performance.

#define ipconfigTCP_WIN_SEG_COUNT   240

This defines the maximum the number of outstanding TCP segments. 240 is a lot, normally 16 or 32 should be more than enough.

You are using TCP sliding windows. Two macros determine the buffer sizes of a socket, in your case:

/* Each TCP socket has a circular buffers for Rx and Tx, which have a fixed
maximum size.  Define the size of Rx buffer for TCP sockets. */
#define ipconfigTCP_RX_BUFFER_LENGTH			( 1000 )

/* Define the size of Tx buffer for TCP sockets. */
#define ipconfigTCP_TX_BUFFER_LENGTH			( 1000 )

The buffers are smaller than your MSS ( which is the maximum size of the payload of a TCP packet ). Here is how MSS is calculated:

#define ipconfigTCP_MSS  ( ipconfigNETWORK_MTU - ( ipSIZE_OF_IPv4_HEADER + ipSIZE_OF_TCP_HEADER  ) )
#define ipconfigTCP_MSS  ( 1200 - ( 20 + 20  ) )
#define ipconfigTCP_MSS  ( 1160 )

The TX or RX buffer size should be at least ipconfigTCP_MSS, which is 1160 in your case, so why not:

#define ipconfigTCP_RX_BUFFER_LENGTH			( 2U * ( ipconfigTCP_MSS ) )
#define ipconfigTCP_TX_BUFFER_LENGTH			( 2U * ( ipconfigTCP_MSS ) )

If your project has logging ( see above ), you can use tcp_mem_stats.c
There is a tcp_mem_stats.md that explains how to use it. Like setting:

#define ipconfigUSE_TCP_MEM_STATS   1

in your FreeRTOSIPConfig.h.

htibosch · August 7, 2020, 7:22am

I just remembered that the TCP windows options are set in the demo, using the socket option FREERTOS_SO_WIN_PROPERTIES:

#if( ipconfigUSE_TCP_WIN == 1 )
    WinProperties_t xWinProps;

    /* Fill in the buffer and window sizes that will be used by the socket. */
    xWinProps.lTxBufSize = ipconfigTCP_TX_BUFFER_LENGTH;     // = 1000
    xWinProps.lTxWinSize = configECHO_SERVER_TX_WINDOW_SIZE; // = 2 x MSS
    xWinProps.lRxBufSize = ipconfigTCP_RX_BUFFER_LENGTH;     // = 1000
    xWinProps.lRxWinSize = configECHO_SERVER_RX_WINDOW_SIZE; // = 2 x MSS
#endif /* ipconfigUSE_TCP_WIN */

FreeRTOS_setsockopt( xListeningSocket,
                     0,
                     FREERTOS_SO_WIN_PROPERTIES,
                     ( void * ) &xWinProps,
                     sizeof( xWinProps ) );

Note that the buffer length is expressed in byte, and the TCP window size is expressed in a number of MSS.

I just tried out my project with you FreeRTOSIPConfig.h. I had to make ipconfigNUM_NETWORK_BUFFER_DESCRIPTORSa lot smaller than 60 before it would run.
After doing so the echo server ran OK without a problem.
Although it will have a problem when large packets are received.

Dave · August 7, 2020, 9:37am

Hein,

Thanks for the detailed points and explanation. Just reducing ipconfigNUM_NETWORK_BUFFER_DESCRIPTORS didn’t resolve it (tried as a quick check). I am working through the list of items you suggested. Most of those settings were left as-is in their default state in the .h files. I’ll try to the built in debug messages ported out as well.

Thanks,

Dave

Dave · August 8, 2020, 5:37am

Hein,

Thanks a lot for your help. Sorry to have taken up your time on this. You nailed it with the heap issue. Increasing the heap resolved the problem. Adding the MallocFailedHook confirmed I was failing when trying to create the new echo tasks with the original heap size. Now that I am getting more familiar with FreeRTOS and my target I need to get all of the failure hooks enabled as well as the kernel DEBUG_PRINTF functionality so these things are easier for me to identify early on.

Dave

htibosch · August 8, 2020, 9:50am

Dave,
Very good, thank you for reporting this back.

I also think that logging and all error hooks are good to use during development.

The windows simulator project is also useful, because the runtime environment checks the validity of all memory access.

My problem is often that I need Ethernet for logging/debugging while Ethernet isn’t running yet. In some cases I logged to a circular array of strings, which I could read during a program break.

This page about tcp_mem_stats.c shows more insight into the memory usage of a FreeRTOS+TCP application.