TCP/IP stack produces spuriouse retransmissions when using multiple sockets

Libraries
1

Dear community,

In short:
Basically I’m experiencing that sending data continously from two sockets contemporarly does not work well in FreeRTOS, as it produces lot’s of spuriouse retransmissions and the packet stream lags a lot.
Can anybody confirm this behaviour?

Detailed description:
I am using FreeRTOS V10.2.1 on a Zynq platform (ARM Cortex A9) with the FreeRTOS Plus TCP/IP stack V2.4.0.

In my test scenario I create two TCP sockets that are both configured as servers and bind on different ports (10000 & 20000).

The application consists of a task that sends a continuous stream of packets to connected client, for each socket. The sockets are serviced (written to) in the same task, one after another. The packets are sent at a 100 ms rate for each socket (10 Packets / second / socket).
After sending a packet on each socket the task just sleeps until the 100 ms expired.
There are no other active tasks.

The priorities are defined as such:

#define tskIDLE_PRIORITY			        ( 0 )
#define configMAX_PRIORITIES				( 7 )

#define ipconfigIP_TASK_PRIORITY			( configMAX_PRIORITIES - 1 ) //Highest
#define	mainTASK_PRIORITY					( tskIDLE_PRIORITY + 2 )

The packets sent by the servers are very short and have the following format:

	typedef struct __attribute__((packed))
	{
		char frameStart[3];   //a random string like "sta"
		uint32_t serialNum;   // an upcounting number, is increased for each packet sent
		uint8_t dummy[16];    // some random data
	}ipstack_perf_payload_t;

The client side is a minimalistic python program on Windows 10 that connects as a client to each server and just prints out the serial number.
client.py.zip (774 Bytes)

Client and server are connected through a LAN cable with now switches in between.

When sending a stream of packets from both sockets, the data stream is interrupted randomly, repeatedly for a few seconds and I get a load of TCP spuriouse retransmissions in wireshark.

When trying the same with a single socket I don’t get any retransmissions at all, even when sending with up to 2 ms rate.
Can sombody explain this behaviour?

I’ll attach the wireshark trace and the FreeRTOSIPConfig.h
WiresharkTrace.zip (8.0 KB)
FreeRTOSIPConfig.h (20.9 KB)

To not make this post even longer and more confusing I opted to leave out the testcode in FreeRTOS for now which really only consists of two sockets being created as TCP servers and waiting for a client to start streaming data. But if you’d still like to see it I’ll attach it in a reply.

If any additional informations are needed to verify/reproduce the behaviour please don’t hesitate to ask.

Thanks for your time!

2

I will let others add their thoughts, the guideline around priorities when using TCP stack is the highest priority goes to the network driver, the next for the IP task and the next is the task which uses the TCP stack. Can you share the priority assigned to the networkinterface, please?, I think it defaults to configMAX_PRIORITIES - 1. If that is the case can you set the IP task priority to configMAX_PRIORITIES -2 and give it a shot?

1 Like
3

@NikhilKamath Thanks for your fast reply.

I tried with the priorities configured as such:

#define niEMAC_HANDLER_TASK_PRIORITY    configMAX_PRIORITIES - 1
#define ipstack_PRIORITY                configMAX_PRIORITIES - 2
#define	mainTASK_PRIORITY               configMAX_PRIORITIES - 3

Unfortunately the problem stays the same.

4

I finally thought of enabling the debug switches for the FreeRTOS TCP/IP stack and found out that it runs out of network buffers, which leads to the errors shown in the wireshark trace.

I’ll post here what it says:

Gain: Socket 10000 now has 1 / 2 child<\n>
prvSocketSetMSS: 1460 bytes for C0A8A811ip:52265<\n>
Socket 10000 -> C0A8A811ip:52265 State eCLOSED->eSYN_FIRST<\n>
prvWinScaleFactor: uxRxWinSize 2 MSS 1460 Factor 0<\n>
Socket 10000 -> C0A8A811ip:52265 State eSYN_FIRST->eSYN_RECEIVED<\n>
TCP: passive 10000 => C0A8A811ip:52265 set ESTAB (scaling 1)<\n>
Socket 10000 -> C0A8A811ip:52265 State eSYN_RECEIVED->eESTABLISHED<\n>
Network buffers: 63 lowest 61<\n>
Network buffers: 62 lowest 60<\n>
Gain: Socket 20000 now has 1 / 2 child<\n>
prvSocketSetMSS: 1460 bytes for C0A8A811ip:52278<\n>
Socket 20000 -> C0A8A811ip:52278 State eCLOSED->eSYN_FIRST<\n>
prvWinScaleFactor: uxRxWinSize 2 MSS 1460 Factor 0<\n>
Socket 20000 -> C0A8A811ip:52278 State eSYN_FIRST->eSYN_RECEIVED<\n>
TCP: passive 20000 => C0A8A811ip:52278 set ESTAB (scaling 1)<\n>
Socket 20000 -> C0A8A811ip:52278 State eSYN_RECEIVED->eESTABLISHED<\n>
Network buffers: 60 lowest 59<\n>
Network buffers: 59 lowest 58<\n>
Network buffers: 57 lowest 57<\n>
Network buffers: 58 lowest 56<\n>
Network buffers: 56 lowest 55<\n>
Network buffers: 55 lowest 54<\n>
Network buffers: 53 lowest 53<\n>
Network buffers: 53 lowest 52<\n>
Network buffers: 51 lowest 51<\n>
Network buffers: 52 lowest 50<\n>
Network buffers: 50 lowest 49<\n>
Network buffers: 48 lowest 48<\n>
Network buffers: 48 lowest 47<\n>
Network buffers: 48 lowest 46<\n>
Network buffers: 46 lowest 45<\n>
Network buffers: 44 lowest 44<\n>
Network buffers: 44 lowest 43<\n>
Network buffers: 42 lowest 42<\n>
Network buffers: 42 lowest 41<\n>
Network buffers: 40 lowest 40<\n>
Network buffers: 40 lowest 39<\n>
Network buffers: 38 lowest 38<\n>
Network buffers: 38 lowest 37<\n>
Network buffers: 37 lowest 36<\n>
Network buffers: 35 lowest 35<\n>
Network buffers: 35 lowest 34<\n>
Network buffers: 34 lowest 33<\n>
Network buffers: 32 lowest 32<\n>
Network buffers: 32 lowest 31<\n>
Network buffers: 29 lowest 29<\n>
Network buffers: 29 lowest 28<\n>
Network buffers: 29 lowest 27<\n>
Network buffers: 27 lowest 26<\n>
Network buffers: 25 lowest 25<\n>
Network buffers: 26 lowest 24<\n>
Network buffers: 24 lowest 23<\n>
Network buffers: 23 lowest 22<\n>
Network buffers: 20 lowest 20<\n>
Network buffers: 20 lowest 19<\n>
Network buffers: 18 lowest 18<\n>
Network buffers: 17 lowest 17<\n>
Network buffers: 15 lowest 15<\n>
Network buffers: 15 lowest 14<\n>
Network buffers: 14 lowest 13<\n>
Network buffers: 12 lowest 12<\n>
Network buffers: 12 lowest 11<\n>
Network buffers: 10 lowest 10<\n>
Network buffers: 11 lowest 9<\n>
Network buffers: 9 lowest 8<\n>
Network buffers: 3 lowest 3<\n>
Network buffers: 2 lowest 2<\n>
Network buffers: 0 lowest 0<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
ulTCPWindowTxGet[52265 - 10000]: Change Tx window: 4380 -> 2920<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 833 < 856<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 3478 < 3616<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 3570 < 3616<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 3593 < 3616<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 925 < 1063<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 971 < 1063<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 994 < 1063<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 1017 < 1063<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 1040 < 1063<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 1086 < 1109<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 3754 < 3800<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 1132 < 1155<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 3869 < 3892<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 1247 < 1408<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 1270 < 1408<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 1293 < 1408<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 3938 < 3984<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 1385 < 1408<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 4536 < 4559<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 1983 < 2144<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 2006 < 2144<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 2029 < 2144<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 2098 < 2144<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 4743 < 4766<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 4835 < 4950<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 4927 < 4950<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 2236 < 2420<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 2259 < 2420<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 2282 < 2420<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 2328 < 2420<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 2374 < 2420<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 2397 < 2420<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 2512 < 2535<\n>
prvTCPWindowFastRetransmit: Requeue sequence number 2558 < 2581<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>
emacps_check_rx: unable to allocate a Network Buffer<\n>

I will look further into it, but for now I don’t really get why the stack does not manage to handle two sockets in time, if it has the highest priority… :thinking:

Edit:

I found out that the problem seems to somehow be related to the windowing functionality of the stack. Changing the window size did not affect the behaviour greatly, but turning-off the windowing functionallity all together solved the problem so far.

/* USE_WIN: Let TCP use windowing mechanism. */
#define ipconfigUSE_TCP_WIN			( 0 )

However, I don’t understand how turning-off windowing improves things in this context, rather than maybe worsening them.

As far as I understand, windowing can be seen as a kind of flow control mechanism, whereas the window size decides over how “often” an ACK is expected, for keeping sending new data.

Can the described behaviour be caused by the python TCP/IP stack on windows (client side) being too slow at sending ACKs which leads to the FreeRTOS sockets not managing to get rid of the data send by the application in time, thus running out of buffers eventually?

Please let me know your theories, I’d really like to understand what’s going on.

5

Hello @Stonebull, would it be possible to share some of the code that you use?
I think of the code that configures the sockets, and that handles the I/O.

And also: would it be possible to make a new PCAP that also shows the beginning of the connection, from the first packet?
Maybe you start with 1 connection for 10 seconds, and then start the second TCP client?

I am surprised that things start going wrong when there are 2 connections…

#define ipconfigUSE_TCP_WIN ( 0 )

There are no known issues with the TCP windowing system.

Your FreeRTOSIPConfig.h looks good to me.

6

I can not look at the pcap from this computer, so here is a shot in the dark: Is it possible that the input side of your device is being kept busy serving rapid broadcast packets (eg if your uplink internet router runs some kind of network management software that your PC routes to your target) and thus runs out of network buffers?

You write that both sides of your communication are directly connected via a LAN cable, so in theory you should see all network traffic directed to your target in the trace, but sometimes that does not hold true (network configurations can take up many different flavors).

I found that the safest way to monitor complete network traffic is to connect your target via an old fashioned dumb hub (I do not think those are being made anymore, I keep mine in a safe because they are so precious) and splice a dedicated machine for monitoring into that segment.

1 Like
7

@RAc wrote:

Is it possible that the input side of your device is being kept busy serving rapid broadcast packets

This reminds me of your FreeRTOSIPConfig.h :

    #define ipconfigARP_CACHE_ENTRIES    6

That is a very small table. Cache entries will easily get cleaned and reused for another IP-address. Can you put that at a safe 32 or so? Although I’m not sure if that solves the problem.

There are more than enough network buffers:

#define ipconfigNUM_NETWORK_BUFFER_DESCRIPTORS		96

In one logging you see that it starts running out of network buffers:

Network buffers: 60 lowest 59<\n>
Network buffers: 59 lowest 58<\n>
Network buffers: 57 lowest 57<\n>
Network buffers: 58 lowest 56<\n>
Network buffers: 56 lowest 55<\n>
...

Any idea why? Is there a UDP socket left unattended?

The problem seems that many packets from DUT to the Python app get lost on-the-way. WireShark doesn’t see them, they must have been dropped.

Now the bigger your WIN size, the more problems you will see. If you make a WIN of 1, there is a strict order and the problems will be minimal.

Somewhere the outgoing packets gets dropped, somewhere between xNetworkInterfaceOutput() and the delivery to DMA?

Would it be possible to try out a newer version of the Zynq driver?

EDIT : the Zynq driver hasn’t changed much since V2.4.0
But please make sure it uses non-cached memories from uncached_memory.c.

1 Like
8

@htibosch, @RAc thanks for your replies!

Of course, I’ll append a main.c that contains the application, and a uport.c that contains the lib that I wrote that creates a TCP/IP socket, which is going to be used by the application.
Note: The application does not compile as is, as many files are missing (like the FreeRTOS sources) but the application part should be present well enough to understand what’s happening.
main.c (8.0 KB)
uport.c (35.9 KB)

Sure!
WiresharkTrace_1.zip (55.9 KB)

Thanks for the hint! I tried with 32, but it does not seem to affect the problems that occure as soon as ipconfigUSE_TCP_WIN is ‘1’.

These logs that show the network buffers decreasing are printed very rapidly down to 0 as soon as I connect the second socket. While the system has only one socket connected, the number of buffers decreases a little, but it stops at: Network buffers: 61 lowest 59 and remains stable after that.

I am very confident that there are no other open sockets, as this is a small test application that really just opens just these two TCP sockets which are then connected to the python clients. Apart from the two listening sockets that wait for new clients, inside the uport.c lib.

I’m not sure how to check that but I will look into it.

The network stack that I’m using makes use of uncached_memory.c inside x_emacpsif_dma.c:

XStatus init_dma( xemacpsif_s * xemacpsif )
{
    NetworkBufferDescriptor_t * pxBuffer;

    int iIndex;
    UBaseType_t xRxSize;
    UBaseType_t xTxSize;
    struct xtopology_t * xtopologyp = &xXTopology;

    xRxSize = ipconfigNIC_N_RX_DESC * sizeof( xemacpsif->rxSegments[ 0 ] );

    xTxSize = ipconfigNIC_N_TX_DESC * sizeof( xemacpsif->txSegments[ 0 ] );

    xemacpsif->uTxUnitSize = dmaRX_TX_BUFFER_SIZE;

    /*
     * We allocate 65536 bytes for RX BDs which can accommodate a
     * maximum of 8192 BDs which is much more than any application
     * will ever need.
     */
    xemacpsif->rxSegments = ( struct xBD_TYPE * ) ( pucGetUncachedMemory( xRxSize ) );
    xemacpsif->txSegments = ( struct xBD_TYPE * ) ( pucGetUncachedMemory( xTxSize ) );
    xemacpsif->tx_space = ( unsigned char * ) ( pucGetUncachedMemory( ipconfigNIC_N_TX_DESC * xemacpsif->uTxUnitSize ) );
...

Hopefully this is the place where it’s supposed to be used.

Although a good advice, I think that this is unlikely to happen in my case.
The wireshark trace does not show any unrelated networktraffic directed to the IP address of the Zynq. Moreover, the IP address that I use for testing resides in a different subnet than any other device in the network. Therefore I doubt that anybody would be spamming the Zynq. I’ll see if I can get hold to a hub to rule out this possibility for sure.

9

Great, glad to be of help! Just to make sure: I am not talking about dedicated packets but broadcast “storms” which will affect your DUT likewise as your ethernet driver must forward those to your stack for procession.

10

Ah okay, now I know what you mean. In wireshark I do see some broadcasts but these are mostly ARPs once every 1 to 20 seconds and some UDP connection whos origins I don’t really understand, approx. every 5 seconds.

By the way, I also tried with the firewall disabled, just to be sure, but it does not change anything.

11

@Stonebull, some good news: I compiled your program (guessing what the structs look like, and started it. I saw:

************ UPORT: "1st Uport Perf" READY FOR CONNECTION *************
************ UPORT: "2nd Uport Perf" READY FOR CONNECTION *************

I used Hercules as a client . I started it 2 times and opened a TCP client to port 10000 and 20000. It worked without any problem!

Could you test your DUT with Hercules? Unfortunately it only runs on Windows.

This is a PCAP that I made: pcap_with_hercules.zip (10.5 KB)

1 Like
12

@htibosch Amazing! If I knew that you wanted to really run it, I would have provided more files.
Thanks A LOT for investing so much time in this, I really appreciate that.

The message that appears on your screen is exactly what it should show when executing correctly.

I tried to run the test with Herkules as you suggested but unfortunately the error seems to be persisiting. It feels like the problem occurs later when using Herkules, but it still happens. Let me show you in this .gif:
I suggest viewing it in a new tab, for a better resolution
HerkulesScreencapt2

At around 136 sec it starts to fail. At the top left you can see the console output in sync to the wireshark capturing.

Can I ask you to run it a bit longer on your machine, to see if it also happens at one point? May I ask on which device you are running it? Also a Zynq?

1 Like
13

You wrote:

Can I ask you to run it a bit longer on your machine, to see if it also happens at one point?

Sure. Right now after 1.5 hour it is still running well.

May I ask on which device you are running it? Also a Zynq?

I took the board that is currently connected to USB, which was an STM32F4. Now I’m running it on my Zynq/Zybo and I get the same positive results.

Nice GIF recording!

Looking at your GIF, it looks like something happens after a while. All of a sudden “something” starts eating network buffers at a high rate.

Mind you that each TCP socket has its own stream buffer for both reception and transmission. I chose to do so that because it gives a predictable behaviour. Only when data is passed to the EMAC/DMA, it is enveloped in a network buffer.

What else happens in your CPU? Are there other processes? Can you stop them as a test?

Maybe you can do some debugging and find out who is claiming all these network buffers?

Here is a long PCAP when running with a Zynq 7000 : running_on_zynq_hein.zip (367.3 KB)

1 Like
14

Hi @Stonebull

  • I had one query regarding the retransmission. Are we giving enough time for the buffer to be emptied and receiving the ACK back for the TCP connections?

  • Can you please also share the observations after increasing the number of buffers, is that helping here?

15

This is fairly straightforward to deduct from the pcap file by analyzing the advertised window size in the connection stream… again, on the computer I follow this thread, I can not look at the pcap file, so it is all theory on my side.

@Stonebull : Can you run your application under tracealyzer supervision? That would give you a fairly good hint as to where the problem might lurk.

16

Sorry for the long delay, I had to do some more testing and thinking which led to me discovering that there is indeed a switch inbetween the computer and the Zynq within my setup :man_facepalming:

For context, I am testing on a custom carrier that amongst other hardware has a tiny switch routed in between the PC and the Zynq. I completely forgot about that before.
So next I tried testing the program on a Zynq, on a different carrier board that now has no switch in there for sure, and it looks promising.

Then I tried again with a different switch hooked up in between and the errors were back.

@htibosch Could I ask you to test the program again with a switch in between, just for reference? Does it create problems with a switch in between for you as well?

Thanks!

I don’t know if a switch explains why all of a sudden the buffers get eaten away, though.

Is this adjustable or are you telling me that this is the way the IP stack is implemented? If it’s configurable, please tell me how I can do that.

Actually, that was one of the reasons for me to write a small test application (see main.c) with just these two sockets and one application task, in order to avoid any side effects from other parts of the application.

I’ll try to investigate that.

I’d love to, but unfortunately I have tried trace analyzer, and it does not work well with the Zynq, as the Plugin cannot be installed in the Xilinx SDK which leads to further problems.
I planned to try Segger’s SystemView software soon, but unfortunately Segger does not provide a Zynq implementation for their port so I’ll have to do that on my own and that will probably take some time to do.

@moninom1
When I decrease the rate to 500 ms I don’t get any errors. But I guess that 2 packets per 100 ms should be possible with the IP stack.

Unfortunately increasing the number of buffers did not change much, they just run out equally.

@htibosch, @RAc, @moninom1
Thank you again all for the effort you put into this.

1 Like
17

Youwrote:

on a different carrier board that now has no switch in there for sure , and it looks promising

My laptop is always connected to a 10/100/1000 Mbps switch, and all MCUs are connected to the same switch. The switch is also connected to an IPv4 and an IPv6 router.

In the old days I did work with hubs, but that was a nightmare.

Could I ask you to test the program again with a switch in between

It makes me wonder if you switch was capable of doing 1 Gbps?

I don’t know if a switch explains why all of a sudden the buffers get eaten away, though.

Just a wild guess: transmission takes a while and fails after a timeout. Time driver “forgets” to release the transmission buffer because the operation ends differently.

If it’s (the stream buffers) are configurable, please tell me how I can do that.

I found a good example in your code:

// Set sliding window - 1 = 1x MSS (Maximum Segment Size), 2 = 2x MSS, ...
// MSS is defined by ipconfigTCP_MSS - if there is no definition in FreeRTOSIpconfig.h, then default value of 1460 is used
WinProperties_t winProps =
{
    .lRxBufSize = 4 * ipconfigTCP_MSS,
    .lRxWinSize = 2,
    .lTxBufSize = 4 * ipconfigTCP_MSS,
    .lTxWinSize = 2
};

retVal_os = FreeRTOS_setsockopt( pCtrl->socketListener,
                                 0,                        //currently unused parameter
                                 FREERTOS_SO_WIN_PROPERTIES ,
                                 &winProps,
                                 sizeof( winProps ) );

For iperf testing, I put the xWinSize to up to 20 x MSS, and xBufSize to 30 x MSS.

If I may summarise all observations:

  • There should be more than enough network buffers:
#define ipconfigNUM_NETWORK_BUFFER_DESCRIPTORS      96
  • Sending 2 packets every 100 ms should be a very easy for a Zynq.
  • During the first half minute or so, it works all well. And then, all of a sudden, some task or driver starts allocating network buffers, without returning them.
  • One connection works perfectly, the problems start when a second connection is made.
  • I ran the same code on my Zybo/Zynq board and there was no problem, not even after an hour. There was not a single resend.

I am very curious from where the network buffers are allocated, and why they’re not returned to the pool.

EDIT
The WIN and buffer sizes set with SO_WIN_PROPERTIES only matter if you are sending lots of (continuous) data.
In your case a big WIN size is only decreasing performance because there is a big “loss of packets”.

1 Like
18

I gradually stepped through the IP stack with the debugger and noticed that the destination IP addresses of the packages that are received while the network buffers are drained, are all broadcast addresses that belong to ARP.

This led me to do some digging in the network adapter configuration and disabling the properties Link Layer Topology Discovery & Reply for Topology Discovery

EthernetadapterProperties
EthernetadapterProperties453×585 83.2 KB

Now the only enabled properties are TCP/IPv4 and NPCAP driver.

Since I turned off the Topology Discovery options, I do not immediately run out of network buffers anymore (just around 25 lowest 23). Still I occasionally get retransmissions in wireshark. But the data flow is seemingly uninterrupted.

@htibosch Out of interest, which version of FreeRTOs & TCP/IP stack are you using? Are you using the same way to open a socket as I do (with the UPORT)?

I tried with a switch that is capable of 1000 Mbps and one with only 100 Mbps and both work as described above. Would you think that 100 Mbps are not enough? Could this whole problem be related to switches beeing too slow?

ARP_UDP_packets
ARP_UDP_packets1642×1390 142 KB

Looking at the wireshark window with no connected TCP clients I noticed that there are a lot of repeated ARP requests of the PC asking what the Zynqs MAC is. And although the Zynq gives a reply, the PC keeps asking. Is this normal?

I also get random UDP packages sent from the PC to every subnet-ip with a length of 8, containing a string saying “TCF2”. I have no clue where that comes from, has anybody any ideas what this could be?

May I ask you, what is the window size that you are using in your successfull tests?

A whole other thing came to my mind. Could a possible delay in processing packets be related to having compiled FreeRTOS with optimization level O0? I tried again with O2 but it looks like nothing changed…

19

IoT2050 firing broadcast to port 1534 every 15sec - 250100 - Industry Support Siemens

The udp packets appear to come from Eclipse.

1 Like
20

I wrote:

I am very curious from where the network buffers are allocated, and why they’re not returned to the pool.

I gradually stepped through the IP stack with the debugger and noticed that the destination IP addresses of the packages that are received while the network buffers are drained, are all broadcast addresses that belong to ARP.

@RAc already suggested:

so here is a shot in the dark: Is it possible that the input side of your device is being kept busy serving rapid broadcast packets

… and I would also recommend to make ipconfigARP_CACHE_ENTRIES much higher than it is now. Although I don’t know if the broadcast ARP packets will litter your ARP cache table. That depends on the +TCP versions that you are using.

Since I turned off the Topology Discovery options, I do not immediately run out of network buffers anymore

That is great but I am still curious about the real cause.

@htibosch Out of interest, which version of FreeRTOs & TCP/IP stack are you using? Are you using the same way to open a socket as I do (with the UPORT)?

For testing I use the library from main and from dev/ipv6_integration. For my private FreeRTOS projects, I use one of the LTS releases.

Would you think that 100 Mbps are not enough? Could this whole problem be related to switches beeing too slow?

Normally it should work as well. I was just wondering if there is a particular problem with the Xilinx driver connecting at 100 Mbps.

I wrote:

In your case a big WIN size is only decreasing performance because there is a big “loss of packets”.

Your wrote:

May I ask you, what is the window size that you are using in your successfull tests?

Any configuration should do as long as there is not so much packet loss.
When measuring TCP performance, I tend to make a WIN size of e.g. 20 x MSS, which is almost 30 KB.

A whole other thing came to my mind. Could a possible delay in processing packets be related to having compiled FreeRTOS with optimization level O0? I tried again with O2 but it looks like nothing changed…

The optimisation level may increase the performance with up to 50%, but -O0 should not lead to packet loss.

1 Like