+TCP: Network buffers getting used up and not getting released

Hello,

I am using FreeRTOS+TCP for my project and everything seems to be working mostly fine. I say mostly because I have noticed that even when my system is idle and there is no active TCP communication going on, at random moments, I notice on the serial console that network buffers are getting used up. This happens over a longer period of time. See image below:

I don’t understand why this is happening. How can I determine what is causing these buffers to get used? And why are they not getting released?

Any help would be appreciated. I will do my best to answer any questions.

Thank you,
Sid

Can you confirm there is no traffic on the network by using Wireshark. It would be very unusual for a network to be silent.

Can you put a breakpoint in the code where the network buffer is obtained to see what the stack is doing at that time, and then follow through to see where the buffer goes?

If you leave the code like that for an extended period, do you eventually run out of buffers, or do they come back eventually (maybe after some timeout).

Also, which port are you using? Is it one in our github repo or some you created yourself?

Looking at your 500K heap: I guess that you are still working on a Xilinx Zynq?

The FreeRTOS+TCP library now has its own URL: https://github.com/FreeRTOS/FreeRTOS-Plus-TCP
There you will also find the latest network drivers.

Beside the checks suggested by Richard, I would think of UDP sockets: when you forget to call recvfrom(), network buffers get stored in the socket and won’t be freed.

The preferred way of using the Zynq driver is with these 4 macros defined:

#define ipconfigZERO_COPY_RX_DRIVER               1
#define ipconfigZERO_COPY_TX_DRIVER               1

#define ipconfigDRIVER_INCLUDED_RX_IP_CHECKSUM    1
#define ipconfigDRIVER_INCLUDED_TX_IP_CHECKSUM    1

Can you check what packets are being received by running Wireshark?
How often do you see the message about the network buffers? Every 20 seconds?
Or rather irregularly?

Time ago I were surprised by unexpected buffer allocations caused by a number of ARP requests in a relatively short period of time sent by a connected host PC.
Found with the strongly recommended Wireshark, solved by FreeRTOSIPConfig.h and heap tuning. Just my 2ct

Hello Richard, Hein,

It is indeed the case that there is traffic on the network. Attached image shows the screenshot from Wireshark:

I will do the breakpoint process you have described a bit later today when I have some more time. To answer your other questions, I have not reached the point where I have run out of buffers since it happens very slowly and irregularly. Sometimes it doesn’t happen at all. I can test this by reducing the max buffer count and then see what happens.

I was using version 2.2.1 of the library but just now, on Hein’s recommendation, have updated to the latest release from Github.

Hein, I have had those 4 macros defined since the beginning of my project. I see the network buffer messages rather irregularly.

Thank you,
Sid

Hello Sid,

These ARP messages correspond to CRC error: 1234 location 3
messages being printed in the serial terminal

Sorry, I forgot to mention that you will also need this change from PR #19. The PR should have been merged much earlier.

About the:

    Network buffers: 45 lowest 45
    Network buffers: 44 lowest 44
    Network buffers: 43 lowest 43
    Network buffers: 42 lowest 42
    Network buffers: 41 lowest 41
    Network buffers: 40 lowest 40

If they come irregularly, and if the messages stop at some point, then there is nothing wrong. It only means that the minimum number of network buffers is dropping, down to a certain point.

These messages are produced by a function vPrintResourceStats() from FreeRTOS_IP.c. It checks some essential resources like network buffers and the available amount of heap. It can be helpful in case there are not enough resources and communication stops.

Hello Hein,

Sorry, I forgot to mention that you will also need this change from PR #19. The PR should have been merged much earlier.

No problem about that, I found the PR and just made the changes manually to my project.

If they come irregularly, and if the messages stop at some point, then there is nothing wrong. It only means that the minimum number of network buffers is dropping, down to a certain point.

My concern here is that I have not really run my project for too long and I saw the number of buffers going down over time. The final product is expected to run for weeks at a time. If the buffers keep going down, it will run out of resources at some point. As you see in my Wireshark capture, there are some UDP packets going out from my laptop. Do you think it is these UDP packets that are causing the buffers to get used up? And if so, I would have expected these buffers to get released too. So I would have expected the lowest: XX number to stay low but the available number of buffers to go back up. I am not seeing that somehow.

I am trying to step through the code to see why the buffer count is not going back up… No luck so far.

Thanks,
Sid

I forgot to mention that you will also need this change from
PR #19. The PR should have been merged much earlier.

The PR has been merged now, and Zynq/NetworkInterface.c is now updated.