FreeRTOS+TCP zero copy driver and statically allocated buffers

Kernel
1

Hi all,

Can someone help me understand how I would go about handling buffer descriptors when using statically allocated buffers in a zero copy driver?

Although the documentation suggests that pucGetNetworkBuffer() would normally be used with zero copy drivers, this is only available in BufferAllocation_2.c and uses malloc to allocate space for the buffer.

Since FreeRTOS only seems to have a single buffer pool for TX and RX frames, my thoughts are to initialise the RX descriptor ring of my ethernet chip (AMD LANCE Am79C90) with half of those buffers, and point each buffer descriptor at one of the statically allocated buffers. The other half is then available for TX frames.

What Im trying to figure out at the moment is how to acquire buffers for this. I could call pxGetNetworkBufferWithDescriptor() and make a pointer to the pucEthernetBuffer member of the returned descriptor and put that into my RX ring entries. It looks like as part of vNetworkInterfaceAllocateRAMToBuffers() a pointer to the descriptor entry is stored just before the memory allocated to the buffer itself, so would this be a legitimate way to keep track of used descriptors? I can subtract ipBUFFER_PADDING from the buffer address to be able to recover a pointer to the descriptor which could then be used to return that descriptor to the free list after the received frame has been processed?

If those descriptors dont need to be returned to the free list, then I can just reconfigure the RX ring entry to say it is available to the ethernet chip again and move on?

Hope that all makes sense.

Thanks :slight_smile:

2

The buffer allocation schemes are deliberately kept in the portable layer as different people need different things. You can then implement anything you want or need. This is as per the heap memory management in the kernel itself - there is no single scheme for all. For the buffers you can pre-allocate everything then have an equivalent of pucGetNetworkBuffer() that grabs buffers from the preallocated pool.

Since FreeRTOS only seems to have a single buffer pool for TX and RX frames, my thoughts are to initialise the RX descriptor ring of my ethernet chip (AMD LANCE Am79C90) with half of those buffers, and point each buffer descriptor at one of the statically allocated buffers. The other half is then available for TX frames.

I’m not sure which device you are using - but generally on microcontrollers that are connected to fast networks I find I need very few Tx buffers. By the time one is passed to the hardware it is normally free again, sometimes before you even unwind the network stack. That is more true where there is a DMA controlling Tx. Its worth you doing a bit of a characterisation exercise to see how many you need.

What Im trying to figure out at the moment is how to acquire buffers for this. I could call pxGetNetworkBufferWithDescriptor() and make a pointer to the pucEthernetBuffer member of the returned descriptor and put that into my RX ring entries. It looks like as part of vNetworkInterfaceAllocateRAMToBuffers() a pointer to the descriptor entry is stored just before the memory allocated to the buffer itself, so would this be a legitimate way to keep track of used descriptors? I can subtract ipBUFFER_PADDING from the buffer address to be able to recover a pointer to the descriptor which could then be used to return that descriptor to the free list after the received frame has been processed?

Sounds like it. Note vNetworkInterfaceAllocateRAMToBuffers() is another function you can implement in any way that suites. It is part of the network driver rathe than part of the core code because different network hardware have different requirements as to the buffer alignment and memory they can come from (sometimes a DMA doesn’t have access to the whole memory space).

3

I had prepared an answer and forgot to post it. That doesn’t help you. @rtel, thanks for answering it.

In addition I would recommend to have a close look at one of the existing drivers, all zero-copy: DriverSAM, STM32Fxx, or Zynq.

Note that BufferAllocation_1.c is used on systems with enough RAM. It is more efficient to have the actual buffers pre-allocated. Also, you can control where the buffers are located (in vNetworkInterfaceAllocateRAMToBuffers()). Sometimes DMA wants the buffers to be located in specific RAM areas.

Yes, pxGetNetworkBufferWithDescriptor() is called to fill the DMA ring buffers. You can use prvPacketBuffer_to_NetworkBuffer() to find a network buffer when only pucEthernetBuffer is known.
Here is an example:

NetworkBufferDescriptor_t * pxBuffer1 = pxGetNetworkBufferWithDescriptor( uxLength, uxBlockTimeTicks );
NetworkBufferDescriptor_t * pxBuffer2 = prvPacketBuffer_to_NetworkBuffer( pxBuffer1->pucEthernetBuffer );

In this fragment, pxBuffer1 will get the same value as pxBuffer2.

4

Thanks @rtel and @htibosch. I feel like I am on the right path then.

The platform is an old Cisco 2500 router which has several MB of RAM on a SIMM, so I am pre-allocating all of the buffers, but just need to find the right technique to juggle them. :slight_smile:

Ive been looking through the existing drivers and using them for inspiration where possible. Once I get it working I’ll look at playing around with buffers and maybe I can reduce the number of TX buffers (although Im not really concerned about memory usage at this point). The LANCE can have asymmetric TX/RX ring sizes, so that shouldnt be an issue.

I’ll do some more reading into prvPacketBuffer_to_NetworkBuffer() too, hadnt come across that yet! Sounds like it will take care of reversing a buffer into a descriptor nicely.

5

Great. Feel free to post some driver code here in case you are in doubt.

1 Like
6

Hello Hein.

What is the first thing that FreeRTOS+TCP does once it is able to transmit packets? I havent yet enabled DHCP, so is it ARPing for the default gateway?

I can see my ethernet chip transmitting packets, and Im capturing them with Wireshark, but something is not right.

I can see dest MAC of ff:ff:ff:ff:ff:ff and the correct source MAC at the start of the ethernet frame, but it looks like things go wrong after that, and Wireshark is unable to determine what protocol is contained within the frame from there on.

Not sure if you have any pointers where I can look to see what is going wrong?

The “cd ef” at address 0x12 seems to be some remnance of the IP address that I have configured statically during initialisation (0xabcdefxx).

Some debug information that I am printing out suggests the the buffer does hold 306 bytes to be transmitted…

pxDescriptor 0x00140d70
pucEthernetBuffer 0x006060aa
xDataLength 306
TD> (0x0060c1c0): 60AA 2360 FECE 0000 
FF FF FF FF FF FF 00 D0 58 AD 1F BA 01 FF FF FF 01 24 CD EF 00 00 00 10 08 3C 00 00 00 00 FF FF FF FF 00 44 00 43 01 10 00 00 01 01 06 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

(just the first 64 bytes of the frame dumped, seems to match what Wireshark is seeing)

Thanks!

Tom

Screen Shot 2021-10-31 at 18.28.03
Screen Shot 2021-10-31 at 18.28.033104Ă—1970 708 KB

7

Can you please attached a zip of the pcap file.

8

@rtel wrote:

Can you please attach a zip of the pcap file.

That is indeed much handier. If you zip it, you can attach it to your post, using this symbol: image

@tomstorey wrote:

What is the first thing that FreeRTOS+TCP does once it is able to transmit packets? I havent yet enabled DHCP, so is it ARPing for the default gateway?

When you have not enabled DHCP, the application will communicate using it’s static IP-address. This address is the one passed to FreeRTOS_IPInit().

I can see my ethernet chip transmitting packets, and Im capturing them with Wireshark, but something is not right.

What you see is a broadcast, but the frame type is 0x01ff, which the library never uses. One would expect one of these:

#define ipARP_FRAME_TYPE                ( 0x0806 )
#define ipIPv4_FRAME_TYPE               ( 0x0800 )
#define ipIPv6_FRAME_TYPE               ( 0xDD86 )

Not sure if you have any pointers where I can look to see what is going wrong?

I think that we are looking at a corrupted gratuitous ARP request, although it is much too long.

The “cd ef” at address 0x12 seems to be some remnance of the IP address that I have configured statically during initialisation (0xabcdefxx).

Sounds like an unusual IP-address. What is the network address of your LAN? Or the IP-address of your PC? On a LAN, the address is often 192.168.1.x or 10.x.x.x.

Have you told already what platform you are using, and what network driver?

Would you mind to attach your copies of FreeRTOSConfig.h and FreeRTOSIPConfig.h?

9

Sure, have attached a PCAP file.

The platform is a Cisco 2500 router - its something that Im “hacking about” with because it has a Motorola 68030 CPU, hence needing to write my own drivers etc from scratch.

The IP address was just something I threw in to make it stand out. I had previously seen a series of bytes 0xC0 and 0xA8 which looked like the start of “192.168.x.x” address, so I changed it to make it stand out. At this stage I am not too concerned with having a working IP, would just like to see it sending some packets that can be decoded.

Ive uploaded my code. I dont expect you to read through it all of course, but its all there for reference. The config headers are included in the root: GitHub - tomstorey/c2500_freertos_tcp: Work-in-progress to get FreeRTOS+TCP running on a Cisco 2501 router

There is a link to my FreeRTOS+TCP fork, my port is currently living in the “lance” branch.

FWIW I have a background in network engineering (my profession), and currently I just have the device plugged into an ethernet adapter hanging off my laptop (easier to capture everything that it sends that way). Hence my blasé attitude towards having FreeRTOS on a functional network with a working IP right now. And you can use all of the lingo if you need to. :smiley:

13745.packets.pcapng.zip (754 Bytes)

10

Well, I dont really know how to explain it, but deciding to have a little bit of an exploration myself tonight, I have recompiled and all of a sudden it would appear to be “working”…

Heres a dump of a packet that is being sent, looks a lot more like an in-tact ARP packet to me!

FF FF FF FF FF FF 00 D0 58 AD 1F BA 08 06 00 01 08 00 06 04 00 01 00 D0 58 AD 1F BA AB CD EF C8 00 00 00 00 00 00 AB CD EF C8

Its one of those “I swear I have not changed anything” kinds of moments. The only thing that has been done since it was failing before is I committed changes to a git repo. :shrug:

But clearly something must have changed for such a dramatic improvement, I just wouldnt be able to say what to save my life!

And I dont know if maybe it is something to do with my Mac (I have given it a restart), but I am not able to capture these good looking packets in Wireshark. Perhaps the Mac network stack is consuming them in some way that prevents them from being captured by Wireshark. I dont really understand OS network internals enough to be able to say why.

Anyhow, a lot closer to having something working now!

11

Ok, more digging, and I figured out why I wasnt getting the ARP packets. :slight_smile:

Does FreeRTOS+TCP assume that ethernet frames will be padded out to 64 bytes, is this perhaps something that should be implemented in the driver if not handled by the hardware itself?

I dont receive the ARP packets from my device unless I add 18 bytes of length to the data that is to be transmitted in order to bring it up to 60 bytes (from the 38 that FreeRTOS generates in vARPGenerateRequestPacket). My knowledge is a little rusty, but from what I was reading, the remaining 4 bytes comes from the CRC.

Presumably the frames are dropped as runts if they are sent out on the wire as 38 bytes?

12

Hello @tomstorey,

The function vARPGenerateRequestPacket generates 42 bytes of data (which is sufficient for an ARP packet) if you look at it closely. It copies the 38 bytes from the default values, but fills in the target IP address itself. See here.

Also, I am not sure that +TCP enforces some limit of 60 bytes. Am I wrong I wonder? :thinking:

Regards,
Aniruddha

13

Hi Aniruddha.

No, its an ethernet thing. Frames must be a minimum of 64 bytes on the wire (I guess 60 bytes of data + 4 bytes of CRC).

I have added a simple loop in my driver to zero out any remaining bytes up to 60 and adjust the length of the data to be transmitted accordingly because it seems my ethernet chip doesnt automatically do this - from a bit of reading around it seems that some ethernet chips may automatically pad smaller frames up to the correct length.

All is good now. :slight_smile:

14

Ah! I misunderstood! Thanks for the clarification Tom :slight_smile:

Thanks for the interesting conversation above in this thread. :slightly_smiling_face:

15

@tomstorey wrote:

I have added a simple loop in my driver to zero out any remaining bytes up to 60 and adjust the length of the data to be transmitted accordingly because it seems my ethernet chip doesnt automatically do this

When using BufferAllocation_2.c: when you adjust the length of a packet, make sure that it has allocated enough space. If it used pvPortMalloc() to allocate 42 bytes, it is not safe to fill it up to 60 bytes.

Does FreeRTOS+TCP assume that ethernet frames will be padded out to 64 bytes, is this perhaps something that should be implemented in the driver if not handled by the hardware itself?

That is why there is ipconfigETHERNET_MINIMUM_PACKET_BYTES, when defined as 60, the libraries will make sure that all packets will have a minimum length of 60 bytes (ex 4-byte Ethernet checksum)

By default, ipconfigETHERNET_MINIMUM_PACKET_BYTES is not defined.

It is possible to send 40-byte packets, but then a collision might not be detected.

16

Hi Hein,

Ah! I was looking for something like ipconfigETHERNET_MINIMUM_PACKET_BYTES, but couldnt see it while skimming through the list of config options last night. I will look into this futher and define it appropriately in my config. Thanks!

Im using BufferAllocation_1.c because in this device, the ethernet chip only has access to the upper 2MB (max) of RAM, so using an out-of-the-box malloc probably wont be able to allocate buffers in the right location.

Im considering as an exercise to enable dynamic allocation of memory, but I think I’ll need to create another “malloc” that works specifically in the “IO memory” region, once I determine where it is (it will be at different locations depending on on how much RAM is installed).

Tom

17

The IP-stack uses two defines for allocating dynamic memory:

pvPortMallocSocket() to allocate the space for a socket.
pvPortMallocLarge() to allocate TCP stream buffers.

We could add a pvPortMallocBuffer() to allocate the pucEthernetBuffer of a packet in BufferAllocation_2.c, and let it default to pvPortMalloc().

EDIT:
Beside using macro’s to allocate memory, it also uses macro’s to free the space: vPortFreeSocket() and vPortFreeLarge().

1 Like
18

That sounds like an idea! I just hope the effort is worth it to more than just me. :slight_smile:

19

One thing to note, the minimum packet size is a function of the protocol, especially the speed, of the ethernet link. Faster speeds tend to need longer packets as they need to be big enough to detect the problems on the wire.

20

Hi all.

Ive got RX packet processing going, although tracking down a little bug that seems to be causing a crash (I think Ive narrowed it down to a yield after sending to a queue and a higher priority task was woken, seems to work nearly flawlessly when I dont yield for the higher priority task, save for some missing error handling on the ethernet controller that causes other issues…)

But anyway, I am able to send ping requests and FreeRTOS is sending back replys, but the checksum is blank (0x0000), thus all pings are considered timed out.

Sorry if this has a really obvious answer, but is there something I am supposed to implement, or an option I am supposed to set to do checksums?

Thanks