FreeRTOS+TCP not updating ARP table with gratuitous ARP request with a changed MAC address

Libraries
1

We have a so called ‘Cluster’ with Windows 2022 servers.
The cluster contains two servers, let call them server-1 and server-2.
To complete the introduction, there is also a witness, but that’s not important for my issue.

The cluster is set up for redundancy. If a server fails (e.g. powerloss) the other server becomes active so the system continues running. Let’s call the server that fails the ‘old server’ and the server that takes over the ‘new server’.

When testing and shutting down a server we saw the embedded module, running on FreeRTOS with FreeRTOS+TCP wasn’t connecting with the new server but still was sending packets to the old server.

Technically, when migrating from the old server to the new server also the server IP address migrates from the old server to the new server.

In Wireshark it is also seen that the server IP address 10.51.1.1 migrates to MAC address ac:1f:6b:ef:af:45 while the old server had another MAC address (ac:1f:6b:ef:ab:01).

1055 781.388672 SuperMic_ef:af:45 Broadcast ARP 60 Gratuitous ARP for 10.51.1.1 (Request) (duplicate use of 10.51.1.1 detected!) 2022-03-16 10:51:45.415845

Frame 1055: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) on interface 0
Ethernet II, Src: SuperMic_ef:af:45 (ac:1f:6b:ef:af:45), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
[Duplicate IP address detected for 10.51.1.1 (ac:1f:6b:ef:af:45) - also in use by ac:1f:6b:ef:ab:01 (frame 897)]
Address Resolution Protocol (request/gratuitous ARP)
    Hardware type: Ethernet (1)
    Protocol type: IPv4 (0x0800)
    Hardware size: 6
    Protocol size: 4
    Opcode: request (1)
    [Is gratuitous: True]
    Sender MAC address: SuperMic_ef:af:45 (ac:1f:6b:ef:af:45)
    Sender IP address: 10.51.1.1
    Target MAC address: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Target IP address: 10.51.1.1

The problem is, FreeRTOS+TCP doesn’t update the ARP table with this gratuitous ARP request in the module and therefore doesn’t send TCP/IP packets to the correct new MAC address after the migration from old to new server.

We use FreeRTOS+TCP V2.3.2 LTS Patch 1
Source FreeRTOS_ARP.c
Function eARPProcessPacket()

            case ipARP_REQUEST:

                /* The packet contained an ARP request.  Was it for the IP
                 * address of the node running this code? */
                if( ulTargetProtocolAddress == *ipLOCAL_IP_ADDRESS_POINTER )

This above check is dropping this gratuitous ARP request because the Target IP address isn’t the local IP address.

I have searched on the internet and I think FreeRTOS+TCP should handle this gratuitous ARP request so an update is done with existing ARP entry’s.

I changed it to the following code:

            case ipARP_REQUEST:

                /* The packet contained an ARP request.  Was it for the IP
                 * address of the node running this code? */                
                if( ulTargetProtocolAddress == *ipLOCAL_IP_ADDRESS_POINTER )
                {
                  // Existing code of FreeRTOS+TCP
                }
                else if ( ulSenderProtocolAddress == ulTargetProtocolAddress )  // Gratuitous ARP request?
                {
                  /* The request is a Gratuitous ARP message.
                   * Refresh the entry if it already exists. */
                  /* Determine the ARP cache status for the requested IP address. */
                  if ( eARPGetCacheEntry( &( ulSenderProtocolAddress ), &( xHardwareAddress ) ) == eARPCacheHit )
                  {
                    vARPRefreshCacheEntry( &( pxARPHeader->xSenderHardwareAddress ), ulSenderProtocolAddress );
                  }
                }

Hereafter the switch from old to new server was done perfectly after receiving and handling the gratuitous ARP request.

What do you think? Is FreeRTOS+TCP lacking support of this feature or is Windows not configured correctly?
After reading some info about gratuitous ARP request’s my personal opinion is that FreeRTOS+TCP should be updated with this code.

2

Thank you for reporting and the provided detail. At this time I agree with your conclusion, but would like to do a little research to see if there are unintended consequences (like one machine being able to redirect another’s traffic). Most likely we will open a bug report.

3

Yes, I agree with you, thank you @nlangenb for sharing these findings.

Do you want to produce a pull-request for this? Or do you want me to do so?

As always, some background about the subject:

I just looked up older versions of FreeRTOS_ARP.c, and found that incoming gratuitous ARP requests have never led to updating the ARP table. The requests were only used to detect IP-conflicts.

Good to see that your new code will check if the IP-address is already present in the ARP cache table:

+ if ( eARPGetCacheEntry(..) == eARPCacheHit )
+ {
+     vARPRefreshCacheEntry(..);
+ }

History:

In the past, when I was testing from a Windows host which had many interfaces, the following could happen:

  • The DUT is found and communication starts.
  • I stop the embedded application for a while.
  • The host sees timeouts, and it start looking.
    for the DUT on a different interface, sometimes
    through a USB modem/gateway.
  • I resume operation of the DUT.
  • The host never finds the DUT anymore, even
    after restarting the application ( e.g. a web browser )

That was frustrating.

The above scenario got solved by letting the DUT send gratuitous ARP requests. The host sees them and concludes that the DUT can be found on the LAN. Right after sending an ARP request, the two will reconnect.

Now that I only have a LAN, on which there is a router/gateway, I don’t see this happening anymore.

Frequency of gratuitous ARP requests

The frequency of sending gratuitous ARP requests is quite high: a request will be sent every 20 seconds:

    /** @brief The time between gratuitous ARPs. */
    #ifndef arpGRATUITOUS_ARP_PERIOD
        #define arpGRATUITOUS_ARP_PERIOD    ( pdMS_TO_TICKS( 20000U ) )
    #endif

You can change the value of arpGRATUITOUS_ARP_PERIOD in your copy of FreeRTOSIPConfig.h

Alternatives:

In cases where the peer host doesn’t send gratuitous ARP requests, or not frequently enough, there are two more ways to update your cache table:

You can send out an ARP request for the peer device:

/**
 * @brief Create and send an ARP request packet.
 *
 * @param[in] ulIPAddress: A 32-bit representation of the IP-address whose
 *                         physical (MAC) address is required.
 */
void FreeRTOS_OutputARPRequest( uint32_t ulIPAddress );

Or change the configuration: entries in the ARP cache have a limited lifetime. The lifetime is the product of these two defines:

#define ipARP_TIMER_PERIOD_MS 10000
#define ipconfigMAX_ARP_AGE 150 /* Should be at least 3 */

So 10000 x 150 = 1,500,000 ms (1,500 seconds), which is a very long time.

You can shorten the life time of ARP cache entries. Just before the entry expires up to 3 ARP requests will be done.

4

I am not familiar with Github, so it would be the best if you handle this one.

5

Hi @htibosch
Do we have PR for this? I tried to look but could not find one.

6

Sorry that I’m responding late, but today I created PR #463 to solve this.
Thank you very much @nlangenb for your corporation.