The ARP flux problem
Address Resolution Protocol (ARP) is used to locate the Ethernet address associated with a desired IP address. It glues together the IP and Ethernet networking layers.
When a machine sends a packet bound to another IP on a locally connected Ethernet network, a broadcast Ethernet frame containing an ARP request is sent on the Ethernet. All machine with the same Ethernet broadcast address will receive this frame. When a machine hosting the requested IP address receives the ARP packet, it responds with the link level address on which it will receive packets for that IP address. It is in this response that the ARP flux problem occurs, specifically when the machine is connected to this network segment (subnet) with multiple network cards. The machine may reply to the ARP request from multiple Ethernet interfaces.
The Linux operating system defaults to a weak host model that provides better network connectivity by accepting any locally destined packet regardless of the network interface on which the packet was received. As a result of this design, when an ARP request for an IP arrives on the machine and there are two interfaces on the same network segment, an ARP response could be sent from both the interfaces, even when the requested IP is available from one of the Ethernet interfaces. On the machine that initiated the ARP request, these responses cause a confusion and messes up with the ARP cache (leads to nondeterministic population of the cache). This is the ARP Flux problem. Note that problem only affects the hosts that have multiple physical connections to the same medium or broadcast domain.
For Example:
[root@csucsm301vm1]# arping -I ens256 -c 3 192.168.82.2
ARPING 192.168.82.2 from 192.168.82.1 ens256
Unicast reply from 192.168.82.2 [00:50:56:9C:14:D9] 0.833ms
Unicast reply from 192.168.82.2 [00:50:56:9C:88:CE] 0.702ms
Unicast reply from 192.168.82.2 [00:50:56:9C:88:CE] 0.715ms
Sent 3 probes (1 broadcast(s))
Received 3 response(s)Avoiding ARP flux
One of the ways to fix this problem involves enabling arp_filter tune-able, which forces the recipient to perform a route lookup to determine the interface through which it should send the ARP response, instead of sending it through all interfaces that received the request. This solution however, might have some unintended effects if the only route to the destination is through one of the network cards.
Alternatively arp_ignore and arp_announce sysctl config parameters can also be used to tune this behavior.
