ARP timeout on Mellanox IB Gateway SX6036G

Hi,

I have set up an installation of Openstack Juno on 8 nodes: one cloud controller and 7 Nova compute nodes. The nodes are all equipped with Mellanox ConnectX-3 Infiniband HCA. Openstack networking (Neutron) is configured to work over the Infiniband interface using the eth_ipoib driver for ethernet para-virtualization. The ethernet interface created by eth_ipoib is used by Openstack networking services over 2 UFM (version 4.8.0) partitions:

  • one pkey is used for the private network between cloud nodes. It is defined as a local network and it is only connected to the Logical Server Group of the cloud nodes
  • the other pkey is used for the Openstack/Neutron external network. It is defined as a local network and it is connected to the Logical Server Group of the cloud nodes and to 2 Infiniband gateways SX6036G (Mellanox OS 3.3.5006) configured in load balancing Active-Active mode (load balancing algorithm: ib-base-ip)

Neutron is configured with the new Distributed Virtual Router (DVR) feature of Juno (https://wiki.openstack.org/wiki/Neutron/DVR https://wiki.openstack.org/wiki/Neutron/DVR ). With DVR enabled, NAT services are distributed across all the nodes (controller and compute). One node (our cloud controller) provides only SNAT to VMs and all the other compute nodes provides DNAT to VMs.

Networking over the private network it’s working fine. Instead, on the external network problems arise when assigning floating IPs to the VMs. DVR manages floating IPs by creating a FloatingIP router namespace, the Floating IP agent gateway (FIP), on each node that runs a VM to which is assigned a floating IP. Each floating IP of a VM is then assigned to the DVR namespace of the tenant network.

I guess that floating IPs are not always reachable from the internet because, sometimes, the gateways map IP addresses with the wrong IB-MAC addresses in the ARP table.

For example, now the ARP tables of the external network (proxy-arp 7) of the 2 gateways contains

root@master# xdsh gw -l admin --devicetype IBSwitch::Mellanox “show ip arp interface proxy-arp 7”

sx60g01: Mellanox MLNX-OS Switch Management

sx60g01:

sx60g01: Total number of entries: 16

sx60g01: Address Type Hardware Address Interface

sx60g01: ------------------------------------------------------------------------

sx60g01: XXX.XXX.XX.188 Dynamic ETH 00:24:F7:14:7A:C1 proxy-arp 7

sx60g01: XXX.XXX.XX.189 Dynamic ETH 00:24:F7:14:B4:C1 proxy-arp 7

sx60g01: XXX.XXX.XX.190 Dynamic ETH 00:00:0C:07:AC:83 proxy-arp 7

sx60g01: XXX.XXX.XX.130 Dynamic IB 50:05:07:00:5B:01:7E:11 proxy-arp 7

sx60g01: XXX.XXX.XX.132 Dynamic IB 50:05:07:00:5B:01:7E:11 proxy-arp 7

sx60g01: XXX.XXX.XX.134 Dynamic IB 50:05:07:00:5B:01:80:0D proxy-arp 7

sx60g01: XXX.XXX.XX.136 Dynamic IB 50:05:07:00:5B:01:80:0D proxy-arp 7

sx60g01: XXX.XXX.XX.138 Dynamic IB 50:05:07:00:5B:01:7F:E5 proxy-arp 7

sx60g01: XXX.XXX.XX.142 Dynamic IB 50:05:07:00:5B:01:78:19 proxy-arp 7

sx60g01: XXX.XXX.XX.144 Dynamic IB 50:05:07:00:5B:01:7F:D5 proxy-arp 7

sx60g01: XXX.XXX.XX.146 Dynamic IB 50:05:07:00:5B:01:7F:D5 proxy-arp 7

sx60g01: XXX.XXX.XX.148 Dynamic IB 50:05:07:00:5B:01:7E:11 proxy-arp 7

sx60g01: XXX.XXX.XX.152 Dynamic IB 50:05:07:00:5B:01:7F:D5 proxy-arp 7

sx60g01: XXX.XXX.XX.154 Dynamic IB 50:05:07:00:5B:01:7F:E5 proxy-arp 7

sx60g01: XXX.XXX.XX.158 Dynamic IB 50:05:07:00:5B:01:78:19 proxy-arp 7

sx60g01: XXX.XXX.XX.160 Dynamic IB 50:05:07:00:5B:01:7E:CD proxy-arp 7

sx60g01:

sx60g02: Mellanox MLNX-OS Switch Management

sx60g02:

sx60g02: Total number of entries: 12

sx60g02: Address Type Hardware Address Interface

sx60g02: ------------------------------------------------------------------------

sx60g02: XXX.XXX.XX.188 Dynamic ETH 00:24:F7:14:7A:C1 proxy-arp 7

sx60g02: XXX.XXX.XX.189 Dynamic ETH 00:24:F7:14:B4:C1 proxy-arp 7

sx60g02: XXX.XXX.XX.190 Dynamic ETH 00:00:0C:07:AC:83 proxy-arp 7

sx60g02: XXX.XXX.XX.129 Dynamic IB 50:05:07:00:5B:01:7E:11 proxy-arp 7

sx60g02: XXX.XXX.XX.133 Dynamic IB 50:05:07:00:5B:01:7E:B1 proxy-arp 7

sx60g02: XXX.XXX.XX.135 Dynamic IB 50:05:07:00:5B:01:7E:CD proxy-arp 7

sx60g02: XXX.XXX.XX.137 Dynamic IB 50:05:07:00:5B:01:80:0D proxy-arp 7

sx60g02: XXX.XXX.XX.139 Dynamic IB 50:05:07:00:5B:01:78:19 proxy-arp 7

sx60g02: XXX.XXX.XX.141 Dynamic IB 50:05:07:00:5B:01:7F:E5 proxy-arp 7

sx60g02: XXX.XXX.XX.143 Dynamic IB 50:05:07:00:5B:01:7E:B1 proxy-arp 7

sx60g02: XXX.XXX.XX.145 Dynamic IB 50:05:07:00:5B:01:7F:D5 proxy-arp 7

sx60g02: XXX.XXX.XX.147 Dynamic IB 50:05:07:00:5B:01:7E:B1 proxy-arp 7

sx60g02:

So, for example, the IP address XXX.XXX.XX.133, that is unreachable from the internet, is defined on sx60g02 with IB-MAC 50:05:07:00:5B:01:7E:B1. However the node that is running the VM to which that floating IP is assigned, has MAC address 50:05:07:00:5b:01:7e:cd.

Moreover, looking at the ARP table of sx60g01, you can see that there are entries for the following IPs

sx60g01: XXX.XXX.XX.148 Dynamic IB 50:05:07:00:5B:01:7E:11 proxy-arp 7

sx60g01: XXX.XXX.XX.152 Dynamic IB 50:05:07:00:5B:01:7F:D5 proxy-arp 7

sx60g01: XXX.XXX.XX.154 Dynamic IB 50:05:07:00:5B:01:7F:E5 proxy-arp 7

sx60g01: XXX.XXX.XX.158 Dynamic IB 50:05:07:00:5B:01:78:19 proxy-arp 7

sx60g01: XXX.XXX.XX.160 Dynamic IB 50:05:07:00:5B:01:7E:CD proxy-arp 7

I have assigned these floating IPs to VMs more that a week ago. At the moment no nodes, either virtual or real, have these IPs assigned.

Why are they still present in the ARP table of the gateway?

Thank you very much in advance

Ale

Hi Eddie,

just to inform the community.

I found the issue I described above is also related to a known bug of Neutron DVR implementation on Juno 2014.2.1

Bug #1389880 “VM loses connectivity on floating ip association w…” : Bugs : neutron https://bugs.launchpad.net/neutron/+bug/1389880

This bug is solved in the release 2014.2.2 of Juno (I have just update it ;-) )

Thanks

Ale

Hi Eddie,

I’m looking forward the new version of MLNX-OS will be released :-)

Thank you very much

Ale

Hi Ale,

There is a known issue with Gratuitous ARP handling in GW HA mode in code below 3.4.0012 and below, we should release a code with the fix soon (Until End Of March),

for now I would suggest clearing the Arp Entry from all of the GW’s with the below commands:

Entire arp table for the Proxy-arp interface:

switch# clear ip arp interface proxy-arp X

when X is the proxy-arp Interface id

or a specific entry:

switch# clear ip arp 1.1.1.133

Hi,

I have a question for you Ale . I’ve installed in my company OpenStack(Juno) with 5 compute node, 1 network node, 1 controller node and 1 storage node using CentOS 7.The compute node have infiniband. I use a qlogic infiniband switch. I would like to virtualize the instance with infiniband. What do you do to realize it? Have you follow this Neutron/ML2 - OpenStack https://wiki.openstack.org/wiki/Neutron/ML2 ? thank you so much

Hi Vincenzo,

what do you mean for “virtualize the instance with infiniband”? Do you want to provide IB adapters to your instances via SR-IOV virtual functions? In the configuration I described above I have not set up SR-IOV virtual functions but I only use the Ethernet paravirtualization driver (eth_ipoib) on top of IPoIB devices. If you are interested in true Infiniband virtualization via SR-IOV virtual functions you should refer to these links:

Mellanox-Neutron-ML2 - OpenStack https://wiki.openstack.org/wiki/Mellanox-Neutron-ML2 (see the Infiniband links)

Mellanox OFED Driver Installation and Configuration for SR-IOV https://community.mellanox.com/s/article/mellanox-ofed-driver-installation-and-configuration-for-sr-iov

I didn’t try it with Juno but last year I set up Openstack Havana with SR-IOV, so please feel free to contact me if you need help.

Thanks,

Ale

Hi Ale,

thank you for your answer!!!

Let me try to explain it better… If I launch an instance I would like that the Infiniband NIC is available, so, for example, I can associate a floating ip using infiniband