VXLAN offload/RSS not working under OVS switch scenario

Infrastructure & Networking Adapters and Cables Ethernet Adapter Cards
1

Hi folks,

We are using two Mellanox Connectx-5 EN (2-port 100 GBE) cards to validate performance under an Openstack vxlan tenant scenario using OVS bridges. The Mellanox cards are NOT in SR-IOV mode and are currently connected back-to-back to create the vxlan tenant network with core IP addresses applied directly to the ports. It is our understanding from reading Mellanox literature that under this scenario the Mellanox should be offloading the VXLAN tunnel udp encap/decap somewhat automatically (as per the OVS switch ‘vxlan’ interface configuration); however, at the moment this does not seem to be occurring. Traffic is being generated across multiple VNID’s, however all ingress traffic from the vxlan/tenant network is getting processed by a single core softirq process. We are not able to verify if this is b/c something is not setup properly in regard to the Mellanox card/drivers or if we do not have RSS tuned as needed. Further details below, any assistance on configuration/debug suggestions or updated documentation for this scenario under Connectx-5 would be appreciated!

Further details:

We have been reading from the following document although it is targeted for Connectx-3 ( HowTo Configure VXLAN for ConnectX-3 Pro (Linux OVS) https://community.mellanox.com/s/article/howto-configure-vxlan-for-connectx-3-pro--linux-ovs-x . This document shows several items to be configured (DMFS & VXLAN port number) but it seems both of these should be enabled now under Connectx-5 and the ‘mlx5_core’ driver. Also the recommended debug/log steps in this document are no longer supported under Connectx-5. In our setup we are using the apt package install for OVS.

Mellanox card: Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]

Server version: Ubuntu 18.04 (kernel 4.15.0-36-generic)

OVS version: ovs-vsctl (Open vSwitch) 2.9.0

root@us-ral-1a-e1:~# mlxfwmanager --query

Querying Mellanox devices firmware …

Device #1:

Device Type: ConnectX5

Part Number: MCX516A-CCA_Ax

Description: ConnectX-5 EN network interface card; 100GbE dual-port QSFP28; PCIe3.0 x16; tall bracket; ROHS R6

PSID: MT_0000000012

PCI Device Name: /dev/mst/mt4119_pciconf0

Base GUID: 98039b03004dd018

Base MAC: 98039b4dd018

Versions: Current Available

FW 16.23.1020 N/A

PXE 3.5.0504 N/A

UEFI 14.16.0017 N/A

Status: No matching image found

root@us-ral-1a-e1:~#

2

Hey guys, sorry did not realize that posting to the Mellanox community was the same as opening a support case. We were told via other email avenues that we needed to open a support ticket to really get anyone technically involved, so we took that avenue and were then instructed to open a support case attached to the card S/N + support contract. Sorry for any confusion.

3

Hello Robert,

Thank you for posting your question on the Mellanox Community.

Based on the information provided, we noticed that you also opened a Mellanox Support case which is already assigned to one of our engineers.

We will continue to assist you further through the Mellanox Support case.

Thanks and regards,

~Mellanox Technical Support

4

Issue for us was solved by Mellanox support team, turned out to b e the need for ‘ethtool -X eth3 hfunc toeplitz’ configuration on all Mellanox vxlan interfaces. This changes the hash mode from ‘xor’ to ‘toeplitz’.