【OS】
VM Version: qemu-kvm-7.0.0
Host And VM OS Verion: Rocky 8.10
Kernel Verion: 4.18.0-553.el8_10.x86_64
【Network Card】
MLX CX6 2*100G RoCE v2
MLNX_OFED_LINUX-23.10-3.2.2.0-rhel8.10-x86_64
【BUG Info】
QSG for High Availability with NVIDIA Enhanced SR-IOV with Bonding Support (VF-LAG) - NVIDIA Docs
OVS-Kernel Hardware Offloads - NVIDIA Docs
【Step】
lspci | grep “Mellanox”
mstconfig -d 31:00.0 set SRIOV_EN=1 NUM_OF_VFS=4
mstconfig -d 31:00.1 set SRIOV_EN=1 NUM_OF_VFS=4
iptables -F
ethtool -K ens3f0np0 hw-tc-offload on
ethtool -K ens3f1np1 hw-tc-offload on
cat /sys/class/net/ens3f0np0/device/sriov_numvfs
cat /sys/class/net/ens3f1np1/device/sriov_numvfs
echo 0 > /sys/class/net/ens3f0np0/device/sriov_numvfs
echo 2 > /sys/class/net/ens3f0np0/device/sriov_numvfs
echo 0 > /sys/class/net/ens3f1np1/device/sriov_numvfs
echo 2 > /sys/class/net/ens3f1np1/device/sriov_numvfs
VFS_PCI=($(lspci | grep “Mellanox” | grep “Virtual” | cut -d " " -f 1));
for i in ${VFS_PCI[@]};
do
echo “unbinding VF $i”;
echo “0000:${i}” >> /sys/bus/pci/drivers/mlx5_core/unbind;
done
devlink dev eswitch set pci/0000:31:00.0 mode switchdev
devlink dev eswitch set pci/0000:31:00.1 mode switchdev
service openvswitch status
systemctl start openvswitch
systemctl enable openvswitch
service openvswitch status
ovs-vsctl show
ovs-vsctl add-br ovs-sriov
ovs-vsctl add-port ovs-sriov bond0
ovs-vsctl add-port ovs-sriov eth0
ovs-vsctl add-port ovs-sriov eth1
ovs-vsctl add-port ovs-sriov eth2
ovs-vsctl add-port ovs-sriov eth3
ovs-vsctl show
ovs-vsctl set Open_vSwitch . other_config:hw-offload=true
systemctl restart openvswitch
When testing the VF LAG functionality of the CX6 RoCE NIC, I encountered a peculiar issue. Using Rocky 8.10 and the native driver, after running the command ovs-vsctl add-port ovs-sriov bond0 and passing through only the VFs of a single PF to the VM, the ib_send_bw test shows no issues with connectivity and failover, but the NIC can only reach a bandwidth of 100G. However, when VFs from two PFs are both added to ovs-sriov, the ib_send_bw test can establish a connection, but no traffic can be transmitted.
