Use bluefiled smartnic to configure hardware vdpa failure, how to fix it?

I’m following the MLNX_OFED v5.1-0.6.6.0 Manual VirtIO Acceleration through Hardware vDPA, try to configure HW vDPA. But, it fails when I change eswitch from legacy to switchdev mode. dmesg print the error log as below:

"

mlx5_core 0000:86:00.0: E-Switch: Disable: mode(LEGACY), nvfs(4), active vports(5)

mlx5_core 0000:86:00.0: E-Switch: Devlink can’t get param fdb_large_groups, uses default (15).

mlx5_core 0000:86:00.0: E-Switch: Supported tc offload range - chains: 3, prios: 16

mlx5_core 0000:86:00.0: mlx5_cmd_check:794:(pid 566983): SET_FLOW_TABLE_ENTRY(0x936) op_mod(0x0) failed, status bad parameter(0x3), syndrome (0xd59406)

mlx5_core 0000:86:00.0: E-Switch: Failed to create fdb chains err(-22)

mlx5_core 0000:86:00.0: E-Switch: Devlink can’t get param fdb_large_groups, uses default (15).

mlx5_core 0000:86:00.0: E-Switch: Enable: mode(LEGACY), nvfs(4), active vports(5)

"

My server is centos 8.2 x86_64, kernel version 4.18.0-193.6.3.el8_2.x86_64.

MLNX_OFED version: MLNX_OFED_LINUX-5.1-0.6.6.0-rhel8.2-x86_64.tgz

user manual:MLNX_OFED Documentation Rev 5.1-0.6.6.0

install ofed command: ./mlnxofedinstall --with-rshim --ovs-dpdk --upstream-libs --add-kernel-support

qemu-4.0.0.tar.xz:https://download.qemu.org/qemu-4.0.0.tar.xz

qemu configure command: …/configure --target-list=x86_64-softmmu --enable-kvm --enable-vnc --enable-spice

libvirt-6.6.0.tar.xz:https://libvirt.org/sources/libvirt-6.6.0.tar.xz

libvirt configure command: …/autogen.sh --no-git --with-qemu --without-vbox --without-lxc --without-openvz --without-vmware

virt-manager-2.2.1.tar.gz:https://virt-manager.org/download/sources/virt-manager/virt-manager-2.2.1.tar.gz

DPDK 20.02.1:https://fast.dpdk.org/rel/dpdk-20.02.1.tar.xz

dpdk config command: make config T=x86_64-native-linuxapp-gcc

everything before change eswitch mode step look ok.

VF configure is ok:

lspci | grep Mel

86:00.0 Ethernet controller: Mellanox Technologies MT416842 BlueField integrated ConnectX-5 network controller

86:00.1 Ethernet controller: Mellanox Technologies MT416842 BlueField integrated ConnectX-5 network controller

86:00.2 DMA controller: Mellanox Technologies MT416842 BlueField SoC management interfac

86:00.3 Ethernet controller: Mellanox Technologies MT416842 BlueField multicore SoC family VF

86:00.4 Ethernet controller: Mellanox Technologies MT416842 BlueField multicore SoC family VF

Hi ,

Please refer to the MLNX_OFED release notes to the known issues section:

https://docs.mellanox.com/display/OFED510660/Known+Issues

2210752

Description: Switching from Legacy mode to SwitchDev mode and vice-versa while TC rules exist on the NIC will result in failure.

Workaround: Before attempting to switch mode, make sure to delete all TC rules on the NIC or stop OpenvSwitch.

2062900

Description: Moving VF from SwitchDev mode to Legacy mode while the representor is being used by OVS-DPDK results in a segmentation fault.

Workaround: To move VF to Legacy mode with no error, make sure to delete the ports from the OVS.

1888574

Description: Kernel support limitations in the current MLNX_OFED version:

SR-IOV SwitchDev is only supported on Kernel 4.14 and above, and on RedHat/CentOS 7.4, 7.5 and 7.6.

SR-IOV Legacy is only supported on Kernel 4.3 and above, and on RedHat/CentOS 7.4, 7.5, 7.6 and 7.7.

Thanks,

Samer

tks, may be the centos 8.2 cause this problem, i will try the centos 7.6.😂