Trouble running dpdk-testpmd on a KVM machine using ConnectX-5 interfaces

Host configuration:

OS: Debian GNU/Linux 10 (buster)

Kernel: 4.19.0-17-amd64

Architecture: x86_64

mlnx_oefd version: MLNX_OFED_LINUX-5.4-1.0.3.0-debian10.8-x86_64

Guest configuration:

OS: Ubuntu 16.04.1 LTS

Kernel: 4.4.0-47-generic

mlnx_ofed version: MLNX_OFED_LINUX-5.4-1.0.3.0-ubuntu16.04-x86_64

Dpdk version: 18.11

Please see the attachment for more detailed configuration.

This is what I get when I try to run testpmd on the guest.

EAL: PCI device 0000:00:03.0 on NUMA socket -1

EAL: Invalid NUMA socket, default to 0

EAL: probe driver: 15b3:101a net_mlx5

net_mlx5: mlx5_flow.c:364: mlx5_flow_discover_priorities(): port 0 verbs maximum priority: 8 expected 8/16

testpmd: /home/sbandi/dpdk/drivers/net/mlx5/mlx5.c:1259: mlx5_dev_spawn: Assertion `(mlx5_glue->dealloc_pd(pd)) == 0’ failed.

Please let me know what is going wrong here.

mlnx.txt (25.9 KB)

mlx5_flow_discover_priorities is a verbs based function and Mellanox OFED v5.4 more is rdma-core based. There were changes in this portion of the code

Check DPDK v18.11 release notes to find tested MOFED version

http://doc.dpdk.org/guides-18.11/rel_notes/release_18_11.html

I would suggest to switch to latest LTS DPDK and see if the issue is still there. It is also worth trying latest DPDK version.

In addition, firmware version running on the host is not supported by this version of the MOFED. Please, check MOFED release notes for supported version. As you are running OEM (DELL) card, you might contact OEM to get relevant version. If you have a Mellanox branded PSID, you should be able to get latest version from Mellanox web site.

You might also try MOFED v4.9 LTS, that still using verbs. Looks like your OS’es are supported - https://docs.mellanox.com/display/MLNXOFEDv494080/General+Support+in+MLNX_OFED

Thank you for the response.

I did try downgrading to MOFED v4.9 LTS, but now I’m facing an issue with loading libibverbs library while running testpmd.

root@cep:~# ./testpmd -w 0000:00:03.0

net_mlx5: mlx5.c:1712: mlx5_glue_init(): cannot load glue library: /usr/lib/libibverbs.so.1: version `IBVERBS_1.8’ not found (required by /usr/lib/x86_64-linux-gnu/librte_pmd_mlx5_glue.so.18.11.0)

net_mlx5: mlx5.c:1730: mlx5_glue_init(): cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx5)

The libibverbds is part of the MOFED installation, so I’m not sure what dpdk is complaining about. Would you have any idea?

Seems like MOFED installation issue. MOFED does contain libibverbs.

I would highly recommend to use latest 5.4 and 20.11 DPDK version first. Install MOFED with ‘mlnxofedinstall --all’ and the compile DPDK

Download, extract and run

meson --prefix= build

Using prefix is very recommended until you finish our development cycle.

ninja -C build install

It could be that you have to add meson/ninja packages to your OS