DPDK-Pktgen: Did not find any ports to use with ConnectX-5 VF

aniurm35 · April 4, 2024, 9:37am

What’s the problem

I create a VM and attach a VF(ConnectX-5 SRIOV) to it using virt-manager (Reference)

VF should be configured successfully, as I can ping other IPs with it.

I compiled DPDK successfully following this Guide

~/dpdk-23.11$ cat VERSION
23.11.0

Then I compiled DPDK-Pktgen following this Guide

I tried to run pktgen:

$ sudo ./pktgen -l 0,1 -n 4 -a 07:00.0 -- -P -m "[1].0" -T

*** Copyright(c) <2010-2023>, Intel Corporation. All rights reserved.
*** Pktgen  created by: Keith Wiles -- >>> Powered by DPDK <<<

EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created

!PANIC!: *** Did not find any ports to use ***
EAL: PANIC in pktgen_config_ports():
*** Did not find any ports to use ***
0: /usr/local/lib/x86_64-linux-gnu/librte_eal.so.24 (rte_dump_stack+0x42) [7fbd3eb7d062]
1: /usr/local/lib/x86_64-linux-gnu/librte_eal.so.24 (__rte_panic+0xd4) [7fbd3eb552a0]
2: ./pktgen (55a87c076000+0x36175) [55a87c0ac175]
3: ./pktgen (55a87c076000+0xb587) [55a87c081587]
4: /lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main+0xf3) [7fbd3e776083]
5: ./pktgen (55a87c076000+0xbb7e) [55a87c081b7e]
Aborted

Info of my system

$ ./dpdk-23.11/usertools/dpdk-devbind.py -s

Network devices using kernel driver
===================================
0000:01:00.0 'Virtio network device 1041' if=enp1s0 drv=virtio-pci unused=vfio-pci *Active*
0000:07:00.0 'MT416842 BlueField multicore SoC family VF a2d3' if=enp7s0np0 drv=mlx5_core unused=vfio-pci

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'DMA' devices detected
=========================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

Misc (rawdev) devices using kernel driver
=========================================
0000:03:00.0 'Virtio block device 1042' drv=virtio-pci unused=vfio-pci
0000:04:00.0 'Virtio block device 1042' drv=virtio-pci unused=vfio-pci

No 'Regex' devices detected
===========================

No 'ML' devices detected
========================

$ grep -i huge /proc/meminfo
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:    1024
HugePages_Free:     1023
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2097152 kB

$ uname -r
5.4.0-174-generic

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.6 LTS
Release:        20.04
Codename:       focal

What I tried

Change driver

Reference: This Github Issue

$ sudo ./dpdk-23.11/usertools/dpdk-devbind.py -b vfio-pci 07:00.0
Error: bind failed for 0000:07:00.0 - Cannot bind to driver vfio-pci: [Errno 19] No such device
Error: unbind failed for 0000:07:00.0 - Cannot open /sys/bus/pci/drivers//unbind: [Errno 13] Permission denied: '/sys/bus/pci/drivers//unbind'
$ sudo ./dpdk-23.11/usertools/dpdk-devbind.py -b vfio-pci 0000:07:00.0
Error: bind failed for 0000:07:00.0 - Cannot bind to driver vfio-pci: [Errno 19] No such device

Error: No such device - I don’t know why.

Add parameter to pktgen

Reference: Stack Overflow

$ sudo ./pktgen -l 0,1 -n 4 -a 07:00.0 -d librte_net_mlx5.so -- -P -m "[1].0" -T

*** Copyright(c) <2010-2023>, Intel Corporation. All rights reserved.
*** Pktgen  created by: Keith Wiles -- >>> Powered by DPDK <<<

EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:a2d3) device: 0000:07:00.0 (socket -1)


====== Pktgen got a Segment Fault

Obtained 12 stack frames.
./pktgen(+0x29535) [0x558259321535]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f9d49dc1420]
/usr/local/lib/x86_64-linux-gnu/librte_net_mlx5.so(+0x1ca8c) [0x7f9d494dea8c]
/usr/local/lib/x86_64-linux-gnu/librte_net_mlx5.so(+0x1e2e3) [0x7f9d494e02e3]
/usr/local/lib/x86_64-linux-gnu/librte_net_mlx5.so(+0x15e0b0) [0x7f9d496200b0]
/usr/local/lib/x86_64-linux-gnu/librte_common_mlx5.so.24(+0xa7cc) [0x7f9d4949b7cc]
/usr/local/lib/x86_64-linux-gnu/librte_bus_pci.so.24(+0x4500) [0x7f9d49b7f500]
/usr/local/lib/x86_64-linux-gnu/librte_eal.so.24(rte_bus_probe+0x33) [0x7f9d49fc5313]
/usr/local/lib/x86_64-linux-gnu/librte_eal.so.24(+0x15654) [0x7f9d49fc1654]
./pktgen(+0xaf38) [0x558259302f38]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f9d49bdd083]
./pktgen(+0xbb7e) [0x558259303b7e]

Got a Segment Fault

Set `CONFIG_RTE_LIBRTE_MLX5_PMD=y`

Reference: Nvidia Forum

~/dpdk-23.11$ cat config/common_base
CONFIG_RTE_LIBRTE_MLX5_PMD=y

I added config/common_base in DPDK folder and compiled DPDK.

I don’t know whether this work or not.

Reinstall MLNX_OFED driver

I download MLNX_OFED_LINUX-24.01-0.3.3.1-ubuntu20.04-x86_64 from Nvidia website and install:

sudo ./mlnxofedinstall --upstream-libs --dpdk --force

But it didn’t solve my problem.

Could anyone help me resolve this bug that has been troubling me for a long time?

aniurm35 · April 5, 2024, 12:46pm

I am still exploring:

Add GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt" to /etc/default/grub
sudo update-grub and reboot my VM

Unfortunately it didn’t let dpdk-devbind.py work, then I refer to Stack Overflow

$ modprobe vfio enable_unsafe_noiommu_mode=1
$ echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

After this, I can use dpdk-devbind.py to change the driver of my VF to vfio-pci.

Network devices using DPDK-compatible driver 
============================================
0000:07:00.0 'MT416842 BlueField multicore SoC family VF a2d3' drv=vfio-pci unused=mlx5_core

But I still cannot use Pktgen:

$ sudo ./pktgen -l 0,1 -n 4 -a 07:00.0 -- -P -m "[1].0" -T

*** Copyright(c) <2010-2023>, Intel Corporation. All rights reserved.
*** Pktgen  created by: Keith Wiles -- >>> Powered by DPDK <<<

EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created

!PANIC!: *** Did not find any ports to use ***
EAL: PANIC in pktgen_config_ports():
*** Did not find any ports to use ***
0: /usr/local/lib/x86_64-linux-gnu/librte_eal.so.24 (rte_dump_stack+0x42) [7fdbebd60062]
1: /usr/local/lib/x86_64-linux-gnu/librte_eal.so.24 (__rte_panic+0xd4) [7fdbebd382a0]
2: ./pktgen (56525adea000+0x36175) [56525ae20175]
3: ./pktgen (56525adea000+0xb587) [56525adf5587]
4: /lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main+0xf3) [7fdbeb959083]
5: ./pktgen (56525adea000+0xbb7e) [56525adf5b7e]
Aborted

So this problem seems to be Nvidia-specific?
Hope somebody can help!

aniurm35 · April 7, 2024, 8:20am

Finally I solved this.
I think this problem is related to shared library about MLX5 PMD or the version of DPDK.
What I have done to fix this:

Use DPDK 24.03.0 instead of DPDK 23.11.0
Install MLNX_OFED_LINUX-5.8-4.1.5.0, recompile DPDK 24.03.0 and Pktgen

Now I can run Pktgen with option -d librte_net_mlx5.so

I think in the first time I compiled DPDK, I didn’t install MLNX_OFED_LINUX-5 at that time,
so DPDK was compiled without libraries about MLX5.

system · April 21, 2024, 8:21am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
DPDK ports not getting detected on CX-5 EX.. What did i miss here..? net_mlx5: mlx5.c:1740: mlx5_glue_init(): cannot load glue library: librte_pmd_mlx5_glue.so.18.11.0: cannot open shared object file: No such file or directory Software And Drivers	4	894	June 30, 2020
SR-IOV binding inside VM for PktGen	2	349	August 24, 2017
DPDK-OVS using connectx3-Pro binding issues Application Accelerator Software ethtool , test_pmd	10	980	January 26, 2016
DPDK APIs rte_eth_dev_count_avail() returns 0 with the MLX5 NIC card Software And Drivers mst	8	2139	November 29, 2019
DPDK testpmd can't access device Application Accelerator Software	4	1377	January 12, 2017
DPDK can't probe ConnectX-4 card (Operation not supported) Software And Drivers	1	837	January 2, 2020
dpdk and connectx3 problem Application Accelerator Software	8	948	December 6, 2018
DPDK not working with ConnectX-3 card on Openstack virtual setup Software And Drivers openstack , port	8	1126	May 10, 2019
Failed to config the SR-IOV and attach the VF to dpdk Ethernet Adapter Cards dpdk , software-and-drivers , set , bind	2	828	June 20, 2019
Unable to use Mellanox CX5 VF in my DPDK application Mellanox OFED	2	3094	March 31, 2022