DPDK-Pktgen: Did not find any ports to use with ConnectX-5 VF

What’s the problem

I create a VM and attach a VF(ConnectX-5 SRIOV) to it using virt-manager (Reference)

VF should be configured successfully, as I can ping other IPs with it.

I compiled DPDK successfully following this Guide

~/dpdk-23.11$ cat VERSION
23.11.0

Then I compiled DPDK-Pktgen following this Guide

I tried to run pktgen:

$ sudo ./pktgen -l 0,1 -n 4 -a 07:00.0 -- -P -m "[1].0" -T

*** Copyright(c) <2010-2023>, Intel Corporation. All rights reserved.
*** Pktgen  created by: Keith Wiles -- >>> Powered by DPDK <<<

EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created

!PANIC!: *** Did not find any ports to use ***
EAL: PANIC in pktgen_config_ports():
*** Did not find any ports to use ***
0: /usr/local/lib/x86_64-linux-gnu/librte_eal.so.24 (rte_dump_stack+0x42) [7fbd3eb7d062]
1: /usr/local/lib/x86_64-linux-gnu/librte_eal.so.24 (__rte_panic+0xd4) [7fbd3eb552a0]
2: ./pktgen (55a87c076000+0x36175) [55a87c0ac175]
3: ./pktgen (55a87c076000+0xb587) [55a87c081587]
4: /lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main+0xf3) [7fbd3e776083]
5: ./pktgen (55a87c076000+0xbb7e) [55a87c081b7e]
Aborted

Info of my system

$ ./dpdk-23.11/usertools/dpdk-devbind.py -s

Network devices using kernel driver
===================================
0000:01:00.0 'Virtio network device 1041' if=enp1s0 drv=virtio-pci unused=vfio-pci *Active*
0000:07:00.0 'MT416842 BlueField multicore SoC family VF a2d3' if=enp7s0np0 drv=mlx5_core unused=vfio-pci

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'DMA' devices detected
=========================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

Misc (rawdev) devices using kernel driver
=========================================
0000:03:00.0 'Virtio block device 1042' drv=virtio-pci unused=vfio-pci
0000:04:00.0 'Virtio block device 1042' drv=virtio-pci unused=vfio-pci

No 'Regex' devices detected
===========================

No 'ML' devices detected
========================
$ grep -i huge /proc/meminfo
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:    1024
HugePages_Free:     1023
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:         2097152 kB
$ uname -r
5.4.0-174-generic
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.6 LTS
Release:        20.04
Codename:       focal

What I tried

Change driver

Reference: This Github Issue

$ sudo ./dpdk-23.11/usertools/dpdk-devbind.py -b vfio-pci 07:00.0
Error: bind failed for 0000:07:00.0 - Cannot bind to driver vfio-pci: [Errno 19] No such device
Error: unbind failed for 0000:07:00.0 - Cannot open /sys/bus/pci/drivers//unbind: [Errno 13] Permission denied: '/sys/bus/pci/drivers//unbind'
$ sudo ./dpdk-23.11/usertools/dpdk-devbind.py -b vfio-pci 0000:07:00.0
Error: bind failed for 0000:07:00.0 - Cannot bind to driver vfio-pci: [Errno 19] No such device

Error: No such device - I don’t know why.

Add parameter to pktgen

Reference: Stack Overflow

$ sudo ./pktgen -l 0,1 -n 4 -a 07:00.0 -d librte_net_mlx5.so -- -P -m "[1].0" -T

*** Copyright(c) <2010-2023>, Intel Corporation. All rights reserved.
*** Pktgen  created by: Keith Wiles -- >>> Powered by DPDK <<<

EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:a2d3) device: 0000:07:00.0 (socket -1)


====== Pktgen got a Segment Fault

Obtained 12 stack frames.
./pktgen(+0x29535) [0x558259321535]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f9d49dc1420]
/usr/local/lib/x86_64-linux-gnu/librte_net_mlx5.so(+0x1ca8c) [0x7f9d494dea8c]
/usr/local/lib/x86_64-linux-gnu/librte_net_mlx5.so(+0x1e2e3) [0x7f9d494e02e3]
/usr/local/lib/x86_64-linux-gnu/librte_net_mlx5.so(+0x15e0b0) [0x7f9d496200b0]
/usr/local/lib/x86_64-linux-gnu/librte_common_mlx5.so.24(+0xa7cc) [0x7f9d4949b7cc]
/usr/local/lib/x86_64-linux-gnu/librte_bus_pci.so.24(+0x4500) [0x7f9d49b7f500]
/usr/local/lib/x86_64-linux-gnu/librte_eal.so.24(rte_bus_probe+0x33) [0x7f9d49fc5313]
/usr/local/lib/x86_64-linux-gnu/librte_eal.so.24(+0x15654) [0x7f9d49fc1654]
./pktgen(+0xaf38) [0x558259302f38]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f9d49bdd083]
./pktgen(+0xbb7e) [0x558259303b7e]

Got a Segment Fault

Set CONFIG_RTE_LIBRTE_MLX5_PMD=y

Reference: Nvidia Forum

~/dpdk-23.11$ cat config/common_base
CONFIG_RTE_LIBRTE_MLX5_PMD=y

I added config/common_base in DPDK folder and compiled DPDK.

I don’t know whether this work or not.

Reinstall MLNX_OFED driver

I download MLNX_OFED_LINUX-24.01-0.3.3.1-ubuntu20.04-x86_64 from Nvidia website and install:

sudo ./mlnxofedinstall --upstream-libs --dpdk --force

But it didn’t solve my problem.

Could anyone help me resolve this bug that has been troubling me for a long time?

I am still exploring:

  • Add GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt" to /etc/default/grub
  • sudo update-grub and reboot my VM

Unfortunately it didn’t let dpdk-devbind.py work, then I refer to Stack Overflow

$ modprobe vfio enable_unsafe_noiommu_mode=1
$ echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

After this, I can use dpdk-devbind.py to change the driver of my VF to vfio-pci.

Network devices using DPDK-compatible driver 
============================================
0000:07:00.0 'MT416842 BlueField multicore SoC family VF a2d3' drv=vfio-pci unused=mlx5_core

But I still cannot use Pktgen:

$ sudo ./pktgen -l 0,1 -n 4 -a 07:00.0 -- -P -m "[1].0" -T

*** Copyright(c) <2010-2023>, Intel Corporation. All rights reserved.
*** Pktgen  created by: Keith Wiles -- >>> Powered by DPDK <<<

EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
TELEMETRY: No legacy callbacks, legacy socket not created

!PANIC!: *** Did not find any ports to use ***
EAL: PANIC in pktgen_config_ports():
*** Did not find any ports to use ***
0: /usr/local/lib/x86_64-linux-gnu/librte_eal.so.24 (rte_dump_stack+0x42) [7fdbebd60062]
1: /usr/local/lib/x86_64-linux-gnu/librte_eal.so.24 (__rte_panic+0xd4) [7fdbebd382a0]
2: ./pktgen (56525adea000+0x36175) [56525ae20175]
3: ./pktgen (56525adea000+0xb587) [56525adf5587]
4: /lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main+0xf3) [7fdbeb959083]
5: ./pktgen (56525adea000+0xbb7e) [56525adf5b7e]
Aborted

So this problem seems to be Nvidia-specific?
Hope somebody can help!

Finally I solved this.
I think this problem is related to shared library about MLX5 PMD or the version of DPDK.
What I have done to fix this:

  • Use DPDK 24.03.0 instead of DPDK 23.11.0
  • Install MLNX_OFED_LINUX-5.8-4.1.5.0, recompile DPDK 24.03.0 and Pktgen

Now I can run Pktgen with option -d librte_net_mlx5.so

I think in the first time I compiled DPDK, I didn’t install MLNX_OFED_LINUX-5 at that time,
so DPDK was compiled without libraries about MLX5.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.