Fail to start l2fwd-nv

Hi,

Followed the instructions shown here: https://github.com/NVIDIA/l2fwd-nv. The binary builds flawlessly but fails to start:

[root@cu-l2fwd l2fwd-nv]# taskset -pc 1

pid 1’s current affinity list: 2,7-9,11,13

[root@cu-l2fwd l2fwd-nv]# ./build/l2fwdnv -l 2,7-9,11,13 -n 8 -a b5:00.1,txq_inline_max=0 – -m 1 -w 0 -b 64 -p 4 -v 0 -z 0

************ L2FWD-NV ************

EAL: Detected 24 lcore(s)

EAL: Detected 1 NUMA nodes

EAL: Detected static linkage of DPDK

EAL: failed to parse device “b5:00.1”

EAL: Unable to parse device ‘b5:00.1,txq_inline_max=0’

EAL: Error - exiting with code: 1

Cause: Invalid EAL arguments

[root@cu-l2fwd l2fwd-nv]# ethtool -i enp1f1

driver: mlx5_core

version: 5.3-1.0.0

firmware-version: 22.31.1014 (MT_0000000500)

expansion-rom-version:

bus-info: 0000:b5:00.1

supports-statistics: yes

supports-test: yes

supports-eeprom-access: no

supports-register-dump: no

supports-priv-flags: yes

[root@cu-l2fwd l2fwd-nv]# ethtool -l enp1f1

Channel parameters for enp1f1:

Pre-set maximums:

RX: 0

TX: 0

Other: 512

Combined: 24

Current hardware settings:

RX: 0

TX: 0

Other: 0

Combined: 24

[root@cu-l2fwd l2fwd-nv]# ethtool -a .

Pause parameters for .:

Cannot get device pause settings: No such device

[root@cu-l2fwd l2fwd-nv]# ethtool -a enp1f1

Pause parameters for enp1f1:

Autonegotiate: off

RX: off

TX: off

################

Additional Info:

controller-0:/home/sysadmin# lsmod | grep nvidia

nvidia_peermem 4467 0

nvidia_modeset 1185108 0

nvidia_uvm 995677 0

nvidia 35219551 68 gdrdrv,nvidia_modeset,nvidia_peermem,nvidia_uvm

drm 441205 1 nvidia

ib_core 349049 10 rdma_cm,ib_cm,iw_cm,mlx5_ib,ib_umad,nvidia_peermem,ib_uverbs,rdma_ucm,ib_ipoib

controller-0:/home/sysadmin# mstflint -d b5:00.0 v

Reading Boot image component - OK

FS4 failsafe image

/0x00000018-0x0000001f (0x000008)/ (HW_POINTERS) - OK

/0x00000020-0x00000027 (0x000008)/ (HW_POINTERS) - OK

/0x00000028-0x0000002f (0x000008)/ (HW_POINTERS) - OK

/0x00000030-0x00000037 (0x000008)/ (HW_POINTERS) - OK

/0x00000038-0x0000003f (0x000008)/ (HW_POINTERS) - OK

/0x00000040-0x00000047 (0x000008)/ (HW_POINTERS) - OK

/0x00000500-0x0000053f (0x000040)/ (TOOLS_AREA) - OK

/0x00001000-0x00003c33 (0x002c34)/ (BOOT2) - OK

/0x00004000-0x0000401f (0x000020)/ (ITOC_HEADER) - OK

/0x00006000-0x0001e87f (0x018880)/ (IRON_PREP_CODE) - OK

/0x0001e880-0x0001e97f (0x000100)/ (FS3_RESET_INFO) - OK

/0x0001ed00-0x004cee7f (0x4b0180)/ (MAIN_CODE) - OK

/0x004cee80-0x004e5723 (0x0168a4)/ (PCIE_LINK_CODE) - OK

/0x004e5780-0x004e616f (0x0009f0)/ (POST_IRON_BOOT_CODE) - OK

/0x004e6180-0x00533807 (0x04d688)/ (PCI_CODE) - OK

/0x00533880-0x0053522b (0x0019ac)/ (UPGRADE_CODE) - OK

/0x00535280-0x0054daff (0x018880)/ (PHY_UC_CODE) - OK

/0x0054db00-0x00560d3f (0x013240)/ (PHY_UC_CMD) - OK

/0x00560d80-0x0056292b (0x001bac)/ (UNKNOWN:0xd) - OK

/0x00562980-0x005631ab (0x00082c)/ (UNKNOWN:0xe) - OK

/0x00563200-0x005635ff (0x000400)/ (IMAGE_INFO) - OK

/0x00563600-0x005641ff (0x000c00)/ (FW_MAIN_CFG) - OK

/0x00564200-0x0056467f (0x000480)/ (FW_BOOT_CFG) - OK

/0x00564680-0x0056587f (0x001200)/ (HW_MAIN_CFG) - OK

/0x00565880-0x00565e3f (0x0005c0)/ (HW_BOOT_CFG) - OK

/0x00565e80-0x0056637f (0x000500)/ (PHY_UC_CONSTS) - OK

/0x00566380-0x005664bf (0x000140)/ (IMAGE_SIGNATURE_256) - CRC IGNORED

/0x00566500-0x00566dff (0x000900)/ (PUBLIC_KEYS_2048) - OK

/0x00566e00-0x00566e8f (0x000090)/ (FORBIDDEN_VERSIONS) - OK

/0x00566f00-0x0056713f (0x000240)/ (IMAGE_SIGNATURE_512) - CRC IGNORED

/0x00567180-0x0056827f (0x001100)/ (PUBLIC_KEYS_4096) - OK

/0x00568280-0x0056937f (0x001100)/ (FS4_RSA_PUBLIC_KEY) - OK

/0x00569380-0x0056997f (0x000600)/ (FS4_RSA_4096_SIGNATURES) - CRC IGNORED

/0x00569980-0x0061a1a7 (0x0b0828)/ (ROM_CODE) - OK

/0x0061a200-0x0061abab (0x0009ac)/ (DBG_FW_INI) - OK

/0x0061ac00-0x0061ac07 (0x000008)/ (DBG_FW_PARAMS) - OK

/0x0061ac80-0x00642cff (0x028080)/ (CRDUMP_MASK_DATA) - OK

-I- FW image verification succeeded. Image is bootable.

controller-0:/home/sysadmin# mstflint -d b5:00.0 q

Image type: FS4

FW Version: 22.31.1014

FW Release Date: 30.6.2021

Product Version: 22.31.1014

Rom Info: type=UEFI version=14.24.13 cpu=AMD64,AARCH64

type=PXE version=3.6.403 cpu=AMD64

Description: UID GuidsNumber

Base GUID: 0c42a1030098c870 4

Base MAC: 0c42a198c870 4

Image VSD: N/A

Device VSD: N/A

PSID: MT_0000000500

Security Attributes: secure-fw

controller-0:/home/sysadmin# mstflint -d b5:00.1 q

Image type: FS4

FW Version: 22.31.1014

FW Release Date: 30.6.2021

Product Version: 22.31.1014

Rom Info: type=UEFI version=14.24.13 cpu=AMD64,AARCH64

type=PXE version=3.6.403 cpu=AMD64

Description: UID GuidsNumber

Base GUID: 0c42a1030098c870 4

Base MAC: 0c42a198c870 4

Image VSD: N/A

Device VSD: N/A

PSID: MT_0000000500

Security Attributes: secure-fw

controller-0:/home/sysadmin# mstvpd b5:00.1

ID: ConnectX-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, with PPS In/Out, PCIe 4.0 x16, Crypto and Secure Boot

PN: MCX623106PC-CDAT

EC: A2

V2: MCX623106PC-CDAT

SN: MT2022X19516

V3: 8ae07ad3b8abea1180000c42a198c870

VA: MLX:MN=MLNX:CSKU=V2:UUID=V3:PCI=V0:MODL=CX623106P

V0: PCIeGen4 x16

controller-0:/home/sysadmin# mstvpd b5:00.0

ID: ConnectX-6 Dx EN adapter card, 100GbE, Dual-port QSFP56, with PPS In/Out, PCIe 4.0 x16, Crypto and Secure Boot

PN: MCX623106PC-CDAT

EC: A2

V2: MCX623106PC-CDAT

SN: MT2022X19516

V3: 8ae07ad3b8abea1180000c42a198c870

VA: MLX:MN=MLNX:CSKU=V2:UUID=V3:PCI=V0:MODL=CX623106P

V0: PCIeGen4 x16

Hi,

Are you able to run testpmd to using similar syntax?

Are you able to run l2fwd from DPDK and not external Nvidia version?

What happens if you remove “txq_inline_max=0”?

  1. dpdk 20.11.3 lts and dpdk 21.08 from l2fwd-nv both work with this syntax

./app/dpdk-testpmd -l 3-5,8,11,13 --main-lcore=3 -a 0000:b5:00.0,txq_inline_max=512 – --port-numa-config=0,0 --socket-num=0 --burst=64 --txd=1024 --rxd=1024 --mbcache=512 --rxq=8 --txq=8 --forward-mode=mac -i --nb-cores=4 --txonly-multi-flow --mbuf-size=2176

2 ) using dpdk-l2fwd from dpdk 20.11.3 works:

./examples/dpdk-l2fwd -l 2,6,7,9,10,12 -n 8 -a b5:00.1 – -p0x1 -q8

check traffic:

[root@cu-testpmd share]# ./rxtxbw.sh enp1f0

enp1f0: 51312 Mbps RX | 51312 Mbps TX

3 ) using l2fwd-nv Fails:

[root@cu-l2fwd lib64]# pwd

/opt/nvidia/cuBB/share/l2fwd-nv/external/dpdk/x86_64-native-linuxapp-gcc/install/lib64

[root@cu-l2fwd lib64]# export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PWD

[root@cu-l2fwd lib64]# cd /opt/nvidia/cuBB/share/l2fwd-nv

[root@cu-l2fwd l2fwd-nv]# cd build/

[root@cu-l2fwd build]# ./l2fwdnv -l 2,6,7,9,10,12 -n 8 -a b5:00.1 – -p0x1 -q8

************ L2FWD-NV ************

EAL: Detected 24 lcore(s)

EAL: Detected 1 NUMA nodes

EAL: Detected static linkage of DPDK

EAL: failed to parse device “b5:00.1”

EAL: Unable to parse device ‘b5:00.1’

EAL: Error - exiting with code: 1

Cause: Invalid EAL arguments

[root@cu-l2fwd build]#

I would suggest to open an issue on project github page regarding the failure.

OK will do. thanks.