DPDK APIs rte_eth_dev_count_avail() returns 0 with the MLX5 NIC card

vikram.thirthappa · October 17, 2019, 3:07pm

Hi

I am trying to bind DPDK with the MLX(mlx5_core). On running the application which uses DPDK API rte_eth_dev_count_avail() returns 0 on the MLX5 NIC card.

dpdk-devbind.py -s says that the

interface is bound to mlx5_core( and not binding to igb_uio)

1.RDMA packages was installed from Oracle Linux - https://docs.oracle.com/cd/E93554_01/E95779/html/uek5_install_ofed.html

MFT tools were installed from Mellanox.

3.DPDK version - 18.11 compiled with CONFIG_RTE_LIBRTE_MLX5_PMD=y

Below are the commands:

# lsmod | grep mlx5

mlx5_ib 196608 0

ib_core 233472 11 rdma_cm,ib_ipoib,resilient_rdmaip,rds_rdma,iw_cm,ib_umad,i40iw,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm

mlx5_core 507904 1 mlx5_ib

mlxfw 20480 1 mlx5_core

ptp 20480 3 i40e,tg3,mlx5_core

# lsmod | grep ib_

ib_umad 24576 0

ib_ipoib 135168 1 resilient_rdmaip

ib_uverbs 110592 1 rdma_ucm

ib_cm 69632 3 rdma_cm,ib_ipoib,rds_rdma

ib_core 233472 11 rdma_cm,ib_ipoib,resilient_rdmaip,rds_rdma,iw_cm,ib_umad,i40iw,rdma_ucm**,ib_uverbs**,mlx5_ib,ib_cm

#mst status

MST modules:

MST PCI module is not

loaded

MST PCI configuration

module loaded

MST devices:

/dev/mst/mt4115_pciconf0 -

PCI configuration cycles access.

domain:bus:dev.fn=0000:08:00.0

addr.reg=88 data.reg=92 cr_bar.gw_offset=-1

Chip

revision is: 00

# ibv_devinfo

hca_id: mlx5_0

transport: InfiniBand

(0)

fw_ver: 12.25.1020

node_guid: 506b:4b03:00db:fe0a

sys_image_guid: 506b:4b03:00db:fe0a

vendor_id: 0x02c9

vendor_part_id: 4115

hw_ver: 0x0

board_id: MT_2150110033

phys_port_cnt: 1

port: 1

state: PORT_ACTIVE

(4)

max_mtu: 4096

(5)

active_mtu: 1024

(3)

sm_lid: 0

port_lid: 0

port_lmc: 0x00

link_layer: Ethernet

hca_id: mlx5_1

transport: InfiniBand

(0)

fw_ver: 12.25.1020

node_guid: 506b:4b03:00db:fe0b

sys_image_guid: 506b:4b03:00db:fe0a

vendor_id: 0x02c9

vendor_part_id: 4115

hw_ver: 0x0

board_id: MT_2150110033

phys_port_cnt: 1

port: 1

state: PORT_DOWN

(1)

max_mtu: 4096

(5)

active_mtu: 1024

(3)

sm_lid: 0

port_lid: 0

port_lmc: 0x00

link_layer: Ethernet

# mlxconfig -d /dev/mst/mt4115_pciconf0 q

Output attached

#

/var/cache/ocsm/dpdk/dpdk-18.11/usertools/dpdk-devbind.py -s

Network devices using kernel

driver

===================================

0000:02:00.0 'NetXtreme BCM5720

Gigabit Ethernet PCIe 165f’ if=em1 drv=tg3 unused=igb_uio Active

0000:02:00.1 'NetXtreme BCM5720

Gigabit Ethernet PCIe 165f’ if=em2 drv=tg3 unused=igb_uio

0000:08:00.0 'MT27700 Family [ConnectX-4]

1013’ if=p2p1 drv=mlx5_core unused=igb_uio

0000:08:00.1 'MT27700 Family

[ConnectX-4] 1013’ if=p2p2 drv=mlx5_core unused=igb_uio

0000:41:00.0 'Ethernet Controller

X710 for 10GbE SFP+ 1572’ if=p1p1 drv=i40e unused=igb_uio

0000:41:00.1 'Ethernet Controller

X710 for 10GbE SFP+ 1572’ if=p1p2 drv=i40e unused=igb_uio

However testpmd works fine here;

# ./testpmd -l 8-15 -n 4 -w 08:00.0 -w 08:00.1 – --rxq=2 --txq=2 -i

EAL: Detected 32 lcore(s)

EAL: Detected 2 NUMA nodes

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket

EAL: Probing VFIO support…

EAL: VFIO support initialized

EAL: PCI device 0000:08:00.0 on NUMA socket 0

EAL: probe driver: 15b3:1013 net_mlx5

EAL: PCI device 0000:08:00.1 on NUMA socket 0

EAL: probe driver: 15b3:1013 net_mlx5

Interactive-mode selected

testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=203456, size=2176, socket=0

testpmd: preferred mempool ops selected: ring_mp_mc

testpmd: create a new mbuf pool <mbuf_pool_socket_1>: n=203456, size=2176, socket=1

testpmd: preferred mempool ops selected: ring_mp_mc

Configuring Port 0 (socket 0)

Port 0: 50:6B:4B:DB:FE:0A

Configuring Port 1 (socket 0)

Port 1: 50:6B:4B:DB:FE:0B

Checking link statuses…

Done

Is anything missing for DPDK APIs rte_eth_dev_count_avail() not working here which is blocking DPDK application.

Guidance to resolve from this blocking issue is much appreciated.

Thanks in advance

Output_mlxconfig.txt (8.04 KB)

alekseys1 · October 17, 2019, 4:08pm

Hi Vikram,

dpdk-devbind.py is not applicable to Mellanox devices and should not be used. When running for example testpmd application, you just need to specify device using ‘-w’ option

vikram.thirthappa · October 17, 2019, 5:54pm

Hi Aleksey,

dpdk-devbind.py is not been used, and it is just shown as to clarify that igb_uio is not been bound and ml5_core is used for MLX5 NIC. Used driver is mlx5_core and compiled DPDK with - CONFIG_RTE_LIBRTE_MLX5_PMD=y. The issue here is that rte_eth_dev_count_avail returns 0.

While it should return The count of available Ethernet devices for the application to use.

alekseys1 · October 18, 2019, 4:56pm

Hi VIkram,

Using Mellanox OFED 4.7 and DPDK-18.11.2 the issue is not reproducible on my setup and everything works as expected. See the part of the ‘ethtool’ example application and the real-time prints. The problem most likely related to your setup or software. Try to use latest Mellanox OFED and DPDK LTS.

int main(int argc, char **argv)

242 {

243 int cnt_args_parsed;

244 uint32_t id_core;

245 uint32_t cnt_ports;

246

247 /* Init runtime environment */

248 cnt_args_parsed = rte_eal_init(argc, argv);

249 if (cnt_args_parsed < 0)

250 rte_exit(EXIT_FAILURE, “rte_eal_init(): Failed”);

251

252 cnt_ports = rte_eth_dev_count_avail();

253 printf(“Number of NICs: %i\n”, cnt_ports);

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

~/workspace/dpdk/dpdk-stable-18.11.2/examples/ethtool/ethtool-app/x86_64-native-linuxapp-gcc/>sudo ./ethtool

EAL: Detected 12 lcore(s)

EAL: Detected 2 NUMA nodes

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket

EAL: Probing VFIO support…

EAL: PCI device 0000:01:00.0 on NUMA socket 0

EAL: probe driver: 8086:1521 net_e1000_igb

EAL: PCI device 0000:01:00.1 on NUMA socket 0

EAL: probe driver: 8086:1521 net_e1000_igb

EAL: PCI device 0000:81:00.0 on NUMA socket 1

EAL: probe driver: 15b3:1019 net_mlx5

EAL: PCI device 0000:81:00.1 on NUMA socket 1

EAL: probe driver: 15b3:1019 net_mlx5

Number of NICs: 2

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Init port 0…

Init port 1…

EthApp>

vikram.thirthappa · November 26, 2019, 7:07am

Hi Aleksey,

I installed Mellanox OFED 4.7 and compiling DPDK 18.11 fails with incompatible libibverbs version.(Attached compilation errors)

However when I compiled DPDK 16.11.11 (LTS) with Mellanox OFED 4.7 the compilation was successfull.

We need to know if the DPDK 18.11 is compatible with the MLNX OFED 4.7 and how the DPDK compilation was successful for you.

Note - CONFIG_RTE_LIBRTE_MLX5_PMD=y is added during compilation.

Thanks in advance

Regards,

Vikram

alekseys1 · November 26, 2019, 5:30pm

Have you used ‘–upstream-libs’ parameter when installing Mellanox OFED?

vikram.thirthappa · November 27, 2019, 11:42am

Thanks Aleksey,

After using the options ‘–upstream-libs’ DPDK-18.11 was successfully compiled.

Regards

Vikram

vikram.thirthappa · November 29, 2019, 4:49am

Hi Aleksey,

We have an issue with enabling the Promiscuous mode when the application starts. Error Snippet below:

Nov 27 14:05:17 OCOM-PROBE-2 systemd: Starting OCSM Media Sniffer…

Nov 27 14:05:17 OCOM-PROBE-2 rat: [UNDEF_000] Calling rte_eal_init.

Nov 27 14:05:17 OCOM-PROBE-2 rat_dpdk_sniffer[23178]: EAL: Probing VFIO support…

Nov 27 14:05:17 OCOM-PROBE-2 rat_dpdk_sniffer[23178]: EAL: VFIO support initialized

Nov 27 14:05:17 OCOM-PROBE-2 rat_dpdk_sniffer[23178]: EAL: PCI device 0000:08:00.0 on NUMA socket 0

Nov 27 14:05:17 OCOM-PROBE-2 rat_dpdk_sniffer[23178]: EAL: probe driver: 15b3:1013 net_mlx5

Nov 27 14:05:17 OCOM-PROBE-2 rat_dpdk_sniffer[23178]: net_mlx5: MPLS over GRE/UDP tunnel offloading disabled due to old OFED/rdma-core version or firmware configuration

Nov 27 14:05:18 OCOM-PROBE-2 rat_dpdk_sniffer[23178]: EAL: PCI device 0000:08:00.1 on NUMA socket 0

Nov 27 14:05:18 OCOM-PROBE-2 rat_dpdk_sniffer[23178]: EAL: probe driver: 15b3:1013 net_mlx5

Nov 27 14:05:18 OCOM-PROBE-2 rat_dpdk_sniffer[23178]: net_mlx5: MPLS over GRE/UDP tunnel offloading disabled due to old OFED/rdma-core version or firmware configuration

Nov 27 14:05:18 OCOM-PROBE-2 rat_dpdk_sniffer[23178]: rte_eal for MLX successfully initialized.

Nov 27 14:05:20 OCOM-PROBE-2 systemd: Started OCSM Media Sniffer.

Nov 27 14:05:20 OCOM-PROBE-2 rat_dpdk_sniffer[23178]: net_mlx5: port 1 cannot enable promiscuous mode: Operation not permitted

Nov 27 14:05:20 OCOM-PROBE-2 rat_dpdk_sniffer[23178]: net_mlx5: port 0 cannot enable promiscuous mode: Operation not permitted

However the Application code works with the Intel NICs , but have this privileges issue only with NIC Cards.

However the test-pmd works.

Is there something that is missing.

Is there some configuration to address this issue.

Regards

Vikram

vikram.thirthappa · November 29, 2019, 10:22am

Additionally, I learnt that the test-pmd has run with the root user. However the application has run with non-root user.