Issue with Mellanox ConnectX-5 and VFIO-PCI Binding in DPDK — “mlx5_common: Verbs device not found” Error

Infrastructure & Networking Software And Drivers Mellanox OFED
,
1

Hello,

I am using a Mellanox ConnectX-5 MCX516A-CCAT card and trying to achieve 100G Ethernet data transmission between a client PC and an SoC board.

I have installed the Mellanox drivers and DPDK, and I intend to use DPDK with VFIO-PCI for data transmission.
When I test with mlx5_core driver using testpmd, data transmission works fine. However, when switching to VFIO-PCI, despite the device showing as bound, I get an error that the device cannot be found.

Below is a test C program I wrote:

c

복사

#include <stdio.h>
#include <stdint.h>
#include <inttypes.h>
#include <rte_eal.h>
#include <rte_ethdev.h>
#include <rte_mbuf.h>
#include <rte_ether.h>
#include <string.h>

#define RX_RING_SIZE 1024
#define NUM_MBUFS 8191
#define MBUF_CACHE_SIZE 250
#define BURST_SIZE 32

static const struct rte_ether_addr client_mac = {
    .addr_bytes = {0xb8, 0xce, 0xf6, 0x76, 0x11, 0x6f}
};

static const struct rte_ether_addr rf_soc_mac = {
    .addr_bytes = {0xfc, 0xc2, 0x3d, 0x5d, 0x05, 0x9c}
};

int main(int argc, char *argv[]) {
    struct rte_mempool *mbuf_pool;
    uint16_t port_id = 0;
    int ret;
    uint16_t nb_rxd = RX_RING_SIZE;

    ret = rte_eal_init(argc, argv);
    if (ret < 0) rte_exit(EXIT_FAILURE, "Error with EAL initialization\n");
    argc -= ret; argv += ret;

    mbuf_pool = rte_pktmbuf_pool_create("MBUF_POOL", NUM_MBUFS * rte_eth_dev_count_avail(),
        MBUF_CACHE_SIZE, 0, RTE_MBUF_DEFAULT_BUF_SIZE, rte_socket_id());
    if (mbuf_pool == NULL) rte_exit(EXIT_FAILURE, "Cannot create mbuf pool\n");

    struct rte_eth_conf port_conf = {0};
    port_conf.rxmode.max_lro_pkt_size = RTE_ETHER_MAX_LEN;

    ret = rte_eth_dev_configure(port_id, 1, 1, &port_conf);
    if (ret < 0) rte_exit(EXIT_FAILURE, "Cannot configure device: err=%d, port=%u\n", ret, port_id);

    ret = rte_eth_rx_queue_setup(port_id, 0, nb_rxd, rte_eth_dev_socket_id(port_id), NULL, mbuf_pool);
    if (ret < 0) rte_exit(EXIT_FAILURE, "rte_eth_rx_queue_setup:err=%d, port=%u\n", ret, port_id);

    ret = rte_eth_tx_queue_setup(port_id, 0, 512, rte_eth_dev_socket_id(port_id), NULL);
    if (ret < 0) rte_exit(EXIT_FAILURE, "rte_eth_tx_queue_setup:err=%d, port=%u\n", ret, port_id);

    ret = rte_eth_dev_start(port_id);
    if (ret < 0) rte_exit(EXIT_FAILURE, "rte_eth_dev_start:err=%d, port=%u\n", ret, port_id);

    printf("Started port %u\n", port_id);

    struct rte_mbuf *bufs[BURST_SIZE];
    while (1) {
        const uint16_t nb_rx = rte_eth_rx_burst(port_id, 0, bufs, BURST_SIZE);
        if (nb_rx == 0) continue;

        for (int i = 0; i < nb_rx; i++) {
            struct rte_mbuf *mbuf = bufs[i];
            struct rte_ether_hdr *eth_hdr = rte_pktmbuf_mtod(mbuf, struct rte_ether_hdr *);

            if (rte_is_same_ether_addr(&eth_hdr->dst_addr, &client_mac) &&
                rte_is_same_ether_addr(&eth_hdr->src_addr, &rf_soc_mac)) {

                printf("Received matching frame, length: %u bytes\n", rte_pktmbuf_pkt_len(mbuf));
            } else {
                printf("Received frame but MAC not matched\n");
            }

            rte_pktmbuf_free(mbuf);
        }
    }

    return 0;
}

I compiled and executed it using:

bash

복사

gcc -o dpdk_rx_example dpdk_rx_example.c $(pkg-config --cflags --libs libdpdk)
sudo ./dpdk_rx_example -l 0-3 -n 4 -a 0000:0f:00.1

But I continuously receive the following error:

vbnet

복사

mlx5_common: Verbs device not found: 0000:0f:00.1
mlx5_common: Failed to initialize device context.
EAL: Requested device 0000:0f:00.1 cannot be used
EAL: Bus (pci) probe failed.
EAL: Error - exiting with code: 1
Cause: Cannot create mbuf pool

I have confirmed that the device is properly bound to vfio-pci:

csharp

복사

Network devices using DPDK-compatible driver
============================================
0000:0f:00.0 'MT27800 Family [ConnectX-5] 1017' drv=vfio-pci unused=mlx5_core
0000:0f:00.1 'MT27800 Family [ConnectX-5] 1017' drv=vfio-pci unused=mlx5_core

Is Mellanox ConnectX-5 truly compatible with VFIO-PCI in DPDK?
What could be the cause of this issue and how can it be resolved?

Thank you for your assistance.

2

Hello @ginalee04211,

Thank you for posting your query on our community.

The error happens because the ConnectX-5 NIC’s mlx5 DPDK driver needs the kernel Verbs interface to communicate with the device. This interface is provided only when the NIC is bound to the mlx5_core driver. When you bind the device to vfio-pci, the Verbs interface is missing, so the mlx5 PMD cannot initialize and shows this error. This is a known behavior, and the easiest way to run your DPDK app is to keep the device bound to mlx5_core.

For more info on how to run DPDK on Mellanox NIC, you can check the link - https://doc.dpdk.org/guides/nics/mlx5.html.

Thanks,
Bhargavi