Simplest vDPA setup with ConnectX-6 Dx

Infrastructure & Networking Software And Drivers Virtualization For Infiniband And Ethernet
1

I have a dual-port ConnectX-6 Dx card in the latest Fedora 41. I wish to use it such that every VM gets its own Virtual Function in vDPA mode. At this moment I don’t want to setup OVS because the VMs don’t need to talk to each other.

I can’t figure out how to setup this simple scenario.

After boot, lspci shows:
0008:00:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
0008:00:00.1 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]

0008:00:00.0 corresponds to mlx5_2, so I do this to create a VF:
echo 1 > /sys/class/infiniband/mlx5_2/device/sriov_numvfs
and now see a new PCI deice:
0008:00:00.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function

I think I now need vhost-vdpa loaded for this VF, but can’t find how to do that… Am I even on the right track?

2

Hi,

First, use the following command to load vhost-vdpa on the host:
modprobe vhost_vdpa

Next, both NIC ports (physical, not the VF) need to be set to switchdev mode, using the following commands:
devlink dev eswitch set pci/0008:00:00.0 mode switchdev
devlink dev eswitch set pci/0008:00:00.1 mode switchdev

Now, bind the VF to the vDPA driver (part of the mlx5):
echo 0008:00:00.2 > /sys/bus/pci/drivers/mlx5_core/bind

You can also use the vdpa command for this:
vdpa dev add name vdpa0 mgmtdev <device_name>

The final step would be assigning the VF to a VM (using virtio commands or virt-manager if KVM, whichever).

Please let me know if this works.
If you still encounter issues, please open a case with enterprisesupport@nvidia.com and it will be handled based on entitlement.

Thanks,
Jonathan.