Simplest vDPA setup with ConnectX-6 Dx

Infrastructure & Networking Software And Drivers Virtualization For Infiniband And Ethernet
1

I have a dual-port ConnectX-6 Dx card in the latest Fedora 41. I wish to use it such that every VM gets its own Virtual Function in vDPA mode. At this moment I don’t want to setup OVS because the VMs don’t need to talk to each other.

I can’t figure out how to setup this simple scenario.

After boot, lspci shows:
0008:00:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
0008:00:00.1 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]

0008:00:00.0 corresponds to mlx5_2, so I do this to create a VF:
echo 1 > /sys/class/infiniband/mlx5_2/device/sriov_numvfs
and now see a new PCI deice:
0008:00:00.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function

I think I now need vhost-vdpa loaded for this VF, but can’t find how to do that… Am I even on the right track?

2

Hi,

First, use the following command to load vhost-vdpa on the host:
modprobe vhost_vdpa

Next, both NIC ports (physical, not the VF) need to be set to switchdev mode, using the following commands:
devlink dev eswitch set pci/0008:00:00.0 mode switchdev
devlink dev eswitch set pci/0008:00:00.1 mode switchdev

Now, bind the VF to the vDPA driver (part of the mlx5):
echo 0008:00:00.2 > /sys/bus/pci/drivers/mlx5_core/bind

You can also use the vdpa command for this:
vdpa dev add name vdpa0 mgmtdev <device_name>

The final step would be assigning the VF to a VM (using virtio commands or virt-manager if KVM, whichever).

Please let me know if this works.
If you still encounter issues, please open a case with enterprisesupport@nvidia.com and it will be handled based on entitlement.

Thanks,
Jonathan.

3

The 2 ports of my dual-port ConnectX-6 Dx card are connected to each other. My initial goal is to have 2 VMs communicating through this cable, each using a virtio net device and vDPA on the host, so that I can test vDPA.

I’ve checked the cable by isolating the 2 PFs into separate network namespaces on the host, manually assigning IP addresses and pinging between them. (When there is just a single interface in the namespace, the responses are returned to the same interface where requests came from.)

This also works for VFs, so I’ve concluded that the cable is good.

Then I’ve configured vDPA following your directions (details below), apparently successfully. VMs run and have new virtio net devices.

However, I can’t ping between them. The ‘ifstat’ command in the VMs shows Tx packet count increased by pings, but the Rx count is always 0.

It looks like the ping packets never reach the other side. Maybe this has to do with the “switchdev” mode? (If I turn it on before the “cable test” described above, then it also doesn’t work.)

On the host

systemctl stop firewalld
modprobe vhost-vdpa

devlink dev eswitch set pci/0008:00:00.0 mode switchdev
devlink dev eswitch set pci/0008:00:00.1 mode switchdev

echo 1 > /sys/class/net/ens96f0np0/device/sriov_numvfs
echo 1 > /sys/class/net/ens112f1np1/device/sriov_numvfs
sleep 1

vdpa dev add name vf00.2 mgmtdev pci/0008:00:00.2 mac 52:54:00:17:3a:61
vdpa dev add name vf10.1 mgmtdev pci/0008:00:10.1 mac 52:54:00:17:3a:62

ls -l /dev/vhost-vdpa*
crw-------. 1 root root 248, 0 Jan 7 10:11 /dev/vhost-vdpa-0
crw-------. 1 root root 248, 1 Jan 7 10:11 /dev/vhost-vdpa-1

Each VM has the following “interface” section added to the libvirt XML (I had to remove formatting, otherwise this site doesn’t show it):
interface type=“vdpa”
source dev=“/dev/vhost-vdpa-0”
mac address=‘52:54:00:17:3a:61’
model type=“virtio”
address type=“pci”
(same for the other VM, except "vhost-vdpa-1” and ‘52:54:00:17:3a:62’)

On the VMs

Each shows a new “eno1” network interface, and the MAC addresses are those specified above.

Then, as I’ve done on the host while testing the cable, put each “eno1” into a separate namespace:

ip netns add vdpa
ip link set eno1 netns vdpa
ip netns exec vdpa ip link set dev eno1 up
ip netns exec vdpa ip addr add 10.10.10.10/24 dev eno1
(same on the other VM, except 10.10.10.20)

ip netns exec vdpa ping -c2 10.10.10.20
and the other VM doesn’t see any packets.

switchdev vs legacy mode
If I omit the devlink commands switching to the switchdev mode, the packets still don’t go through. (I’ve verified that the NICs stay in “legacy” mode, via the “devlink dev eswitch show” command. Also, “port representor” interfaces then don’t show up).