ConnectX-7 SR-IOV config for splitting one PF across many VMs

Infrastructure & Networking Adapters and Cables Ethernet Adapter Cards
,
1

Hello

We are running a benchmark that multiplexes a single ConnectX-7 across a number of KVM guests using SR-IOV, with each VF passed through to a guest via VFIO. We fix the total core budget at 32 (the maximum per socket on our setup) and vary the number of VMs from 1 to 16, dividing the cores equally (e.g., assigning 32/N vCPUs) and using a dedicated SR-IOV Virtual Function for each VM. We run one iperf network flow on each core, so 32 aggregated network flows over N VMs. We report aggregate throughput across all VMs. We have been seeing some serious total throughput degradation as the number of VFs (and therefore guests) increases, specifically the RX throughput (when VMs run as receiver). In the RX case, for a single guest the throughput is 400 Gb/s. As we double the guest count (which requires more VFs to be created), we see the RX throughput drop by around half (400 → 200 → 100 → 50 → ect.). Before we draw any conclusions from our measurements, we would like a sanity check on whether our NIC-side configuration is the intended way to do this.

We are separately auditing our host CPU/NUMA/VM sizing, so this question is only about the NIC and eswitch configuration.

Environment
NIC ConnectX-7 MCX75310AAS-NEA (15b3:1021), single-port 400GbE OSFP, PCIe 5.0 x16, PF at 0000:98:00.0 (netdev ens1np0)
Firmware 28.46.1006 (MT_0000000838)
Host OS / kernel Ubuntu 22.04, vanilla Linux 6.12.9
Driver In-tree mlx5_core for both host and guest — not the DOCA/OFED. MFT 4.30.1 for management.
Hypervisor QEMU 10.0 + libvirt, VFIO passthrough via IOMMUFD
Host IOMMU intel_iommu=on,sm_on iommu.strict=1 (always enabled — required for VFIO passthrough)
VFs One VF per guest. sriov_totalvfs = 16
Guest Kernel vanilla 6.12.9 with intel_iommu=off

Host Config

  1. Make sure the PF is bound to mlx5_core (it is not left under vfio-pci).

  2. Destroy any existing VFs, then switch the eswitch to switchdev:

echo 0 > /sys/bus/pci/devices/0000:98:00.0/sriov_numvfs
devlink dev eswitch set pci/0000:98:00.0 mode switchdev
  1. Create N VFs:
echo N > /sys/bus/pci/devices/0000:98:00.0/sriov_numvfs
  1. Bind every VF to vfio-pci for passthrough (PF stays on mlx5_core). Each VF’s BDF is
    resolved from /sys/bus/pci/devices/0000:98:00.0/virtfn<i>, since with more than 7 VFs
    ARI numbering places them at 98:01.0, 98:01.1, … rather than continuing 98:00.x:
echo 1 > /sys/module/vfio_pci/parameters/enable_sriov
echo <vf-bdf> > /sys/bus/pci/drivers/mlx5_core/unbind
echo vfio-pci > /sys/bus/pci/devices/<vf-bdf>/driver_override
echo <vf-bdf> > /sys/bus/pci/drivers_probe
  1. Give the VFs external connectivity by putting the PF uplink and all VF representors
    (pf0vf0 … pf0vf<N-1>) into a Linux bridge, moving the PF’s IP onto the bridge:
ip link add name br-data type bridge
ip link set br-data mtu $PF_MTU
ip link set ens1np0 master br-data
ip link set <repN> mtu $PF_MTU
ip link set <repN> master br-data
ip link set <repN> up
ip link set br-data up

No VF rate limits, VF MAC assignments, VLANs, or QoS settings are applied. MTU is 4000 end to end.

Guest Config

Each guest gets exactly one VF, passed through with IOMMUFD:

-object iommufd,id=iommufd0
-device intel-iommu,intremap=on,caching-mode=on,x-scalable-mode=on,x-flts=on
-device vfio-pci,host=<vf-bdf>,id=hostdev0,iommufd=iommufd0

Inside the guest the VF comes up under mlx5_core as a normal netdev with a static IP in the
same subnet as the PF uplink.

Questions

  1. Expected per-VF scaling. For a single-port ConnectX-7 with N VFs each driven by an independent guest, are there known per-VF resource limits (steering/flow-table entries, completion queues, cache/context resources) that would cause aggregate throughput to fall as N grows, independent of host CPU?

  2. switchdev vs. legacy. For pure VF passthrough to VMs, with no OVS/tc offload rules and no SmartNIC/DPU involvement, is switchdev the right eswitch mode, or should we simply stay in legacy mode?

  3. VF driver choice. Is there any functional or performance difference between binding VFs to vfio-pci versus mlx5_vfio_pci for this use case? We currently use vfio-pci.

  4. Firmware-level settings. Beyond SRIOV_EN=1 and NUM_OF_VFS=N, are there mlxconfig parameters we should be setting when a single PF is shared by many VFs? For example anything affecting per-VF resources, steering entries, or BAR sizing? Should NUM_OF_VFS be set to exactly the number we use, or is leaving it at the maximum harmless?

Happy to attach mlxconfig -d <dev> q, ethtool -i, devlink dev eswitch show, devlink resource show, and dmesg output if that helps. Thanks in advance.

2

1. switchdev vs legacy — use legacy. switchdev creates VF representors so a

software vSwitch can install hardware offload rules. You have put the

representors into a Linux bridge with no offload rules, so all VF traffic is

being forwarded in software on the host CPU, which degrades as you add VFs. In

legacy mode the embedded eSwitch forwards in hardware. Remove the bridge, put

the IP back on the PF, and run:

devlink dev eswitch set pci/0000:98:00.0 mode legacy

2. vfio-pci vs mlx5_vfio_pci — no data-path difference. mlx5_vfio_pci only adds

VF live-migration support.

3. mlxconfig — SRIOV_EN=1 and NUM_OF_VFS= are all you need; leave the rest at

default. Set NUM_OF_VFS to what you actually use, as each VF reserves firmware

resources. Requires a reboot.

4. Per-VF scaling — ConnectX supports up to 127 VFs per port, so 16 is well

within range and we would not expect NIC-side limits at that scale. Please

re-test in legacy mode first. Detailed sizing and scaling analysis is outside

what we can cover here — please contact your NVIDIA sales representative for

design services.

3

Hello,

We tried using legacy mode and applying the recommended options. We are still seeing the mentioned RX problems, and we have very high utilization (most cores are at 100%, some are at ~90%). Are there any steps we can take to debug or correct this behavior?

Thanks