CX-7 SR-IOV Live Migration: `LOAD_VHCA_STATE` fails on destination with `bad parameter (0x3), syndrome 0xe9ecae` although all documented prerequisites

[daanwfx]

Environment

Both hosts are identical in hardware, firmware and software.

Item	Value
Hosts	2 × compute nodes (source and destination)
NIC	ConnectX-7, MCX755106AS-HEA_Ax, dual-port QSFP112, 200GbE
PSID	MT_0000000834 (identical on both hosts)
Firmware	28.47.2682 (identical on both hosts)
CPU	Intel Xeon Platinum 8457C (identical on both hosts)
OS	Rocky Linux 9.7 (Blue Onyx)
Kernel	5.14.0-611.55.1.el9_7.x86_64
DOCA-Host	doca-host-3.4.0-085000_26.04_rhel9
OFED	OFED-internal-26.04-0.8.6
mlx5_core	/lib/modules/5.14.0-611.55.1.el9_7.x86_64/extra/mlx5_core.ko.xz, version 26.04-0.8.6
mlx5_vfio_pci	/lib/modules/5.14.0-611.55.1.el9_7.x86_64/extra/mlx5-vfio-pci.ko.xz
QEMU	9.1.0
libvirt	10.10.0
iproute2	6.14.0
eswitch mode	switchdev on both PFs (pci/0000:38:00.0, pci/0000:38:00.1)
VFs	16 per PF, NUM_OF_VFS=127, SRIOV_EN=True(1)
VF_MIGRATION_MODE	MIGRATION_ENABLED(2) on both hosts (set via mlxconfig, host rebooted afterwards)
MULTI_PORT_VHCA_EN	False(0) — VFs are regular single-port VFs, not MPV
Orchestration	OpenStack 2025.1 (nova 31.1.1) driving libvirt-managed live migration

Note on the driver source: the mlx5 stack on these hosts is not the inbox kernel driver. Both
mlx5_core and mlx5_vfio_pci are provided by DOCA-Host / OFED 26.04 and live in
/lib/modules/$(uname -r)/extra/. The kernel tree contains no inbox mlx5 VFIO module
(/lib/modules/$(uname -r)/kernel/drivers/vfio/pci/mlx5/ does not exist).

Problem Summary

A live migration of a QEMU/KVM guest with one ConnectX-7 VF assigned via
<interface type='hostdev'> fails on the destination host. The VFIO migration protocol is
actually exercised — the device state is transferred and the destination attempts to load it — but
the firmware rejects the load.

The guest is unharmed; the migration aborts and rolls back cleanly to the source.

Exact Error

Destination host, kernel log:

mlx5_vfio_pci 0000:38:01.6: enabling device (0000 -> 0002)
mlx5_core 0000:38:00.0: mlx5_cmd_out_err:839:(pid 186047): LOAD_VHCA_STATE(0x119) op_mod(0x0) failed,
    status bad parameter(0x3), syndrome (0xe9ecae), err(-22)

Destination host, QEMU:

qemu-kvm: error while loading state section id 105(0000:00:02.4:00.0/vfio)
qemu-kvm: load of migration failed: Invalid argument

Source host, libvirt:

operation failed: job 'migration out' failed: Sibling indicated error 1

This is 100% reproducible — reproduced on every attempt (7 attempts across two different guests).

Prerequisites — all verified present

Every item in the SR-IOV Live Migration setup procedure (DOCA 3.4 documentation) has been
performed and individually verified on the live system:

Documented requirement	Verified state
mlxconfig ... VF_MIGRATION_MODE=2 + reboot	VF_MIGRATION_MODE = MIGRATION_ENABLED(2) on both hosts. Before the reboot devlink port show reported no migratable attribute at all; after the reboot the attribute appears on all 32 VF ports, confirming the firmware setting took effect
PF in switchdev mode	pci/0000:38:00.0: mode switchdev inline-mode none encap-mode basic (both PFs, both hosts)
VFs created	16 VFs per PF
VF unbound from mlx5_core before setting migratable	Done. Note: setting migratable on a bound VF fails with Failed setting HCA migratable cap. kernel answers: Invalid argument; unbind → set → rebind works and the setting persists across the rebind
devlink port function set ... migratable enable	Confirmed "migratable": "enable" on the specific source VF used by the guest and on every free VF on the destination host
VF bound to mlx5_vfio_pci (not generic vfio-pci)	Confirmed on both ends via kernel log (mlx5_vfio_pci 0000:38:01.x: enabling device) and /sys/bus/pci/devices/<VF>/driver. libvirt selects the variant driver automatically with managed='yes'
Same PSID and firmware on both hosts	MT_0000000834 / 28.47.2682
migrate_set_capability return-path on / switchover-ack on	Confirmed enabled by libvirt on both source and destination. Captured from the QMP traffic: {"capability":"switchover-ack","state":true} is sent at migration start on both ends and reset to false only after the migration aborts
Destination VF administrative MAC	Confirmed correct during the migration. Sampled devlink port show at 200 ms intervals on the destination while migrating: the claimed VF carries the guest’s MAC (fa:16:3e:6b:e4:1e, matching the source VF) at the time the state is loaded. It is reverted to a random address only after the rollback
≤ 4 parallel VF migrations	Only one VF is being migrated
MPV restriction (“MPV VFs traffic cannot be configured with OVS”)	Does not apply — MULTI_PORT_VHCA_EN = False(0), these are regular single-port VFs

Ruled out by experiment

Hypothesis	Result
switchover-ack / return-path not enabled	Ruled out — QMP capture shows both true on both ends
Destination VF MAC not programmed	Ruled out — live sampling during migration shows the correct MAC
Destination VF not migration-capable	Ruled out — after explicitly enabling migratable on all free destination VFs the same error still occurs
Generic vfio-pci bound instead of the variant driver	Ruled out — kernel log shows mlx5_vfio_pci on both ends
Guest has a mixed NIC configuration	Ruled out — a guest with a single interface, the VFIO one and nothing else, fails identically
Hardware / firmware mismatch between hosts	Ruled out — identical part number, PSID and firmware
eswitch in legacy mode	Ruled out — both PFs are in switchdev, as the documentation requires

Reproduction

1. Two identical CX-7 hosts as described above, PFs in switchdev, VF_MIGRATION_MODE=2, rebooted.
2. On both hosts, for the VFs to be used: unbind from mlx5_core,
   devlink port function set pci/0000:38:00.0/<port> migratable enable, rebind.
3. Start a guest on the source with one VF assigned as
   <interface type='hostdev' managed='yes'> with a <mac> element.
4. Live migrate the guest to the destination (libvirt-managed, peer-to-peer, not tunnelled).
5. The migration aborts; the destination kernel log shows the LOAD_VHCA_STATE failure above.

The VF representor is attached to an OVS bridge (OVN-managed, VLAN access port with offloaded
flows) on both hosts. We could not find any documented restriction against this for non-MPV VFs,
but it is the one aspect of our setup that differs from the plain example in the documentation.

Questions

1. What does LOAD_VHCA_STATE syndrome 0xe9ecae (status bad parameter (0x3)) indicate?
   This is the single most useful piece of information — it should point directly at whichever
   parameter or device-state field the firmware is rejecting.

2. Is SR-IOV Live Migration supported when the mlx5 stack is supplied by DOCA-Host / OFED 26.04
   on a RHEL/Rocky 9 kernel (5.14)? The documented requirement is “Kernel Linux v6.7 or newer”,
   which we read as applying to the inbox driver. The documentation does not state what the
   requirement is when DOCA-Host provides mlx5_core and mlx5_vfio_pci as out-of-tree modules,
   which is our case. Note that RHEL/Rocky 9 cannot reach kernel 6.7 at all — the newest kernel in
   the Rocky 9 repositories is 5.14.0-687.42.1.el9_8 — so if a ≥6.7 kernel is genuinely required,
   this feature is unavailable on RHEL 9 derivatives regardless of DOCA-Host, and we would like that
   confirmed explicitly.

3. Is there a supported firmware ↔ driver compatibility matrix for this feature?
   We run firmware 28.47.2682 with OFED 26.04-0.8.6. The documentation only states a firmware
   floor (28.41.1000).

4. Is any per-VF configuration required beyond migratable enable? In particular, must RoCE be
   disabled on the VF before migration? Our VFs report "roce": "enable" alongside
   "migratable": "enable", and the documentation does not mention the interaction.

5. Does attaching the VF representor to an OVS bridge with offloaded flows affect VHCA state
   migration for non-MPV VFs? The documentation only restricts this for MPV VFs.

Data available on request

• Full destination dmesg and /var/log/libvirt/libvirtd.log around the failure
• Full QMP trace of the migration (libvirt qemu_monitor debug log, both ends)
• mlxconfig -d <PF> q and devlink port show -jp output from both hosts
• Guest domain XML (source) and the destination XML generated for the migration
• mstflint/mlxlink output if required