CX-7 SR-IOV live migration: `LOAD_VHCA_STATE` fails with `syndrome 0xe9ecae`, but **only after the guest driver has initialized the VF** — same VF mig

Infrastructure & Networking Software And Drivers Virtualization For Infiniband And Ethernet
1

Summary

We are unable to live-migrate a ConnectX-7 SR-IOV VF between two identical hosts. The destination
firmware rejects the migration image that the source firmware itself produced:

mlx5_core 0000:38:00.1: mlx5_cmd_out_err:821:(pid 77520): LOAD_VHCA_STATE(0x119) op_mod(0x0)
    failed, status bad parameter(0x3), syndrome (0xe9ecae), err(-22)

We have isolated the trigger with a controlled A/B experiment (details below):

If the guest never touches the VF, migration succeeds (21 ms downtime).
As soon as the guest’s mlx5_core binds to the VF and brings the link up, the same VF,
between the same two hosts, in the same direction, fails with the syndrome above — 100% reproducible.

This is a follow-up to our earlier report which received no replies:

Since then we have upgraded the OS/kernel, upgraded the firmware to the latest GA, switched to the
inbox driver, and tested the direct-QEMU path. The syndrome is byte-for-byte identical in every case.

1. Environment — every documented requirement is met

Documented requirement Our environment OK
ConnectX-7 ETH MCX755106AS-HEA_Ax (MT2910), dual port ✅
Identical PSID on both hosts MT_0000000834 on both ✅
Identical FW, ≥ 28.41.1000 28.49.1014 on both (latest GA, June 2026) ✅
Kernel ≥ v6.7 Rocky Linux 10.2, 6.12.0-211.16.1.el10_2.0.1.x86_64 ✅
Kernel built with CONFIG_MLX5_VFIO_PCI CONFIG_MLX5_VFIO_PCI=m ✅
iproute2 ≥ 6.2 6.14 ✅
QEMU ≥ 8.1 9.1.0 ✅
libvirt ≥ 8.6 10.10.0 ✅
Ethernet mode only ETH ✅
Not MPV MULTI_PORT_VHCA_EN = False(0) ✅
≤ 4 VFs migrating in parallel 1 VF ✅

Both hosts are the same server model with the same CPU. A full mlxconfig -e q diff of the two
cards shows all 275 parameters identical.

Relevant firmware settings:

SRIOV_EN                    True(1)
NUM_OF_VFS                  127          (also tested with 16 — see §4)
MULTI_PORT_VHCA_EN          False(0)
VF_MIGRATION_MODE           MIGRATION_ENABLED(2)

Both PFs are in switchdev mode:

# devlink dev eswitch show pci/0000:38:00.1
pci/0000:38:00.1: mode switchdev inline-mode none encap-mode basic

VF is migratable and bound to the variant driver:

# devlink port show pci/0000:38:00.1/65550 | tail -1
    hw_addr 52:54:00:aa:bb:01 roce enable migratable enable max_io_eqs 24
​
# basename $(readlink /sys/bus/pci/devices/0000:38:11.6/driver)
mlx5_vfio_pci

2. The controlled A/B experiment (the key data point)

Same VF (0000:38:11.6), same PF, same host pair, same direction, same libvirt domain definition.
The only difference is whether a bootable disk is attached, i.e. whether a guest OS comes up and
initializes the VF:

Domain Guest initialized the VF? Migration result
No disk attached (BIOS stops at “no bootable device”) No ✅ succeeds, Total downtime: 21 ms
Same domain + Ubuntu 22.04 disk Yes (mlx5_core bound, link up, packets flowing) ❌ syndrome 0xe9ecae

The “guest initialized the VF” condition was verified on the host side, independently of the guest:

# ip -br link show ens1f1r13          # the VF's representor
ens1f1r13  UP  <BROADCAST,MULTICAST,UP,LOWER_UP>
​
# cat /sys/class/net/ens1f1r13/statistics/rx_packets   -> 16
# cat /sys/class/net/ens1f1r13/statistics/tx_packets   -> 45

The “no disk” case was reproduced successfully 4 times (both PFs, both directions).
The “with guest” case fails 100% of the time.

3. Exact failure

Source (libvirt path):

Migration: [ 0.00 %]error: internal error: QEMU unexpectedly closed the monitor:
qemu-kvm: error while loading state section id 52(0000:00:01.2:00.0/vfio)
qemu-kvm: load of migration failed: Invalid argument

Source (direct QEMU path, QMP):

{"return": {"status": "active"}}
{"return": {"status": "failed", "error-desc": "Sibling indicated error 1"}}

Destination dmesg (identical in both paths):

mlx5_vfio_pci 0000:38:11.6: enabling device (0000 -> 0002)
mlx5_vfio_pci 0000:38:11.6: resetting
mlx5_core 0000:38:00.1: mlx5_cmd_out_err:821:(pid 77520): LOAD_VHCA_STATE(0x119) op_mod(0x0)
    failed, status bad parameter(0x3), syndrome (0xe9ecae), err(-22)

The guest is not damaged — the migration aborts and rolls back to the source, which keeps running.

4. What we have already ruled out

Every item below was changed on both hosts and the full test re-run. In every case the result was
the same syndrome, byte for byte.

# Variable Changed from → to Result
1 OS / kernel Rocky 9.7 / 5.14 → Rocky 10.2 / 6.12 (meets the documented ≥6.7) same syndrome
2 Adapter firmware 28.47.2682 (2025 LTS U2) → 28.49.1014 (latest GA) same syndrome
3 Driver stack DOCA-Host 3.4.0 / OFED 26.04 DKMS mlx5_vfio_pci → kernel inbox mlx5_vfio_pci same syndrome
4 VF RoCE roce enable → roce disable same syndrome
5 VFIO pre-copy active → bypassed (guest suspended before migrating, so QEMU 9.1 puts the device to STOP and skips PRE_COPY) same syndrome
6 NUM_OF_VFS 127 → 16 (device default) same syndrome
7 Virtualization layer libvirt → direct QEMU (-device vfio-pci + -incoming, QMP migrate) same syndrome
8 PF PF0 / PF1 both reproduce
9 Direction A→B / B→A both reproduce

Notes on the most informative ones:

  • #3 (driver stack). The mlx5_cmd_out_err line number changes between the two drivers
    (:839 with the OFED DKMS module, :821 with the inbox module) — proving a genuinely different
    driver binary is loaded — while the firmware syndrome stays identical. We took the inbox
    modules from the distro kernel-modules / kernel-modules-core RPMs and disabled the OFED
    extra/ directory. (Mixing is not possible: the inbox mlx5-vfio-pci.ko will not load against the
    OFED mlx5_core — disagrees about version of symbol mlx5_cmd_cleanup_async_ctx.)

  • #7 (direct QEMU). We started qemu-kvm by hand on both hosts, set return-path and
    switchover-ack via QMP (both confirmed true by query-migrate-capabilities), and issued
    {"execute":"migrate", ...}. The pid in the destination dmesg line matches the PID of the
    hand-started qemu-kvm process, so libvirt is definitively not involved.

  • OVS is not involved at all. The VF’s representor is not attached to any OVS bridge
    (ovs-vsctl port-to-br ens1f1r13 → no port named ens1f1r13).

5. Why we believe this points at the firmware

LOAD_VHCA_STATE takes only three inputs (kernel v6.12, drivers/vfio/pci/mlx5/cmd.c):

MLX5_SET(load_vhca_state_in, in, vhca_id, mvdev->vhca_id);
MLX5_SET(load_vhca_state_in, in, mkey,    buf->mkey);
MLX5_SET(load_vhca_state_in, in, size,    buf->length);

  • vhca_id — read by the driver from the destination device itself.

  • mkey — freshly created by the driver for the receive buffer; a bad one would fail earlier in
    mlx5vf_dma_data_buffer().

  • size — this is the value the source firmware itself reported as actual_image_size in the
    SAVE_VHCA_STATE output. The driver writes it verbatim into the migration stream header
    (MLX5_MIGF_HEADER_TAG_FW_DATA), and the destination passes exactly that many received bytes back.
    It is far below MAX_LOAD_SIZE (32-bit field) and below MAX_CHUNK_SIZE (8 MiB).

So: the source firmware produced an N-byte image and reported N; the destination firmware — same
model, same firmware version, same PSID, identical mlxconfig — refuses to load those N bytes with
“bad parameter”. And it only refuses when the image describes a VHCA that a guest driver has
initialized.

6. Questions

  1. What does syndrome 0xe9ecae mean for LOAD_VHCA_STATE? Which specific validation failed?

  2. Given that LOAD_VHCA_STATE has only vhca_id, mkey and size — which parameter is being
    rejected    , and against what does the firmware validate it?

  3. Is there a known limitation or errata about migrating a VF after the guest driver has created
    its resources (EQs/CQs/queues/steering)? The A/B result in §2 points squarely at this.

  4. Is the OFED/DOCA-Host DKMS mlx5_vfio_pci a supported configuration for SR-IOV live
    migration, or is the inbox driver required? (We tested both; the result is identical, so this
    is a documentation question rather than a suspected cause.)

  5. Is there any firmware version — released or upcoming — in which this is fixed?

  6. Are there additional diagnostics we can collect on our side? We can run mstresourcedump /
    FUNC_SYND_REC or any other tool you suggest and attach the output.

Happy to provide the full mlxconfig -e q diff, complete dmesg, QMP transcripts, or to run any
specific reproduction you would like. The reproducer is small and self-contained: one libvirt domain
with a single <interface type='hostdev'> VF, no OpenStack and no OVS involved.