BlueField-3 SR-IOV: RDMA fails when 2 VFs from the same PF are simultaneously active (legacy eSwitch, NIC mode, RoCE)

Infrastructure & Networking BlueField
1

Severity: High — blocks P/D disaggregation for LLM inference on multi-GPU nodes

Component: mlx5_core / BlueField-3 eSwitch (legacy mode)

Environment:

  • Hardware: Dell PowerEdge XE9680L, 10× BF3 B3140H SuperNIC (900-9D3D4-00EN-HA0, PSID MT_0000001069)
  • Firmware: 32.43.1014
  • Driver: Tested on both DOCA 3.3.0 (OFED 26.01-1.0.0) and DOCA 3.4.0 (OFED 26.04-0.8.6) — same result
  • OS: RHCOS 9.6, kernel 5.14.0-570.99.1.el9_6, OpenShift 4.21
  • eSwitch: legacy mode, NIC mode
  • Kernel: iommu=pt pci=noacs
  • SR-IOV: 8 VFs per PF, netdevice, isRdma=true
  • Network: L3 routed RoCE via NVIDIA Cumulus switches, nv-ipam + sbr-custom

Problem:
When 2 SR-IOV VFs from the same PF simultaneously perform RDMA operations, all RDMA traffic fails with IBV_WC_RETRY_EXCEEDED (status 12, syndrome 0x81). Zero or near-zero completions. TCP/ping between the same
VFs continues to work — only RDMA verbs traffic is affected. The degradation persists after removing the second VF’s workload and only clears with a full node reboot.

Test Results:

┌───────────────────────────────────────┬────────────────────────────────────┐
│ Scenario │ Result │
├───────────────────────────────────────┼────────────────────────────────────┤
│ 1 VF per PF, single PF │ 391 Gb/s (line rate) │
├───────────────────────────────────────┼────────────────────────────────────┤
│ 1 VF per PF, all 8 PFs simultaneously │ 390 Gb/s per rail │
├───────────────────────────────────────┼────────────────────────────────────┤
│ 2 VFs from same PF, both doing RDMA │ 0.005 Gb/s / IBV_WC_RETRY_EXCEEDED │
├───────────────────────────────────────┼────────────────────────────────────┤
│ 2 VFs from different PFs │ 390 Gb/s (no issue) │
├───────────────────────────────────────┼────────────────────────────────────┤
│ After removing 2nd VF pod (no reboot) │ Still degraded │
├───────────────────────────────────────┼────────────────────────────────────┤
│ After node reboot │ Recovers │
└───────────────────────────────────────┴────────────────────────────────────┘

Error output:
Completion with error at client
Failed status 12: qp_num 473 wr_id 0 syndrom 0x81
scnt=128, ccnt=0

RDMA HW counters after failure:
mlx5_19: out_of_sequence=2, rx_write_requests=5000
mlx5_24: out_of_sequence=13, rx_write_requests=1627

What was ruled out:

  • ATS_ENABLED (disabled, rebooted — no change)
  • Switch (SSH verified: MACs learned, no port security, ping works)
  • Physical link (0 errors, BER 15E-255, FEC clean)
  • GID table (4 valid GIDs per VF)
  • DRA drivers (suspended, rebooted — no change)
  • IOMMU/ACS settings (iommu=pt pci=noacs as recommended)
  • IPAM (tested nv-ipam+sbr and whereabouts — same result)
  • PCIe VF DevCtl (verified identical on working CX-6 Dx cluster)
  • RDMA exclusive netns mode (breaks SR-IOV device plugin with isRdma=true)
  • BIOS SR-IOV (PCIe SR-IOV cap shows VF_Enable=True, VF_MSE=True)
  • Driver upgrade (DOCA 3.3.0 → 3.4.0: slightly improved from ccnt=0 to ccnt=17 on some tests, but still fails)

Reproducer:

  1. BF3 in NIC mode, legacy eSwitch, fw 32.43.1014
  2. Create SriovNetworkNodePolicy: numVfs=8, isRdma=true, deviceType=netdevice, linkType=eth
  3. Deploy 2 pods on each node, each requesting 1 VF from the same PF
  4. Run: ib_write_bw -d mlx5_XX --gid-index 3 -s 262144 -n 1000 -F --report_gbits
  5. Result: IBV_WC_RETRY_EXCEEDED, zero completions

Reference architecture note:
NVIDIA DGX B200 uses ConnectX-7 (not BF3) for compute SR-IOV with multi-VF RDMA. BF3 is only used for storage/management. The Dell XE9680L uses BF3 SuperNICs for compute, which may not be a tested
configuration for multi-VF RDMA.

Attachments:

  • b200-rdma-dumps.tar.gz — 31 binary dumps from mstresourcedump captured during baseline (1 VF, 391 Gb/s), pre-test (2 VFs, before RDMA),

    b200-rdma-dumps.tar.gz (4.5 KB)

    during-test (2 VFs, RDMA failing), and post-failure states. Segments:
    QUERY_VNIC_ENV, STEER_PIPE_CHK, L2_TBL_ENTRIES, QUERY_FT_ALL, QUERY_FTE_ALL, FLOW_COUNTER, PORT_INFO, BASIC_DEBUG, FW_CNTRS.

  • b200-multi-vf-investigation.md — full investigation document with mstconfig dumps, PCIe config, switch verification, RDMA counters

Request:

  1. Analyze the binary dumps to identify the eSwitch steering failure
  2. Confirm whether multi-VF RDMA on the same PF is supported on BF3 B3140H in NIC mode with legacy eSwitch
  3. If supported, provide a firmware fix (current 32.43.1014, DOCA 3.4.0 bundles 32.49.1014 but no image available for PSID MT_0000001069)
2

Hi bbenshab,

Thank you for posting your inquiry to the NVIDIA Developer Forums.

We see that you have already opened an enterprise support ticket under your entitlement, which was going to be my first recommendation for this level of triage.

Along with that, the mismatch between installed firmware and supported firmware with your tested DOCA releases is a severe issue which will need to be rectified ASAP. DOCA and Bluefield firmware are developed in tandem, as such the qualified firmware versions listed in our DOCA support documentation are strict requirements. We do not test the driver stack with other firmware versions, and unexpected operation is almost certain to occur when using a different firmware version than what has been qualified with DOCA.

Regarding the missing firmware image from the DOCA driver package - The Bluefield-3 firmware upgrade process requires use of the ‘bf-fw-bundle’ package, and is not part of the base DOCA-host installation. This bundle is available from the same download location as DOCA - select ‘Bluefield’ under ‘Host or Bluefield’, ‘BF-FW-Bundle’ for the ‘Deployment Package’, and ‘BFB’ (only option) as the installer type:

From there, guidance for deploying this firmware package on the Bluefield adapter itself (in either NIC or DPU mode) is available here:

Best regards,
NVIDIA Enterprise Experience