PSA: ib_write_bw falsely reports a >64 KiB RDMA WRITE failure on GB10 — the fabric is fine (full characterization + independent probe)

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

If your GB10 cluster fails ib_write_bw at message sizes above 64 KiB with local protection error (0x3b 0x0 0x9d) on the responder — your fabric is very likely healthy. We spent a full day characterizing this on a 3-node Dell Pro Max FCM1253 cluster (DGX OS 7.5.0, CX-7 FW 28.45.4028, dual-rail RoCEv2 via 2× MikroTik CRS804) before proving the failure is specific to the instrument, not the transport.

The short version:

  • ib_write_bw (RC): deterministic failure above exactly 65,536 bytes — every node pair, every RDMA device, 72/72 cells. Boundary is exactly 16 pages. Survives kernel downgrade, IOMMU passthrough, fresh reimage; invariant to MR flags, ODP, and relaxed ordering.
  • A minimal libibverbs probe doing the identical RC WRITE, with responder-side content verification: passes at every size to 8 MiB, every pair, every device — 72/72 clean, byte-exact.
  • NCCL (2.30.7, sm_121, GDR off, NCCL_NET_PLUGIN=none): 3-node all_reduce_perf full sweep, zero validation errors, 24.0 GB/s busbw — with per-MAC byte counters proving both rails and both MACs per cage loaded within 0.02%.

Prior reports we believe are the same defect class, both unresolved: thread 243518 (CX-5, 2023) and thread 282142 (CX-7/KVM, 2024) — same boundary, same syndrome, both perftest-mediated. Full writeup, elimination matrix, and probe source: [GITHUB-ISSUE-URL]

Practical notes for GB10 cluster builders that we validated along the way:

  • NCCL_NET_PLUGIN=none — the bundled AWS OFI plugin fails on GB10 unified memory regardless of NCCL_IB_DISABLE.
  • Check NCCL’s detected topology: ours read the GPU PCIe link as Gen1 x1 and cost-modeled itself to ~0.9 GB/s; a corrected NCCL_TOPO_FILE restored expected performance.
  • RoCE data is NIC-offloaded: netdev soft counters and tcpdump see nothing. Use *_phy counters for port totals and *_vport_unicast_bytes for per-MAC attribution (the two MACs of one cage share the phy counter).
  • One interface per subnet per node (four subnets for the four fabric netdevs) plus arp_ignore=1/arp_announce=2 scoped to the fabric interfaces prevents cross-interface ARP collapsing both MACs onto one PCIe x4 root.

If any ASUS GX10 / Founders Edition owners can run the probe (or plain ib_write_bw at 68 KiB and 4 MiB) and report pass/fail with their perftest version, that cross-vendor comparison would help pin down the trigger.