6-Node DGX Spark Ring Topology - NCCL Fails on Non-Adjacent Node Pairs (Routed RDMA)

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

6-Node DGX Spark Ring Topology - NCCL Fails on Non-Adjacent Node Pairs (Routed RDMA)

TL;DR: 6x DGX Spark connected in a switchless ring (dual-rail, 200G per hop). Direct neighbor RDMA works perfectly. But vLLM/NCCL fails during initialization because it attempts to establish RDMA queue pairs between non-adjacent nodes too, which in a ring requires routed (multi-hop) RDMA — and that fails at the raw ibv_modify_qp level, independent of NCCL/vLLM. Looking for anyone who has solved this beyond the officially-supported 2-node/3-node configurations.

Setup

  • 6x NVIDIA DGX Spark, each with ConnectX-7 (2 physical cards, 4 RoCE interfaces per node: rocep1s0f0, rocep1s0f1, roceP2p1s0f0, roceP2p1s0f1).
  • Wired in a ring topology: each node connected to exactly two neighbors, using both ConnectX-7 cards per link (dual-rail, ~2x200G per hop).
  • L3 point-to-point addressing (/31 per physical link), OSPF (FRR) for inter-node routing, stable per-node loopback/identity addresses on a dummy0 interface (to work around Gloo/NCCL treating lo as loopback-only).
  • Goal: vLLM with --pipeline-parallel-size 6 --tensor-parallel-size 1 (PP-only, since each node has a single GPU).

What works

  • Full OSPF mesh, all 6 loopbacks reachable from every node.
  • ib_write_bw between directly connected neighbor pairs: full bandwidth, no issues.
  • Ray cluster forms correctly across all 6 nodes (ray status shows 6/6 GPUs).
  • NCCL/Gloo bootstrap over TCP (OOB) works fine once GLOO_SOCKET_IFNAME / NCCL_SOCKET_IFNAME are pinned to the stable dummy0 identity address and routing has set src applied via FRR route-maps (kernel was choosing whichever physical ring interface matched the route’s next-hop as source, causing asymmetric-path issues before this fix). GID table asymmetry between the two ConnectX-7 cards was also an issue (extra privacy-extension IPv6 addresses shifting the IPv4-mapped RoCEv2 GID index between cards) — fixed by disabling IPv6 on all ring interfaces and forcing a consistent NCCL_IB_GID_INDEX across all 6 nodes.

What fails

During vLLM engine startup, NCCL’s world-group initialization (ensure_model_parallel_initialized → is_in_the_same_node_as self-test) attempts to establish RDMA connections between all rank pairs, not just adjacent ones in the PP schedule. For non-adjacent nodes (e.g., rank 0 ↔ rank 3), this requires the RDMA queue pair handshake to traverse intermediate nodes via IP forwarding. This consistently fails:

NCCL WARN Call to ibv_modify_qp failed with 110 Connection timed out, on dev rocep1s0f0:1,
curr state INIT, next state RTR, local GID index 3, local GID ::ffff:10.0.X.0, remote GID ::ffff:10.0.Y.0

I isolated this from NCCL/vLLM entirely using a raw RDMA bandwidth test between non-adjacent nodes’ identity addresses:

ib_write_bw -d rocep1s0f0 <non-adjacent-node-loopback> --report_gbits

Result:

Failed to modify QP to RTR
Unable to Connect the HCA's through the link

This confirms the failure is at the RDMA/verbs layer, not something NCCL-algorithm-specific — routed RoCE (through Linux kernel IP forwarding on intermediate nodes) does not satisfy whatever RC/RTR handshake requirements standard RoCEv2 QPs expect between non-adjacent hosts in this setup.

What I’ve found so far

I found community documentation (spark-vllm-docker / sparkrun project) confirming:

  • Official/community support exists for 2-node (direct) and 3-node (switchless mesh) topologies only.
  • Even the 3-node mesh requires a custom-patched NCCL build (not stock NCCL) with NCCL_NET_PLUGIN=none, NCCL_IB_SUBNET_AWARE_ROUTING=1, NCCL_IB_MERGE_NICS=0.
  • For 4+ nodes, a physical switch is the recommended architecture.

Questions for the community

  1. Has anyone gotten a 4+ node switchless ring/mesh working with vLLM/NCCL — with or without a patched NCCL build?
  2. Is there a way to make vLLM/NCCL’s initialization skip the full-mesh connectivity self-test when only adjacent-node communication is actually needed for a given parallelism strategy (PP-only)?
  3. Are NCCL_IB_SUBNET_AWARE_ROUTING / NCCL_IB_MERGE_NICS (from the patched build referenced above) available/effective in stock NCCL 2.28.9, or are they exclusive to that custom build?
  4. For anyone who moved from switchless to a physical 200G switch for >3 node clusters — any gotchas with reusing existing node-to-node DAC/AOC cables, or does switch connectivity require different optics?

Happy to share full logs / configs if useful. Appreciate any pointers — currently blocked on this before we can validate our full 6-node pipeline-parallel deployment.

2

I think. You should use lastest nccl version. Also have another network for initialization. Declare all hosts management IP on .env file
Nccl work on rdma/RoCE so normal routing protocols are useless. If you go the IP route you don’t need nccl.

Start with two then grow your ring to three then four… This way you make sure the setup is correct.

There is a flag for merging interfaces and routing through them.

Try this image with pipeline parallel on a model that fit one spark and split it over 6 sparks

3

RoCE needs directly connected interfaces from a layer 2 point of view. If you do bridging through the intermediate nodes it should work.

But the main thing is you will need a full mesh of RoCE links, even if some of them are logical. Or of course you can run pipeline parallel workloads.

4

Thanks for the responses — both were very helpful, and one of them nailed the root cause I think.

I confirmed, via raw ib_write_bw between non-adjacent nodes (completely bypassing NCCL/vLLM), that the RDMA queue pair handshake fails at the verbs layer:

Failed to modify QP to RTR
Unable to Connect the HCA's through the link

This happens consistently for any node pair that isn’t a direct physical neighbor in the ring — i.e., traffic that has to be IP-routed (L3, via OSPF/FRR) through an intermediate node rather than reaching the peer directly at L2.

Why

“RoCE needs directly connected interfaces from a layer 2 point of view. If you do bridging through the intermediate nodes it should work. But the main thing is you will need a full mesh of RoCE links, even if some of them are logical.”

This matches my failure mode exactly. My ring uses L3 IP routing between non-adjacent nodes (via FRR/OSPF) — the intermediate node re-routes the IP packet rather than transparently bridging the raw Ethernet frame. RoCE’s RC queue pair setup apparently requires L2 adjacency (real or bridged), which routed L3 hops don’t provide. This explains why the officially-documented topologies stop at 3-node full-mesh (where every node is a direct L2 neighbor of every other) and require a switch beyond that.

What I did instead — pragmatic workaround and stable (but stiil not sure that it’s the optimum)

Rather than pursue L2 bridging in a live ring (which reintroduces the broadcast-storm/loop risk we hit earlier in this build without STP or similar), we set:

NCCL_IB_DISABLE=1

This forces NCCL onto its TCP/Socket transport instead of RoCE( that decreased token speed dramatically I assume) . Combined with a stable per-node identity address (dummy interface, to work around Gloo/NCCL’s special-casing of lo), Ray + vLLM now init cleanly across all 6 nodes and pipeline-parallel inference (PP=6) works end-to-end.

Load test results (concurrent requests, continuous batching):

“model”: “qwen3.6-35b-a3b-nvfp4”

==> Response-based summary:
No     Duration(s)  Tokens         Tok/s

1      3.786006272  65             17.1

2      23.414076016 500            21.3

3      23.409236944 500            21.3

4      23.312883216 500            21.4

5      6.116056464  118            19.2

6      3.782063168  64             16.9

7      23.410090896 500            21.3

8      23.317220368 500            21.4

9      23.402731792 500            21.3

10     6.504174048  121            18.6

11     4.300269072  72             16.7

12     23.317153536 500            21.4

13     23.448524464 500            21.3

14     23.401799904 500            21.3

15     7.774614336  151            19.4

16     3.954799056  67             16.9

17     23.404767216 500            21.3

18     23.401025072 500            21.3

19     20.198301056 429            21.2

20     7.098478720  135            19.0

==> OVERALL RESULT
Successful requests : 20 / 20
Total tokens        : 6722
Total duration      : 23.475134736s
Throughput          : 286.3 tok/s (system-wide, all requests combined)
  • 20 concurrent requests: ~286 tok/s aggregate (20 tok/s per request)
  • 100 concurrent requests: ~590 tok/s aggregate, 100/100 succeeded, no instability

(APIServer pid=258984) INFO 07-20 13:53:47 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 606.6 tokens/s, Running: 60 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 0.0%
(APIServer pid=258984) INFO 07-20 13:53:57 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 593.8 tokens/s, Running: 60 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 0.0%
(APIServer pid=258984) INFO:     127.0.0.1:34244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=258984) INFO 07-20 13:54:07 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 572.4 tokens/s, Running: 59 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 0.0%

So: fully functional, no RDMA, over the ring’s physical 200G cabling (just carried as ordinary IP/TCP instead of RoCE — the enterprise/management network is untouched, this all stays on the ring links).

Still open

  • Considering testing L2 bridging (with STP or manual loop-safe topology) on the intermediate nodes as a future experiment to recover RDMA throughput — haven’t attempted this yet given the broadcast-storm history and the fact that the current TCP-based setup is already stable and serving traffic.
  • Curious if anyone has tried the “merge interfaces” NCCL flag (NCCL_IB_MERGE_NICS) or a newer NCCL build specifically for a 4+ node routed (not bridged) topology — or whether L2 bridging is genuinely the only path to RDMA beyond 3 nodes without a switch.

Appreciate the pointers — this thread helped a lot in narrowing down exactly where the wall is.

5

Check out post 35 for doing L2 bridging to make the NCCL work:

6

Thanks again for the reply. I have checked that post(35) as well.

Follow-up to my earlier post. Thanks again for the NCCL_SKIP_TREE_CONNECT / ring-cabling writeup and the rank-swizzle trick (RANK_IDX="0 1 3 2") — both pushed the investigation further and helped us actually isolate the real mechanism.

My last update

1. vLLM’s native multi-node launcher (--nnodes/--node-rank, distributed_executor_backend=mp), not Ray. This gives explicit, guaranteed control over which physical box gets which rank — no more guessing whether Ray’s placement group assigned ranks in physical order. Worker-first launch (--headless), then head, per your README.

2. NCCL_IB_MERGE_NICS=0. Confirmed via logs this actually took effect (Skipping makeVDevice, NCCL_IB_MERGE_NICS=0 — previously NCCL was silently fusing our two dual-port cards into virtual 400Gbps devices).

3. Tested on a 2-node pair (MASTER + one direct physical neighbor) that we verified via show_gids is a genuine, cabled, same-subnet L2 link — no routing involved, no non-adjacent hop, rank mapping guaranteed correct by the native launcher. This should have been the easiest possible case.

It still failed with the same ibv_modify_qp ... Connection timed out on one specific physical port. I couldn’t figure out why. :((

Root cause for me

Each of our nodes has 2 dual-port ConnectX-7 cards = 4 physical RoCE ports, but only 2 physical neighbors in the ring (one dual-rail link per neighbor). NCCL assigns its 16 logical channels to physical HCAs by straightforward round-robin (channel0→dev0, channel1→dev1, channel2→dev2, channel3→dev3, channel4→dev0...), assuming — correctly, in a switched fabric — that any physical port can reach any peer. In our switchless ring, 2 of those 4 ports are cabled to a different neighbor entirely, so a subset of channels get silently routed onto hardware that isn’t wired to the peer at all. Neither NCCL_ALGO=Ring, NCCL_SKIP_TREE_CONNECT=1, nor NCCL_IB_MERGE_NICS=0 change this assignment — none of them is topology-aware at the NIC-selection level.

So: rank-swizzle (or any correct rank↔box mapping) solves the “which rank do I talk to” problem — confirmed necessary but not sufficient. The remaining problem is “which of my 4 physical ports actually reaches that rank,” which stock NCCL has no mechanism to answer in a routed/switchless topology. This matches the earlier reply here almost exactly: full RoCE reachability (even logical, e.g. via the relay patch) is required per pair, not just correct rank ordering.

to sum up

Given the relay patch (nccl-spark-switchless) is exactly the piece designed to solve this (per-pair topology-aware forwarding), and given the risk/effort of building + LD_PRELOAD-ing a source fork against our stock 2.28.9 runtime, we’re running production on the TCP/Socket fallback (NCCL_IB_DISABLE=1) for now — stable, ~300-600 tok/s aggregate under concurrent load, and treating the relay-patch route as a follow-up experiment for a maintenance window rather than blocking on it.

If anyone’s run the relay patch successfully on a dual-rail (4-port, 2-neighbor) ring specifically — as opposed to single-rail — curious whether NCCL_RELAY_ENABLE handles the “wrong port” case above or only the “no port at all” (uncabled diagonal) case described in the README.

I know that switch is the best using scenario for these kind of issues but till I get the switch , I’m trying to find best solution.

7

I am sharing my setup. Make sure you have the latest nccl. Older version fail very easy.

This is the contents of .env file . I am using the netplan files from spark-vllm project

CLUSTER_NODES=192.168.8.100,192.168.8.102,192.168.8.101
COPY_HOSTS=192.168.177.12,192.168.187.13

LOCAL_IP=192.168.8.100
UCX_NET_DEVICES=enP7s7

GLOO_SOCKET_IFNAME=enP7s7
ETH_IF=enP7s7
IB_IF=rocep1s0f0,roceP2p1s0f0,rocep1s0f1,roceP2p1s0f1
NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0,rocep1s0f1,roceP2p1s0f1

NCCL_SOCKET_IFNAME=enP7s7

OMPI_MCA_btl_tcp_if_include=enP7s7
# Mesh mode NCCL settings
CONTAINER_NCCL_IB_SUBNET_AWARE_ROUTING=1
CONTAINER_NCCL_IB_MERGE_NICS=0

Also

I add this to the recipe

env:

  NCCL_NET: "IB"
  NCCL_DEBUG: "info"

command: |

  export NCCL_IB_MERGE_NICS=0
  export NCCL_IB_SUBNET_AWARE_ROUTING=1
  # --- CRITICAL ENVIRONMENT FIXES TO PREVENT NCCL FREEZING ---
  export NCCL_IB_GID_INDEX=3     # Forces RoCE v2 (IPv4) data transport paths
  export NCCL_P2P=1              # Prevents cross-node P2P ring initialization deadlocks
  export NCCL_CROSS_NIC=1        # Restricts traffic to the selected interface lane
  export NCCL_IB_RETRY_CNT=7
  export VLLM_SKIP_CUSTOM_ALLREDUCE=1 # Forces vLLM to use your specialized native compiled NCCL build
     # Extends connection retry attempts before failure timeouts
  export NCCL_IB_TIMEOUT=22      # Adjusts packet confirmation wait times for stable connections


   # Symmetrically cross-mapped interface selection for 3-node switchless mesh ring
  case "$VLLM_HOST_IP" in
    192.168.8.100) export NCCL_IB_HCA="rocep1s0f0,rocep1s0f1" ;; # Link to .138 (177.x) and .162 (187.x)
    192.168.8.102) export NCCL_IB_HCA="rocep1s0f1,rocep1s0f0" ;; # Inverted to match .140 (177.x) and .162 (197.x)
    192.168.8.101) export NCCL_IB_HCA="rocep1s0f0,rocep1s0f1" ;; # Match to .140 (187.x) and .138 (197.x)
  esac
  echo "Node: $VLLM_HOST_IP -> NCCL_IB_HCA=$NCCL_IB_HCA"

It might be an overkill, but it is working.

8

RDMA fully connects across all 6 nodes (subnet-aware routing + no-merge was the fix) — but throughput gain is only ~7%, root cause looks like missing GPUDirect RDMA on this platform

Follow-up to my last two posts. Good news and a puzzling new finding.

The fix that finally got 6-node ring RDMA connecting cleanly

Combining two things, both required together:

NCCL_IB_MERGE_NICS=0            # stops NCCL from bonding our 2 physical ports per card into one virtual 400G device
NCCL_IB_SUBNET_AWARE_ROUTING=1  # (patched NCCL) selects the correct physical port per peer via GID/subnet lookup

With just MERGE_NICS=0 alone: round-robin channel→device assignment still picked the wrong physical port for some channels (a port that goes to a different neighbor). With just SUBNET_AWARE_ROUTING=1 alone (merge left on): the merged/bonded virtual device still tried to open a QP on both underlying physical ports for a single logical peer connection, and since our two ports per card go to different neighbors (dual-rail, but each rail terminates at a different ring peer, not both rails to the same peer), the second QP always timed out.

With both together: ncclCommInitRank completes cleanly, Connected all rings, no ibv_modify_qp timeouts, full 6-node PP=6 pipeline loads and serves traffic. Confirmed working end to end (model loads, inference works, Application startup complete).

The surprising part — throughput barely moved vs our TCP/Socket fallback

TCP fallback (NCCL_IB_DISABLE=1) RDMA (this fix)
20 concurrent requests, aggregate 326 tok/s 349 tok/s

~7% gain. Given the effort to get here, we expected something closer to the 2-4x people usually see moving off TCP.

Suspected root cause — GPUDirect RDMA appears unavailable on this platform, not just unconfigured

NCCL logs this for every HCA, every run, regardless of our settings:

NCCL INFO NET/IB : GPU Direct RDMA Disabled for HCA 0 'rocep1s0f0'
NCCL INFO NET/IB : GPU Direct RDMA Disabled for HCA 1 'rocep1s0f1'
... (same for all 4)

We tried to fix this the conventional way:

sudo modprobe nvidia-peermem
modprobe: ERROR: could not insert 'nvidia_peermem': Invalid argument

No dmesg output at all for the failure (checked dmesg -T, journalctl -k -b, both empty for this specific error beyond the systemd-modules-load line). ib_core is loaded (10 refs, in use by mlx5_ib etc.), nvidia-peermem.ko matches the running kernel’s vermagic exactly (6.17.0-1021-nvidia), so it’s not a version mismatch. It just refuses to insert with zero diagnostic info from the kernel.

Noticed in your (this fork’s) source tree there’s a gdaki/doca-gpunetio/ directory, and our own NCCL logs show:

NCCL INFO GIN/Plugin: Assigned plugin GIN_IB_GDAKI type 3 to comm

Our working theory: on Grace-Blackwell (DGX Spark / GB10), the classical nvidia_peermem PCIe P2P path may not be the intended/supported GPU-NIC data path at all — DOCA GPUNetIO / GDAKI (GPU-initiated async, presumably leveraging the NVLink-C2C coherent CPU-GPU memory rather than classic PCIe peer-to-peer) might be what’s actually expected here, and nvidia_peermem’s silent refusal to load might simply be “not applicable on this architecture” rather than a misconfiguration on our end.

Where we’re leaving it

Given:

  • RDMA now works correctly (huge improvement over the earlier ibv_modify_qp wall)
  • but the throughput gain over plain TCP is marginal (~7%)
  • and getting real GPUDirect (via DOCA GPUNetIO, if that’s even the right path here) looks like a substantial separate undertaking with no guarantee of success

We’re keeping our production deployment on the simpler, more maintainable TCP/Socket fallback (NCCL_IB_DISABLE=1) for now, and treating the RDMA-with-GDR path as a longer-term investigation rather than something blocking day-to-day use.

Question for anyone who’s run this on DGX Spark / GB10 specifically: has anyone gotten nvidia_peermem to actually load on this platform, or is GDAKI/DOCA GPUNetIO the only supported GPU-direct path on Grace-Blackwell? Would appreciate any pointers before we sink more time into DOCA specifically.

9

GDR is not supported on the Spark. That question was asked and answered a lot when the Spark was launched last year.

If you want to cluster more than three Sparks just get a switch. Otherwise you’re self-sabotaging your project. The Spark doesn’t have the pass-through capabilities, so if your node1 wants to talk to node5 in a 6-ring cluster the packets can’t just hop from one node to the other. You’ll have to rely on hacks and add layer-3 routing which kills the low-latency needed by RoCE, thus ~7% throughput.

Once every node has a single-hop layer to any other node in the cluster, via the switch, your low throughput will disappear. A switch will make NCCL and RoCE happy. And your team!

10

Good news. For pipeline parallel using small model there is no big gain . Only for bigger models you can see the difference.

Also the gain here is more memory. Use dspark or dflash drafters to increase the speed. This is the best route for your 6 spark ring.

Try UCX plug-in. I was using it with older nccl versions.

Please share your setup once everything is done. I think tensor parallel is not the best way with spark for inference for large models using a switch.