Has anyone done DDP training with 2x Sparks?

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Very curious if anyone has successfully done DDP training with 2x sparks to increase training throughput. I tried with musubi tuner but both GPU’s would stay at 100% use but never complete the first step with both GPU’s never going above 20w. Without DDP each GB10 hits its normal power use and its not the the power bug. Its something with DDP itself and I cant pin down what it is. Tried many times with opus to fix it with no luck.

2

Hi @corbett_korbett

First let’s check if this is not a NCCL busy-wait deadlock (the GPUs aren’t computing, they’re spinning inside a NCCL collective kernel waiting for a peer that never answers)

Step 0: Sanity check both nodes

bash

ibdev2netdev                     # RoCE devices ↔ interfaces, should show Up
ip a | grep -A2 enp1s0f          # both sub-interfaces of the cabled port have IPs?
sudo mlxfwmanager --query        # CX7 firmware (healthy is 28.45.4028)
cat /etc/nccl.conf 2>/dev/null   # must NOT contain NCCL_IB_DISABLE=1
ip link | grep mtu               # MTU consistent on both nodes

Step 1: Raw RDMA path (no NCCL)

bash

# node A:
ib_write_bw -d rocep1s0f1 -R
# node B:
ib_write_bw -d rocep1s0f1 -R <nodeA-CX7-IP>

Expect ~100–110 Gbps (one x4 rail). Crash at 128KB with “syndrom 0x88” = firmware issue, stop here and fwupdmgr update (syndrom 0x88 thread).

Step 2: NCCL collectives (nccl-tests)

Build with current NCCL master ≥2.28.9, older pinned versions fail to compile (build thread):

bash

make -j src.build NVCC_GENCODE="-gencode=arch=compute_121,code=sm_121"

Run the reference test:

bash

mpirun -np 2 -H <ip1>:1,<ip2>:1 \
  --mca plm_rsh_agent "ssh -o StrictHostKeyChecking=no" \
  -x LD_LIBRARY_PATH \
  -x NCCL_SOCKET_IFNAME=enp1s0f1np1 \
  -x NCCL_IB_HCA=rocep1s0f1,roceP2p1s0f1 \
  -x NCCL_IB_DISABLE=0 \
  ./build/all_gather_perf -b 16G -e 16G -f 2

Healthy: ~23–24 GB/s busbw. ~13–16 GB/s = single rail only. ~1–3 GB/s = TCP fallback. Hangs = fabric problem, not DDP. Go back to steps 0–1 (Why is my NCCL broken?).

Step 3: Minimal PyTorch DDP repro (this is where your bug lives)

nccl-tests passing but this hanging is exactly the pattern from the all-reduce deadlock thread. Save as ddp_test.py:

python

import os, torch, torch.distributed as dist
dist.init_process_group("nccl")
rank = dist.get_rank()
print(f"rank {rank}: init OK", flush=True)
x = torch.ones(1_000_000, device="cuda")
dist.all_reduce(x)
torch.cuda.synchronize()
print(f"rank {rank}: all_reduce OK, sum={x[0].item()*dist.get_world_size()}", flush=True)
dist.destroy_process_group()

Export on both nodes (same shell that runs torchrun):

bash

export NCCL_SOCKET_IFNAME=enp1s0f1np1
export GLOO_SOCKET_IFNAME=enp1s0f1np1
export NCCL_IB_HCA=rocep1s0f1,roceP2p1s0f1
export NCCL_IB_DISABLE=0
export NCCL_DEBUG=INFO
# containers only:
export NCCL_NET_PLUGIN=none

bash

# node A (master):
torchrun --nnodes=2 --node_rank=0 --nproc_per_node=1 --master_addr=<A-CX7-IP> --master_port=29500 ddp_test.py
# node B:
torchrun --nnodes=2 --node_rank=1 --nproc_per_node=1 --master_addr=<A-CX7-IP> --master_port=29500 ddp_test.py

Step 4: Read the result

  • No “init OK” → bootstrap hang: Gloo/TCP bound the wrong interface (Wi-Fi, docker0, 10GbE). Fix GLOO_SOCKET_IFNAME/NCCL_SOCKET_IFNAME.
  • “init OK” but no “all_reduce OK” → your exact deadlock, reproduced without musubi. Check NCCL_DEBUG=INFOoutput: look for the line saying via NET/IB (good) vs via NET/Socket (fallback); verify it lists both rocep1s0f1 and roceP2p1s0f1 and bound the CX7 IP. Capital P in roceP2p1s0f1 matters (0x51 thread).
  • Both print → the fabric and DDP are fine; the bug is musubi/accelerate not propagating these env vars into its worker processes. Set them in accelerate config’s environment or a wrapper script on both nodes.

While hung, you can confirm the diagnosis by:

py-spy dump --pid <pid> shows the trainer blocked in all_reduce/broadcast, and mlnx_perf -i enp1s0f1np1 shows near-zero RDMA traffic, GPUs spinning, nothing on the wire.