Setting up a plain Docker + NVIDIA Container Toolkit workflow across two directly-cabled DGX Sparks (no AI Workbench, no Kubernetes) for PyTorch DDP training. A few things we found along the way that might save others some time — everything below is empirically verified on our own two Sparks, not just read off docs.
1. You probably don’t need to build NCCL from source. The official playbooks (connect-two-sparks, nccl on build.nvidia.com) have you build NCCL from source targeting sm_121 via sudo apt-get install libopenmpi-dev + a from-source build. If you’re running a reasonably current NGC container (we used nvcr.io/nvidia/pytorch:26.03-py3), it already bundles NCCL 2.29.7 with Blackwell support out of the box:
docker run --rm --gpus all <image> python3 -c \
"import torch; print(torch.cuda.nccl.version(), torch.cuda.get_device_capability())"
# (2, 29, 7) (12, 1) <- (12,1) = sm_121, GB10's compute capability
No sudo, no host-level build needed — just point NCCL at the direct-cable interface from inside the container.
2. The physical link + IP layer likely already works with zero setup. Both our Sparks had enp1s0f1np1 up with an auto-assigned 169.254.x.x/16 link-local address out of the box — no netplan config needed (matches the “Option 1: Automatic IP Assignment” case in NVIDIA’s own connect-two-sparks playbook, just already done for us). Worth checking before assuming you need the netplan steps:
ip -br addr show enp1s0f1np1
ping -c 4 <other Spark's 169.254.x.x address>
3. Container-native multi-node test that worked first try:
docker run -d --rm --gpus all --network=host --ipc=host \
-e RANK=<0 or 1> -e WORLD_SIZE=2 -e MASTER_ADDR=<node0's 169.254.x.x> \
-e MASTER_PORT=29500 -e NCCL_SOCKET_IFNAME=enp1s0f1np1 \
<image> <your torch.distributed entrypoint>
Launch one rank per Spark, correct all_reduce result on both sides.
4. The gotcha — this “worked” but wasn’t using RDMA. --network=host shares the IP/socket namespace but not the RDMA verbs device nodes (/dev/infiniband/*), which is a separate kernel subsystem. Without passing those through explicitly, NCCL’s IB plugin finds no device and silently falls back to NET/Socket — still correct, just not fast, and nothing warns you unless you check NCCL_DEBUG=INFO. Fix:
docker run ... --device=/dev/infiniband/rdma_cm --device=/dev/infiniband/uverbs0 \
--device=/dev/infiniband/uverbs1 --device=/dev/infiniband/uverbs2 \
--device=/dev/infiniband/uverbs3 --cap-add=IPC_LOCK ...
After that, NCCL_DEBUG=INFO shows NET/IB : Using [0]rocep1s0f1:1/RoCE [1]roceP2p1s0f1:1/RoCE instead of NET/Socket — real RDMA (RoCE), using both ConnectX-7 ports. Measured bandwidth: 179.54 Gbps on a 2GiB all_reduce, which clears the 175 Gbps minimum NVIDIA’s own cluster-setup script validates against for a direct 2-node connection.
5. End-to-end validation: wired this into nanoGPT’s stock (unmodified) multi-node torchrun invocation and ran a real GPT-2-124M DDP training job across both nodes — converged correctly, both ranks in sync.
Bonus, separate finding — debugging/profiling inside containers: the commonly-suggested --cap-add=SYS_ADMIN --security-opt seccomp=unconfined is broader than needed. --cap-add=CAP_PERFMON alone was sufficient for Nsight Systems CPU profiling on our kernel (6.17.0-1026-nvidia) despite perf_event_paranoid=4 — a generic Ubuntu bug report suggesting CAP_PERFMON is broken at that paranoid level did not reproduce here. --cap-add=SYS_PTRACE separately covers debugger-attach (gdb/py-spy) to an already-running process.
Happy to share more detail (exact scripts, full NCCL debug logs) if useful to anyone else on 2-Spark setups.