I couldn’t find many real-world notes on dual-Spark behavior when I added a second unit, so here’s a field report: sourcing the hardware in Canada, the setup path that worked, measured throughput, and two open questions — including how people measure what crossing the interconnect actually costs.
Getting the hardware in Canada — NVIDIA, please ship here
It would be genuinely nice if NVIDIA supplied the DGX Spark and its accessories in Canada. Right now:
- The NVIDIA Marketplace sells the DGX Spark at US $3,999 — but it’s a US-only store. Worse, the country selector on the NVIDIA web store just routes Canadian visitors back to the USA store — there is no Canadian purchase path at all.
- At today’s USDCAD (~1.418), US $3,999 converts to about CAD $5,670. But most Canadian retailers are selling around CAD $7,200 — that’s US $5,077, a ~27% premium (≈ US $1,080 per unit) over NVIDIA’s own price. The cheapest I found is CAD $6,567.95 at Fleet Network — still ~$900 over converted, and showing Canadian inventory out of stock. Across two units for a stacked setup, the Canada premium is over CAD $3,000 at typical street prices.
- The stacking cable is worse: NVIDIA’s branded “DGX Spark Stacking DAC Cable” is sold at Micro Center — US only. In Canada the cable is hard to find and expensive, and importing meant unknown duties on a small accessory. I eventually found this 200G QSFP56 DAC at Memory Express, which worked perfectly.
For a product aimed at individual developers, having the official store, the official price, and the official accessory all inaccessible from Canada adds real cost and friction to a two-unit setup.
Setup — pairing the two units over ConnectX-7
The lowest-configuration path that worked:
1. Set the new DGX up with NVIDIA Sync.
2. Connect the two boxes with a single QSFP DAC directly between the CX-7 ports — no switch.
On the cable spec: NVIDIA’s Spark Stacking guide says to use “the approved QSFP/CX7 cable” but publishes no part number. The certified part appears to be Amphenol NJAAKK-N911 (QSFP112, 400 mm). My standard 200G QSFP56 passive DAC linked at 200 Gb/s immediately. Per the guide, use the same port on both units (both left or both right, viewed from the rear).
3. I let Claude Code do the network bring-up rather than configuring by hand. The prompt, roughly:
“Two DGX Sparks, connected directly with the QSFP cable between the ConnectX-7 ports. Join the new box to my tailnet, set up SSH keys between the boxes, then bring up the CX-7 link: detect the live ports, assign IPs on a dedicated subnet, make it persistent across reboots, enable and validate jumbo frames, add hosts aliases, and test connectivity and throughput. Verify it survives a reboot.”
One useful thing it discovered: both CX-7 ports on both machines sit in a single L2 domain via the NIC’s eSwitch, so exact port-to-port choice doesn’t matter in practice.
The resulting config, for anyone doing it by hand
- Link: 200 Gb/s immediately;
rdma linkshows the RoCE devices ACTIVE. - Addressing: dedicated subnet, persistent via NetworkManager:
nmcli con add type ethernet ifname enp1s0f0np0 con-name cx7-cluster ipv4.method manual ipv4.addresses 10.77.0.1/24 ipv6.method disabled connection.autoconnect yes(.2on the other node), plus/etc/hostsaliases. - MTU 9000, validated with
ping -M do -s 8972to rule out a silent blackhole.
Measured expectations
- Plain TCP (iperf3, 8 streams): ~16 Gb/s — that’s the Grace CPU pushing TCP, not the link. Jumbo frames didn’t change it.
- The 200G is really there for RDMA/RoCE (NCCL, multi-node engines).
- Ordinary file moves over ssh: ~600 MB/s — a 51 GB checkpoint ships between nodes in ~85 s, so we download models once and distribute over the fabric.
- Management plane stays on Tailscale/LAN; the fabric carries only data.
The questions
1. Where do you draw the federated-vs-cluster line? Specifically for models that would fit on one Spark: does anyone run them 2-node (TP/EP) anyway, and how bad is the single-stream latency hit vs the extra KV/context headroom you gain? Our default is federated — each Spark serves its own models at full local memory bandwidth, and multi-node is reserved for models that genuinely don’t fit in 128 GB — but that’s a prior, not a measurement, and I’d like to see numbers that argue either way.
2. What telemetry do you collect on a cluster that you wouldn’t bother with on a single box? Ours, beyond per-node basics (node-exporter + DCGM — which works fine on GB10, including XID error and PCIe replay counters):
- Per-link fabric counters (throughput, retransmits on the CX-7 interfaces) — scraped over the fabric itself so monitoring survives LAN/WiFi flaps.
- “Cluster tax” — multi-node throughput ÷ sum of the same nodes run independently: one number for what the interconnect costs.
- Engine-level metrics per node — tokens/s, TTFT, KV-cache utilization from the serving engines’
/metrics. - One alerting lesson: static “low free memory” thresholds are wrong for inference nodes — an engine that reserves 95% of unified memory up front leaves ~9 GB free by design and is healthy. We moved OOM alerting to PSI (pressure stall) + swap-out rate, which catches real contention instead of paging you about a working reservation.
What’s on your list? Anyone measuring interconnect efficiency for 2-8 node EP setups (accept rates, per-token fabric hops)?
Mark
