Decided to jump in with the DGX system. I know it’s probably not the best thing for the buck or the most powerful but this is where I started here is where we’re at. I’ve built a 6-node DGX Spark cluster (soon to be 8) on a 200G RoCE fabric and I’m documenting everything just for reference and to gather criticism or things I should try. I’m fairly sure I’m leaving performance on the table and you all are much smarter than me. I’m happy to re-run any test and report back to help show results.
The hardware
Nodes : 6x DGX Spark - GB10, 128 GB unified LPDDR5X, ConnectX-7, 3.7 TB NVMe
GPU arch : sm_121 (compute capability 12.1)
Switch : MikroTik CRS804-4DDQ, 4x 400G QSFP-DD, RouterOS 7.20.8
Cabling : FS QDD-400G-2QPC01 breakout - 400G QSFP-DD to 2x 200G QSFP56 (one cable = two Sparks)
Fabric : Dual-rail - each node’s two ConnectX ports on SEPARATE rails, own private /24 each, isolated (no gateway). MTU 9000 on every port.
CUDA : 13.0
Each Spark’s two ConnectX ports go to separate rails; each rail is PCIe Gen5 x4 (~128 Gbps theoretical per rail).
Usable memory: 128 GB nominal → ~121 GB usable → we budget ~104.5 GB GPU-visible per node after DGX OS overhead (~13 GB). Across 6 nodes that’s ~627 GB of practical model budget.
Switch configuration
/interface ethernet set qsfp56-dd-N-M auto-negotiation=no speed=200G-baseCR4
Auto-negotiation off with an explicit speed. With auto-neg on, links came up but negotiated inconsistently between ports.
Fabric validation
| Test | Result |
|---|---|
ib_send_bw |
~196 Gbps |
| NCCL all-reduce | 184 Gbps, 0 errors |
NVIDIA’s guidance is 175 Gbps, so we’re above bar.
Q1: Is dual-rail actually buying us anything for inference, or would a single 200G rail perform identically? We built both rails because we could, but I’ve never isolated the benefit. Q2: Anyone tuned RoCE beyond MTU 9000 — PFC/ECN, congestion control — and measured a difference on Sparks specifically?
Benchmark
Is there a standard benchmark this community uses that I should run instead, so my numbers are comparable to yours? I’d rather report something you all recognize.
Results — single-stream decode
All NVFP4, all measured with the same method:
| Model | Params | Nodes | Parallelism | Decode | Context |
|---|---|---|---|---|---|
| Qwen3.6-35B-A3B | 35B | 2 | TP=2 | 99.0 tok/s | 256K |
| Phi-4-reasoning-plus | 14B | 1 | solo | 26.9 tok/s | 32K |
| Nemotron-3-Super | 120B | 2 | TP=2 | 25.5 tok/s | 131K |
| GLM-5.2 | 753B | 6 | TP=2 × PP=3 | 7.4 tok/s | 131K |
Two results I did not expect:
A 14B on one node roughly matches a 120B on two (26.9 vs 25.5 tok/s).
More nodes made things slower. Qwen3.6-35B at TP=2 gives 99.0 tok/s; the same model at TP=4 gives 89.2. Sharing a model that already fits costs real throughput.
Results — concurrency
This is where the cluster looks very different from the single-stream numbers, and it’s the part I’d most like scrutinized.
| Model | Nodes | Peak concurrent | Queued | Aggregate |
|---|---|---|---|---|
| Qwen3.6-35B | 2 | 128 | 0 | 1403 tok/s |
| Phi-4-reasoning | 1 | 32 | 0 | 765 tok/s |
Qwen sustains 14× its single-stream throughput across 128 simultaneous streams on two nodes, with nothing queued — so 128 wasn’t the ceiling either, it’s just where I stopped. Per-stream degrades gracefully: 99 tok/s at 1 stream, 26.8 at 32, 11.0 at 128.
Phi-4 does 765 tok/s aggregate across 32 lanes on a single Spark.
Multiple models coexist cleanly. We ran three models on three separate node pairs at once. The 120B measured 25.5 tok/s under that load vs 26.4 solo — about 3%, within run-to-run noise. Separate node groups don’t appear to contend meaningfully.
What I’d most like from this thread
Honestly: tell me what’s wrong with the above. Im just documenting the journey of learning and providing data where I can. I’ll re-run any test people want and post the results here. Setup notes and both benchmark tools can be shared if useful.
