Hi all,
I wanted to share SparkRing, an experimental inference stack I’ve been building for running GLM-5.2 across four directly connected DGX Sparks - without an Ethernet switch in the inference path.
This started as a response to “Your Mikrotik 804 is backordered until September…”
So here we are.
TL;DR
-
4× DGX Spark
-
Four direct 200GbE ConnectX-7 links arranged as a physical ring.
-
No Ethernet switch. Management backplane is wifi/usb ethernet (future 10Gbe plans)
-
GLM-5.2 MXFP4-Experts-GPTQ, TP4/DCP4/MTP4.
-
Approximately 500K tokens of logical KV capacity.
-
Roughly 19–20 tok/s single-stream decode.
-
Roughly 50–63 tok/s aggregate at C8 in the controlled shared-prefix matrix.
-
Short workload-dependent C8 windows have reached 66.3 aggregate tok/s.
-
I’m seeing consistent 20-30+ C1 mixed use. It’s faster in-use than the benchmarks show
EDIT: You’ll need to point your LLM agent of choice at the repo to get this stood up.
Measured results
These are 30-second sustained-decode cells. C1 is single-request throughput; C2/C4/C8 are aggregate throughput.
Prefill is standalone uncached C1 prefill. Most context lengths are single-sample scouts.
| Context | Prefill tok/s | C1 | C2 aggregate | C4 aggregate | C8 aggregate |
|---|---|---|---|---|---|
| 8K | 844 | 20.3 | 27.1 | 40.5 | 49.2 |
| 16K | 876 | 19.0 | 26.4 | 37.9 | 53.3 |
| 32K | 830 | 20.3 | 27.6 | 38.6 | 51.9 |
| 64K | 832 | 20.3 | 27.0 | 39.4 | 50.9 |
| 128K | 796 | 19.7 | 26.3 | 37.2 | 47.7 |
Configuration:
- Checkpoint:
aidendle94/GLM-5.2-MXFP4-Experts-GPTQ - TP4 / DCP4 using
ag_rs - Dynamic MTP2/4
nvfp4_ds_mlaKV with per-token scaling RoPE fp8- 4,000,000,000 KV bytes per rank
- 500,224-token measured logical KV pool
- 458,752-token request ceiling
- 4,096 maximum batched tokens
- 8 sequence slots
- FULL_AND_PIECEWISE CUDA graphs
How the switchless part works
Stock NCCL/RoCE on a four-node ring tries to establish queue pairs between non-adjacent ranks. Those ranks are not direct layer-2 neighbors, so the communicator fails before choosing a collective algorithm.
SparkRing avoids requiring those non-adjacent hardware paths.
Its low-level transport layer is called SIRCL: Switchless Inference RDMA Collective Layer. It uses direct-neighbor RDMA RC links, mapped pinned-memory arenas on GB10, explicit sequence/doorbell ordering, and inference-specific collective plans.
The current stack includes:
- Custom TP4 all-reduce.
- Custom DCP query and fused output/LSE combine.
- Custom vocabulary and all-gather paths.
- CUDA-graph-aware command rings.
- Explicit software decomposition/relay for communication that would otherwise require a non-adjacent connection.
- A patched ring-only NCCL fallback for operations not yet moved onto the custom path.
Management, SSH, Gloo, and NCCL bootstrap use a separate management interface. The direct RoCE links carry inference payloads.
SparkCache
The repository also includes SparkCache, an experimental DCP4-sharded persistent context cache. It can snapshot and restore very large KV contexts from local NVMe without affecting unrelated requests.
It works, but its current snapshot/store path can still interfere with unique-context decode slightly. Eliminating that interference is one of the next priorities to remove this regression.
Current limitations
- Only supports one model?
Next goals
- Produce a clean-room, end-to-end public runtime build
- Remove the remaining patched-NCCL fallback operations.
- Improve unique-context and SparkCache interference behavior.
- Attack the remaining MoE/expert-weight and launch-latency floor.
- Move single-stream mixed-workload decode toward a consistent 25–30 tok/s.
- Compare the direct ring against a proper low-latency switched fabric.
- Explore whether the transport design generalizes beyond four Sparks, 4+N GB10s
- Support of more models
Input wanted
I’d especially appreciate help from anyone interested in:
- Reviewing the RDMA ordering, fences, and CUDA-graph integration
- Running the cable/link probes on another four-Spark cluster.
- Reproducing the runtime builder on clean hardware.
- Comparing against a switched four-node cluster using the same model.
- MoE expert reuse, persistent-kernel, or launch-reduction ideas
- Testing failure recovery and long-duration stability
- Prefill enhancements
- Correctness testing
This builds on a lot of community work around B12X, vLLM on GB10, Aiden’s GLM checkpoint, and switchless NCCL experiments from these forums and elsewhere. Special credit to all the rtx6k discord community members who work on this stuff constantly to improve it for everyone. Full credits and provenance details are in the repository.
It is still experimental, but four-switchless-Spark GLM-5.2 inference is real and usable. Feedback, criticism, reproduction attempts, and PRs are very welcome. This was clearly coded with the help of LOTs of ai, so code cleanup, coherence, and on and on, will be an ongoing chore for a bit.
Repo needs a bit more added for maintainers / deets, will have that all added soon.
EDIT: I will have some llm benchmark pictures to add when I get back to mi casa.
Just ran, all of these are several tok/s lower than they should be due to how sparkcache is currently working. But, wanted to get some “proof” on the board.
EDIT: speeds closer to those documented in my A/B runs:
