Most of the work was done by GPT. All I did was plug in the cables and press the power button. (lol)
Here are some quick notes:
1. C=1, 50K context length, cold: TTFT ~ 20s, prefill ~ 2,500 tok/s, decode ~ 90 tok/s
2. C=1, 150K context length, KV cache hit: TTFT ~ 0.776s, effective prefill ~ 193,280 tok/s, decode ~ 90 tok/s
3. C=6, 150K context length, KV cache hit: TTFT ~ 4.334s, decode ~ 40.4 tok/s per request (avg)
## 2. Hardware Configuration
Private 4-node Spark cluster, wired through a QRS812 switch fabric.
Component Configuration
| Compute | 4x ASUS Spark / DGX Spark-class nodes |
| Switch | QRS812 |
| Cabling | 2x MK-TW-QDD-20x QSFP56 400G breakout DAC |
| Fabric | Spark port2 / QRS812 fabric |
| Cluster mode | 4-node tensor parallel, TP=4 |
Total build cost was roughly **USD $20.5K-$21.1K**, with some parts bought before the recent price hikes.
## 3. QRS812 Switch / RDMA Latency Benchmark
Ran a full-mesh RDMA verbs latency matrix across the QRS812-facing Spark port2 fabric.
Test profile:
Item Value
| RDMA device | `roceP2p1s0f1` |
| RoCE GID index | `11` |
| Message size | `2B` |
| Iterations | `10000` |
| Tests | `ib_write_lat`, `ib_read_lat`, `ib_send_lat` |
Summary:
| Test | Avg Latency Range | p99 Range | Notes |
|—|—:|—:|—|
| RDMA write | `2.93-3.49 us` | `4.17-4.94 us` | Tight across the full mesh |
| RDMA read | `5.64-6.34 us` | `8.27-10.45 us` | Higher than write/send, but stable |
| RDMA send | `2.55-3.44 us` | `2.95-4.65 us` | Most consistent p99 |
Full matrix:
QRS812 Spark Fabric RDMA Latency Matrix
device=roceP2p1s0f1 gid=11 size=2B iters=10000 tests=write read send
write latency
FROM\TO spark1 spark2 spark3 spark4
spark1 self 3.49us p99=4.94 3.20us p99=4.44 3.22us p99=4.52
spark2 3.49us p99=4.82 self 3.22us p99=4.54 3.20us p99=4.47
spark3 3.19us p99=4.35 3.23us p99=4.62 self 2.93us p99=4.17
spark4 3.25us p99=4.62 3.22us p99=4.70 2.97us p99=4.36 self
read latency
FROM\TO spark1 spark2 spark3 spark4
spark1 self 6.28us p99=9.55 6.34us p99=9.50 6.33us p99=10.21
spark2 6.32us p99=10.00 self 6.23us p99=9.47 6.28us p99=10.45
spark3 5.68us p99=8.27 5.66us p99=8.78 self 5.71us p99=9.73
spark4 5.66us p99=8.27 5.64us p99=8.83 5.70us p99=8.66 self
send latency
FROM\TO spark1 spark2 spark3 spark4
spark1 self 3.44us p99=4.65 3.17us p99=4.40 3.17us p99=4.50
spark2 3.10us p99=3.50 self 2.82us p99=3.22 2.83us p99=3.23
spark3 2.85us p99=3.25 2.85us p99=3.26 self 2.55us p99=2.95
spark4 2.84us p99=3.26 2.84us p99=3.24 2.55us p99=2.95 self
```
Additional fabric checks:
| Check | Result |
|— |— |
| Full-mesh ICMP RTT over QRS812 | about `0.49-1.26 ms` observed |
| TCP iperf2, Spark1 → Spark2, 16 streams | about `105 Gbit/s` |
| RDMA `ib_write_bw`, Spark1 → Spark2 over QRS812 | `107.66 Gbit/s` |
| Direct QSFP112 baseline, Spark1 → Spark3 | `107.66 Gbit/s` |
| NIC PHY errors after transfer / benchmark | CRC, symbol, discard, link-down all `0` |
The QRS812 fabric looks clean for Spark-to-Spark RoCE traffic. Write/send latencies sit in the low single-digit microseconds, read around 6 us.
## 4. Model And Quantization Profile
The cluster serves **DeepSeek-V4-Flash-0731** with **DSpark speculative decoding**.
| Item | Value |
|— |— |
| Model | `deepseek-ai/DeepSeek-V4-Flash-0731` |
| Runtime | vLLM + DSpark |
| Parallelism | TP=4 across 4 Spark nodes |
| Max model length | `512000` |
| KV cache dtype | `nvfp4_ds_mla` |
| Prefix cache | enabled |
| Scheduling | async scheduling + chunked prefill |
| DSpark speculative tokens | `MTP_NUM_TOKENS=3` |
| vLLM fingerprint | `vllm-0.21.1rc1.dev339+g1967a5627bc3-tp4-d2211cb5` |
Note this isn’t a plain FP8 deployment. Other vLLM examples serve the official model with FP8 KV; this GR7a path uses the experimental **NVFP4 DS-MLA KV** path instead.
## 5. vLLM Image / GR7a Runtime Notes
Current profile label is **GR7a**: a 4-node QRS812 profile based on existing community DSpark / NVFP4 work, adapted to this hardware layout.
Community sources we built from or cross-checked:
- - DeepSeek official model
- - Fraser Price DSpark model/runtime lineage: fraserprice/DeepSeek-V4-Flash-DSpark · Hugging Face
- - MiaAI-Lab 2-node DGX Spark recipe: GitHub - MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark: DeepSeek-v4-Flash 0731 recipe for 2x DGX Sparks · GitHub
- - drowzeys / Keys DSpark concurrency patch: GitHub - drowzeys/Keys-Concurrency-Patch-for-DSpark-DeepSeek-V4-Flash: Concurrency Patch to enable on DSpark - DeepSeek V4 Flash/Pro C=16 can be modified for more · GitHub
- - TonyD2Wild 0731 NVFP4 / Patch 4 lineage: GitHub - tonyd2wild/DeepSeek-v4-Flash-0731-DSpark-1M-NVFP4-KV-2x-DGX-Spark: DeepSeek V4 Flash DSpark 1M NVFP4 KV recipe for 2x DGX Spark · GitHub
Our main modifications:
| Area | Modification |
|—|—|
| Topology | Expanded the recipe to 4 nodes over QRS812, `NNODES=4`, `TP=4` |
| Fabric | Switched NCCL/Gloo to Spark port2 QRS812 fabric: `enP2p1s0f1np1`, `roceP2p1s0f1`, GID `11` |
| Model | Swapped to official `DeepSeek-V4-Flash-0731` weights |
| Runtime | Kept Stage-C NVFP4 DS-MLA KV path |
| Patch 4 | Bind-mounted patched `dspark.py` with `shared_experts.gate_up_proj` mapping |
| Concurrency | Used Keys-style request-stable DSpark slot mapping and ragged mixed prefill/decode handling |
| GR7a C12 profile | Raised `MAX_NUM_SEQS` to `12` for C8/C12 shared-prefix cache-hit tests |
Patch 4 was the big one. When we first brought the 4-node cluster up, the patch wasn’t applied and we didn’t notice — decode sat around `23-26 tok/s` in C=1 long-context tests, and DSpark acceptance dropped to roughly `~25%`, well below the ~60% we normally see on the 2-node setup. After we applied the `shared_experts.gate_up_proj` mapping fix, C=1 decode came back to roughly the `90 tok/s` class and acceptance recovered to the `60-80%` range in typical runs (varies with concurrency — C=1 cold reached ~95%).
## 6. C=1 Benchmark: Cold Start vs KV Cache Hit
Method: one fresh prompt primes the cache, then the exact same prompt is sent again. The cache-hit FFTS number is `prompt_tokens / TTFT`, so it’s effective cached first-token speed, not raw uncached prefill throughput.
| Context | Mode | TTFT | Prefill / Effective FFTS | Decode |
|--- |--- |--- |--- |--- |
| 50K | Cold prime | `19.622s` | `2,548 tok/s` | `93.15 tok/s` |
| 50K | KV cache hit | `0.413s` | `120,977 tok/s` | `84.80 tok/s` |
| 100K | Cold prime | `41.510s` | `2,409 tok/s` | `85.34 tok/s` |
| 100K | KV cache hit | `0.578s` | `173,082 tok/s` | `81.02 tok/s` |
| 150K | Cold prime | `64.433s` | `2,328 tok/s` | `90.15 tok/s` |
| 150K | KV cache hit | `0.776s` | `193,280 tok/s` | `90.30 tok/s` |
Health stayed clean after the run: `/v1/models` healthy, QRS812-facing NIC error counters at zero.
## 7. 150K Benchmark: C=1 / 2 / 4 / 6, Cold vs KV Cache Hit
Target context: `150K` prompt tokens.
Generation: `max_tokens=512`, streaming, `temperature=0`, `ignore_eos=true`.
### Cold Unique-Prompt Wave
| C | Success | TTFT p50 / p95 | Decode p50 | Aggregate TPS | DSpark Acceptance |
|---:|---: |---: |---: |---: |---: |
| 1 | `1/1` | `64.153s / 64.153s` | `93.99 tok/s` | `7.21` | `95.71%` |
| 2 | `2/2` | `98.308s / 126.740s` | `41.36 tok/s` | `7.49` | `90.67%` |
| 4 | `3/4` | `134.139s / 193.381s` | `3.64 tok/s` | `1.28` | `77.45%` |
| 6 | `3/6` | `132.949s / 191.495s` | `1.85 tok/s` | `1.28` | `59.96%` |
C=4 and C=6 cold unique-prompt waves hit client-side timeouts. Kept them in anyway — they’re useful negative evidence: simultaneous uncached 150K requests aren’t this config’s sweet spot.
### KV Cache-Hit Average
Average of `cache_hit_1` and `cache_hit_2`.
| C | Success | Avg TTFT p50 | Decode p50 | Aggregate TPS | Spark Accelerator Power |
|---|--- |--- |--- |--- |--- |
| 1 | `2/2` | `0.657s` | `92.18 tok/s` | `82.41` | `145.4 W` |
| 2 | `4/4` | `1.055s` | `64.03 tok/s` | `102.35` | `166.3 W` |
| 4 | `8/8` | `3.265s` | `48.69 tok/s` | `123.08` | `136.7 W` |
| 6 | `12/12` | `4.334s` | `40.39 tok/s` | `165.54` | `169.3 W` |
Prefix/KV cache reuse completely changes the serving profile. C=6 cache-hit finished `12/12` at around `165 tok/s` aggregate output.
## 8. 150K Benchmark: C=8 / 12 KV Cache Hit
Method: one shared 150K C=1 request primes the cache first, then C8 and C12 waves reuse the exact same prompt.
Shared C=1 prime:
| Success | TTFT p50 | Decode p50 | Aggregate TPS |
|--- |--- |--- |--- |
| `1/1` | `65.086s` | `74.46 tok/s` | `6.98` |
Cache-hit waves:
| C | Wave | Success | TTFT p50 / p95 | Decode p50 | Aggregate TPS | DSpark Acceptance |
|---|---|--- |--- |--- |--- |--- |
| 8 | cache_hit_1 | `8/8` | `5.634s / 5.646s` | `29.97 tok/s` | `149.18` | `48.04%` |
| 8 | cache_hit_2 | `8/8` | `5.888s / 5.897s` | `30.59 tok/s` | `159.25` | `53.95%` |
| 12 | cache_hit_1 | `12/12` | `5.199s / 8.218s` | `26.25 tok/s` | `217.85` | `63.85%` |
| 12 | cache_hit_2 | `12/12` | `8.210s / 8.801s` | `25.38 tok/s` | `199.62` | `57.19%` |
Two-wave average:
| C | Success | Avg TTFT p50 / p95 | Avg Decode p50 | Avg Aggregate TPS | Avg Acceptance |
|---|--- |--- |--- |--- |--- |
| 8 | `16/16` | `5.761s / 5.771s` | `30.28 tok/s` | `154.22` | `51.00%` |
| 12 | `24/24` | `6.705s / 8.509s` | `25.82 tok/s` | `208.74` | `60.52%` |
C12 gives ~35% more aggregate throughput than C8, at the cost of higher p95 TTFT. Usable as a high-concurrency cache-hit lane, but it doesn’t replace cold 150K parallel prefill.
## 9. Throughput Comparison: 2x 2-node Spark vs 1x 4-node Spark
| Setup | Topology | Workload Shape | Concurrency | Aggregate Throughput |
|— | — |— |— |— |
| 2-node Spark cluster | TP=2, single replica | historical DSpark code-generation benchmark | C=12 | `230.10 tok/s` |
| 2x 2-node Spark clusters | two independent TP=2 replicas | estimated from two similar replicas | C=12 + C=12 | `~460 tok/s` theoretical aggregate |
| 4-node Spark cluster | TP=4 over QRS812 | 150K shared-prefix KV cache hit | C=8 | `154.22 tok/s` |
| 4-node Spark cluster | TP=4 over QRS812 | 150K shared-prefix KV cache hit | C=12 | `208.74 tok/s` |
| 4-node Spark cluster | TP=4 over QRS812 | 150K cold unique prompts | C=6 | `1.28 tok/s`, partial timeout |
Bottom line: if you can split requests across replicas, the old **2x 2-node layout still wins on raw aggregate decode throughput**. The **4-node TP=4 QRS812 setup** is mainly worth it for a single large shared-memory / long-context serving lane, especially when prefix/KV cache hits apply. Cold parallel 150K unique prompts stay inefficient even on 4 nodes.
## Notes
- Spark power numbers are `nvidia-smi power.draw` sums only, not wall-plug whole-system power.
- QRS812 switch power wasn’t included in the benchmark tables. A manual telemetry check showed about `34.4 W`, but it wasn’t wired into the benchmark sampler.
- All hardware benchmark numbers here are from our private setup; they weren’t public before this post.