Just wanted to pitch in with my tests on this today.
Using this recipe, DeepSeek-V4-Flash running across two DGX Sparks using the aidendle94/sparkrun-vllm-ds4-gb10:production-ready image (thanks for the recipe). Sharing real measured throughput + a few deployment notes for anyone else attempting this.
Setup
- 2× GB10, 128 GB unified memory each, TP=2.
- Interconnect: one QSFP56 cable over the ConnectX-7.
- vLLM 0.21.1, --max-model-len 1000000 (full 1M served fine), MTP speculative decoding on (num_speculative_tokens=2).
Throughput (llama-benchy, pp2048 / tg128, single node client):
┌──────────────────────────┬──────────┐
│ Test │ t/s │
├──────────────────────────┼──────────┤
│ Prefill @ depth 0, c1 │ 1,574 │
├──────────────────────────┼──────────┤
│ Prefill @ depth 8192, c1 │ 1,586 │
├──────────────────────────┼──────────┤
│ Generation @ d0, c1 │ 35.6 │
├──────────────────────────┼──────────┤
│ Generation @ d0, c4 │ 63.9 │
├──────────────────────────┼──────────┤
│ Generation @ d4096, c4 │ 30.8 │
├──────────────────────────┼──────────┤
│ Generation @ d8192, c4 │ 23.5 │
├──────────────────────────┼──────────┤
│ TTFT @ d0, c1 │ 1,276 ms │
└──────────────────────────┴──────────┘
Takeaways:
- Prefill is excellent and essentially flat across context depth (~1,580 t/s from d0 to d8192) the long-context story holds up well on this hardware.
- Single-stream generation ~36 t/s. Scales to ~64 t/s at c4/shallow but degrades under concurrency × depth (23.5 t/s at d8192 c4). Expected MoE-over-two-boxes behaviour — generation is gated by the cross-node NCCL hop on every token.
- reasoning_effort (high vs max) made no difference to raw token rate, as expected.
Quality (sanity check, tool-calling bench, 69 scenarios): scores in the same band as our other production models. Best autonomous planner of anything we’ve run locally; strong structured reasoning/output.
Running it at the official sampling (temp 1.0 / top_p 1.0, thinking on) and found reasoning_effort: high beats max for agentic/tool work — max added latency and slightly regressed structured-output/safety.
Overall I’m super impressed. Early days, but this thing really does feel sonnet adjacent in chat. Big test will come next week as it replaces 122b in my hermes kanban profile roles.