Optimizing DeepSeek-V4-Flash on a single NVIDIA GB10/GX10 with DSpark speculative decoding

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10 Projects
, ,
22

@marco.palaferri Follow-up: concurrent decode at deep context (128k / 512k / 850k) on a single GB10

Companion to my earlier 8k multi-stream numbers — here’s how concurrent decode scales at deep context on a single GB10 (121 GB), build 61800f6, DeepSeek-V4-Flash IQ2XXS + Q2 DSpark sidecar.

Method. To isolate decode from prefill: warm N distinct deep contexts serially (turn 1), then fire N byte-identical follow-ups concurrently (turn 2) so KV warm-reuse skips the deep prefill and the timed window is decode only. Verification gate for every run, read from the server log: kv cache hit tokens == prompt tokens, zero prefill chunk lines in the measure window, zero cache evictions. Two things were needed to get there on the 1M profile:

  • Size the KV disk cache to hold all N contexts. Each deep context persists ~3 checkpoint entries (token-text + prefill-complete + thinking-visible) at 7–9 GB, so 3× 512k needs ~60–80 GB; the default 16 GB evicts them mid-run and forces re-prefill. DS4_KV_DISK_SPACE_MB=81920.
  • run-dspark-server-1m.sh hardcodes export DS4_KV_LONG_COLD_ANCHOR_TRIM_TOKENS=65536, which trims 64k tokens off each cached checkpoint and can’t be overridden by inline env — so warm reuse never covers the full context. I bypassed the wrapper and called run-dspark-server.sh directly with the 1M env plus DS4_KV_LONG_COLD_ANCHOR_TRIM_TOKENS=0.

Results (client-side aggregate; single-stream from the depth sweep, same build):

image
image1230×690 49.5 KB

test streams per-stream (server) aggregate single-stream ref
128k 3 ~4 / 4 / (fast tail) 10.1 tok/s 12.9
512k 3 4.1 / 4.6 / 7.3 6.2 tok/s ~9.4
850k 2 — OOM-killed ~14–21

Each clean run confirmed: 0 evictions, cache-hit == prompt tokens, 0 prefill chunks in the timed window.

Reading it:

  1. Concurrent decode holds up but does not scale up at depth. At 128k, 3 streams aggregate to ~10 tok/s vs 12.9 single (~20% overhead, fairly shared ~4 tok/s/stream). At 512k the overhead grows — 6.2 aggregate vs ~9.4 single. So adding streams at depth splits the available decode throughput (with some overhead) rather than multiplying it — consistent with decode being memory-bandwidth-bound at these depths, where the cohort coordinator logs coordinator=serial / requested_r=1 (it declines to physically co-batch very deep sessions).

  2. 850k × 2 is a hard resident-memory wall. Two concurrent 850k contexts don’t fit: warming the second while the first is resident drove unified memory to the limit — the second context’s prefill decelerated under pressure (222 → 176 t/s) and then the kernel OOM-killed the server:

    Out of memory: Killed process ds4-server total-vm:239803800kB oom_score_adj:1000

    (2× ~850k KV + the 80 GB model + the 850k working set > 121 GB. Single-stream 850k is fine.) The CUDA backend sets its own oom_score_adj=1000, so it’s the preferred kill target when the box is squeezed — a graceful-ish failure mode, but a wall.

Caveats: single-run points with a variable-length generation task, so treat them as approximate — they establish the shape (mild overhead → more overhead → OOM wall), not three-sig-fig precision. Single-stream deep decode on this build is itself run-variable (I’ve seen 512k anywhere from ~9 to ~24 tok/s depending on draft acceptance / HybridLC engagement). Happy to average repeats, try the lean/prefill-fast memory profiles (which might let 850k×2 fit by shrinking the working set), or share the full logs.