Been running a two-box Spark cluster on V4 Flash for a while and finally got the
config somewhere worth writing down. Posting the whole thing because most of what
I needed was scattered across this category in pieces.
Hardware
2x DGX Spark, GB10, 128 GB unified each. ConnectX-7 NIC-to-NIC, no switch.
Ubuntu 24.04 aarch64, driver 580.173.02, CUDA 13.0, compute capability 12.1.
Stack
vLLM 0.20.0.dev off the jasl SM12x fork, Ray for two-node tensor parallel, TP=2.
Model is DeepSeek-V4-Flash-0731 (the GA checkpoint, came off the preview
yesterday) with the 3-stage DSpark drafter. OpenAI-compatible endpoint.
The flags that matter:
--max-model-len 400000
--max-num-seqs 4
--gpu-memory-utilization 0.8556
--kv-cache-dtype fp8 # fp8_ds_mla
--block-size 256
--enable-expert-parallel
--enable-prefix-caching
--tensor-parallel-size 2
--distributed-executor-backend ray
--speculative-config {"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic"}
--compilation-config {"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}
--max-cudagraph-capture-size 24
What it delivers
KV pool 1,760,985 tokens, max concurrency 4.40x at 400K per request. Weights load
79.85 / 79.88 GiB per rank.
Numbers below are off an otherwise-idle box, 600-token generations, measured
client-side over the streaming API with stream_options.include_usage for exact
token counts. Worth doing it that way: deltas carry multiple tokens per chunk
under spec decoding, so if you count SSE chunks you’ll undercount badly.
single stream 40.4 tok/s decode (37.7 / 40.1 / 43.3 across 3 runs)
TTFT 0.13-0.15s
4 concurrent 66.7 tok/s aggregate, wall clock
17-22 tok/s per stream, TTFT up to 0.80s
prefill ~3,000 tok/s peak
vLLM’s own generation-throughput line agrees: 29.7-40.1 at Running: 1, and 67.7
then 75.5 at Running: 4. So call it ~70 aggregate at four in flight. The
server-reported figure runs a touch above client wall-clock because it doesn’t
count TTFT or scheduling, which is worth knowing when you compare against numbers
posted in this category.
Spec decode acceptance is the interesting part. On prose and reasoning output the
mean acceptance length sits around 3.0 of 5 drafted, so 37-45% average draft
acceptance. On dense technical output with unusual tokens I’ve watched it fall to
the mid-20s. Per-position decay is steep either way:
good case (44%): 0.741, 0.538, 0.411, 0.291, 0.241
bad case (24%): 0.569, 0.347, 0.188, 0.069, 0.035
num_speculative_tokens=5 is probably one or two past what earns its keep at the
bad end. Haven’t swept it yet.
Two things that cost me time
gpu_memory_utilization and the cudagraph estimator interact in a way that isn’t
obvious. I’d been running VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 to recover
about 1.2 GiB per rank, at the cost of vLLM warning that cudagraph memory is no
longer accounted for during KV allocation. Turning the estimator back on and
letting vLLM tell me the equivalent util was better. It prints a per-rank
suggestion, and the two ranks want different offsets: head said 0.8526, worker
said 0.8556. Take the larger, because the pool is min(rank0, rank1) and the worker
is the binding rank. Pool went 1,735,825 to 1,760,985 and the warning went away.
Don’t extrapolate that number off someone else’s boot, mine included. Read it off
your own.
The other one is container images. I’d been building on each box separately, which
leaves you two functionally identical images with different IDs and a drift problem
waiting to happen. docker save piped over the NIC-to-NIC link into docker load on
the other box took about four and a half minutes, zero downtime, since it only
takes effect on the next container recreate. Build once, distribute, never build
per-box.
Still on the list
The GB10 FP8 kernel config for N=4096,K=12288 doesn’t exist, so every boot logs
“Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal!”.
Haven’t generated it yet.
B12X MoE is the decode lever I haven’t pulled. Running Marlin now. That one needs
a rebuild, not a flag flip.
Happy to answer questions on any of it.