Update:
Correction: my decode-share numbers were wrong, and one of my conclusions was too
Following up on my original post: I re-reviewed the overlap benchmark, found three faults in how it measured decode share during prefill, repaired the instrument, and re-ran the full A/B.
One of the conclusions in my original post does not survive.
What was wrong with the instrument
1. SSE events were counted as tokens.
The original script timestamped streamed chunks with non-empty content and computed an event rate, then divided that by a token rate. On this speculative-decoding stack, one SSE chunk may contain several accepted tokens.
Measured on the same server:
tokens per chunk: {1: 31, 2: 36, 3: 25, 4: 14, 5: 6, 6: 6}
mean: 2.50 tokens per non-empty chunk
Counting SSE events was therefore not counting tokens.
2. There was no real prefill window.
The line intended to select intervals during prefill was effectively a no-op:
during = [g for g in gaps]
It included output before the long request was submitted and after its prefill had completed. This mixed the undisturbed head and tail of the ongoing stream into a measurement that was supposed to cover only the prefill window.
This turned out to be the dominant error.
3. The reference was cold and included TTFT.
The reference was the first request issued by the script and was measured over total request time. Warm, first-token-to-last-token decode measured 21.3 tok/s where the old method measured 17.1 tok/s.
The three faults had opposing effects, which is why a corrected result could not be inferred from the old data. It had to be re-measured.
Corrected results
The A/B used a 262K-token prompt and three repetitions per profile. The 2048 profile was also repeated three times after a server restart.
Each ongoing stream is now used as its own baseline: it runs undisturbed for 25 seconds before the long prefill is submitted. Tokens are counted directly from streamed token_ids and cross-checked against the server’s own usage.completion_tokens.
| Metric | Originally published | Corrected |
|---|---|---|
| Decode share during prefill, 8192 | 7.1% | 1.7% median (1.6 / 1.7 / 1.8) |
| Decode share during prefill, 2048 | 7.3% | 5.0% median (4.5–5.2 across six runs) |
| Prefill cost of 2048 | −3.9% | −7.7% (1,462 vs 1,584 tok/s) |
| p95 output-chunk gap, 8192 → 2048 | 5.20 → 1.59 s | 6.10 → 1.64 s |
Both profiles were then independently checked once more using the fully hardened instrument:
- 8192: 1.5%, 96 decode tokens, 1,563 tok/s prefill
- 2048: 5.1%, 348 decode tokens, 1,455 tok/s prefill
Every integrity guard passed in both verification runs: worker-thread exceptions were propagated, usage.completion_tokens was present and matched, admission was judged from actual timestamps, and the ongoing streams outlived the prefill windows with 27.5 and 31.5 seconds of margin.
The headline comparison remains the multi-run result:
- 8192: 1.7%
- 2048: 5.0%
- 2048 provides 2.9× the decode share during long prefill
The one-run hardened verification pair measured 3.4×. I am reporting that as independent support, not replacing the multi-run 2.9× result with it.
The conclusion that does not survive
I originally concluded:
Chunk size is a jitter lever, not a fairness lever.
That is wrong.
--max-num-batched-tokens 2048 delivered 2.9× the decode share and 3.1× the decode tokens of 8192 during the same long prefill.
The fairness benefit I originally measured as zero is actually the largest single effect of the setting.
For agent-serving workloads, this is the actionable finding:
--max-num-batched-tokensis both a jitter lever and a fairness lever.
At 8192, another request already mid-generation received about 0.6 tok/s while the long prefill ran. At 2048 it received about 1.9 tok/s.
Both figures represent severe starvation, but the difference between them is operationally significant.
Why it happens
My original reasoning assumed that decode would receive proportionally more tokens at the larger chunk size:
Four times as many scheduling opportunities at 2048, but approximately four times fewer decode tokens per opportunity, so the net effect should be zero.
The first half was right: decode appears to receive roughly one scheduling opportunity per prefill chunk.
The second half was wrong. A decode step yielded roughly the same small number of accepted speculative tokens regardless of prefill chunk size:
| 8192 | 2048 | |
|---|---|---|
| Prefill chunks for the 262K prompt | 32 | 128 |
| Decode tokens delivered | 106 | 332 |
| Decode tokens per prefill chunk | 3.3 | 2.6 |
The resulting model is:
decode_tokens ≈
(prefill_tokens / chunk_size)
× accepted_tokens_per_decode_step
Moving from 8192 to 2048 creates four times as many prefill chunks and therefore approximately four times as many decode scheduling opportunities. The measured improvement was 3.1× rather than 4× because the average accepted-token yield per step was slightly higher at 8192.
This model is falsifiable and predicts decode share should scale roughly with 1 / chunk_size until the accepted-token yield saturates.
Practical trade-off
For this stack, the 2048 profile:
- costs approximately 7.7% prefill throughput
- increases the reported KV pool from approximately 1.60M to 2.61–2.67M tokens, or about 63–67%
- reduces p95 output-chunk gaps by approximately 73%
- provides approximately 2.9× more decode throughput during a long prefill
The KV-pool value varied by about 2.3% across restarts on the same 2048 profile, so it should be treated as startup-dependent capacity rather than an exact deterministic constant.
For shared agent workloads, 2048 is a substantially better trade than my original post suggested.
What chunk size does not fix
New-request admission remained blocked in both profiles.
A short request submitted 20 seconds into the 262K prefill produced no output until the long prefill was nearly complete:
- approximately 145 seconds of waiting at 8192
- approximately 159 seconds at 2048
- normal warm TTFT: approximately 1.66 seconds
The difference mainly reflects the different prefill durations. Reducing the chunk size helps ongoing decode, KV capacity, and visible jitter, but it does not restore normal admission for new requests during a large prefill.
Agent-capture cache analysis also corrected
The original agent-capture analysis did not use the server’s real chat template. The saved requests contained 20 tool definitions, which made the approximation materially wrong.
Re-measured using the server’s actual template:
| Transition | Originally published | Corrected |
|---|---|---|
| turn 1 → 2 | 97.1% | 95.7% |
| turn 5 → 6 | 98.7% | 98.4% |
| turn 9 → 10 | 97.5% | 98.1% |
The append-only conclusion still holds: the first divergence occurs at the final token of the previous prompt in every transition.
What changed in the benchmark
The repaired instrument now includes:
- streamed
return_token_ids: true - hard failure if visible output is returned without token IDs
- token counting even when
delta.contentis empty - mandatory matching against
usage.completion_tokens - explicit submission-to-first-token prefill windows
- each ongoing stream used as its own undisturbed baseline
- output throughput separated from output-chunk jitter
- boundary-crossing gaps included in jitter measurements
- propagation of exceptions from all worker threads
- admission classification from actual submission timestamps
- verification that the ongoing stream covers the entire prefill window
- unique salting of every measurement prompt
- 27 regression tests in CI
The standalone token-ID probe demonstrates the original counting error directly:
non-empty content events : 95
tokens via token_ids : 200
usage.completion_tokens : 200
Counting SSE events would understate decode by 2.11× on this server.
A written measurement rule is not a guard. I had the correct rule documented and still violated it in one script. The cross-check against the server’s own usage accounting now makes that original failure mode impossible to publish silently.
Reproducibility
Release:
Full correction and affected/unaffected results:
Raw output from all eleven runs, including instrument revisions, profile-switch windows and verification runs:
The original v0.1.0 release remains unchanged for provenance.
I’ve been testing DeepSeek V4 Flash 0731 on two DGX Sparks using vLLM/DSpark, TP=2 and a direct RoCE connection.
There are already good posts here covering installation and peak coding speed, so I focused more on how the setup behaves as an actual interactive agent server: KV-cache capacity, concurrent requests, long prefills, prompt caching and GB10 unified-memory issues.
I’ve published the test scripts, configurations, full results and limitations here:
These results are from one patched DSpark deployment. The scheduling behavior has not yet been reproduced on clean upstream vLLM, and the 72-hour soak is still pending.
Changing max_num_batched_tokens
I compared 8192 and 2048 with the same model, runtime and hardware.
Metric
8192
2048
KV-cache pool
1.60M tokens
2.67M tokens (+67%)
p95 token gap during 256K prefill
5.20 s
1.59 s (−69%)
Prefill throughput
1,529 tok/s
1,469 tok/s (−3.9%)
Decode share during prefill
7.1%
7.3%
For my workload, 2048 was the better agent profile. It gave substantially more KV capacity and much smoother token delivery, while prefill performance dropped by only around 4%.
What surprised me was that smaller chunks did not improve scheduling fairness.
Long prefill still starves interactive traffic
During a 256K prefill, an existing decode stream continued in short bursts between the prefill chunks.
The token gaps closely followed:
chunk size / prefill throughput
Reducing the chunk size shortened the pauses from around five seconds to around 1.6 seconds, but decode still received only about 7% of the available capacity.
New short requests made almost no useful progress until the long prefill was nearly finished. In the overlap test, TTFT reached around 221× the normal warm TTFT.
This seems different from the intended decode-prioritization behavior described for vLLM chunked prefill. I have included a minimal reproduction in the repository. I plan to report it to the patched DSpark stack first and only take it upstream if it can also be reproduced without the DSpark patches.
The practical result for me is simple: very long contexts are treated as exclusive batch jobs, not mixed with interactive traffic.
Prefill and decode scale differently
For a 32K prefill-dominated workload, aggregate throughput stayed almost flat from concurrency 1 to 6. More requests just divided the same prefill capacity.
Decode scaled much better:
Single-stream coding: approximately 62–68 tok/s
Four concurrent coding streams: approximately 126 tok/s aggregate
Draft acceptance remained around 71%
Four simultaneous streams were the best interactive compromise in my tests. Six streams added relatively little aggregate throughput while increasing latency.
Long-context retrieval
The 1M serving configuration booted successfully.
I tested retrieval at 32K, 128K, 512K and 900K:
12/12 needles recovered
no distractor hits
approximately 15.2 minutes TTFT at 900K
The model can genuinely retrieve information at very long context depths, but 512K–1M is batch capacity on this setup rather than something I would use interactively.
GB10 UMA memory accounting
I also reproduced vLLM issue #48140 on a separate single-node Laguna S 2.1 deployment.
Three boots reported:
4.22 GiB
5.18 GiB
5.34 GiB
available for KV cache. The 262K profile required 18.35 GiB and was refused every time, while the 32K profile required about 2.3 GiB and booted successfully.
This matches the documented GB10 issue where reclaimable Linux page cache is effectively treated as unavailable by the startup memory check.
The repository documents a working lower-context profile and two possible mitigation paths for 262K:
explicit, logged drop_caches before startup
a local UMA-aware memory-accounting patch
I have not yet completed the 262K causality test, so I’m not claiming that either workaround is verified on this deployment yet.
Prompt-cache locality
I also measured how the position of changing prompt content affects prefix-cache reuse.
Moving the same changing field from the bottom to the top increased per-turn cost by:
39.6× on DeepSeek/DSpark
13.7× on Laguna
The principle itself is already well known: stable content first, dynamic content last. The useful part for me was measuring it on these stacks and verifying the real client traffic.
The repository includes:
a wire-level capture proxy for OpenAI-compatible clients
a vLLM-aware prompt-locality analyzer using the server’s own tokenizer
exact first-divergence and invalidated-block reporting
a redacted mode that does not expose prompt contents
I tested it with two real coding-agent tasks. In the captured ten-turn bug-fix task, the client maintained 97–99% reusable prefix cache per turn, with the prompt changing only at the append-only tail.
So in this case the tool did not find a cache problem, which was still a useful result.
I’d be very interested to hear whether anyone else can reproduce the approximately 7% decode share during long prefill, especially with another DSpark image, another speculative-decoding implementation or unmodified upstream vLLM.