The new V3 image with further long context optimization and fixes is now live:
From llama-benchy (coherent corpus, tg=1500, up to 300K context). General-purpose corpus, so DSpark acceptance is on the lower side:
| test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|---|---|---|---|---|---|
| pp2048 @ d4000 | 675.10 | — | 8,260.59 | 8,256.52 | 8,260.59 |
| tg1500 @ d4000 | 19.82 | 35.00 | — | — | — |
| pp2048 @ d50000 | 748.43 | — | 62,921.28 | 62,917.21 | 62,921.28 |
| tg1500 @ d50000 | 20.09 | 39.00 | — | — | — |
| pp2048 @ d100000 | 712.79 | — | 129,595.93 | 129,591.86 | 129,600.92 |
| tg1500 @ d100000 | 17.41 | 37.00 | — | — | — |
| pp2048 @ d200000 | 657.64 | — | 277,772.00 | 277,767.93 | 277,780.66 |
| tg1500 @ d200000 | 15.66 | 33.00 | — | — | — |
| pp2048 @ d300000 | 607.73 | — | 449,404.11 | 449,400.05 | 449,417.24 |
| tg1500 @ d300000 | 14.85 | 28.00 | — | — | — |
Single-stream coding benchmark (temp=0, prompt 125 tok, tg=1500) — coding workloads see higher DSpark acceptance than the coherent corpus above:
Completion tokens : 1500
Prompt tokens : 125
Wall time : 59.09s
Decode tok/s : 25.38
What’s new in v3:
- Faster decode: ~+5-8% vs V2 .
- Long-context loop fixed: the NaN/generation-loop bug observed past ~270K context under DCP is fixed by keeping the recurrent KDA token-position cache unsharded under DCP — a fix found and upstreamed by the
local-inference-lab/vllmproject as PR #418. Credit for identifying and fixing this issue goes to that repository’s authors. - 2M total context capacity: the TP16+DCP8 recipe holds ~2M tokens of KV — 4 concurrent requests can each use a 500K context.