Modal just released a new MTP / speculative decoding model for Qwen3.5-122B-A10B:
I set it up so you can run it with albond’s optimizations for the Spark on the latest vLLM, and tested that it enables 80+ tok/s decode on my actual Hermes Agent prompt history.
TLDR: A one-shot Docker/vLLM installer that runs Qwen3.5-122B-A10B on a single Spark with DFlash block-speculative decoding. On agent/tool-call traffic it sustains ~81 tok/s single-stream, and on @albond’s own end-to-end benchmark it clears his published number unpatched (59 vs 51 tok/s).
This builds directly on @albond’s excellent INT4 + MTP recipe
That work is the foundation. What’s new here is swapping the speculative-decode path to DFlash and re-stacking the community patches to run on current vLLM (0.23) with DFlash support, plus memory/load tuning for the Spark’s unified memory.
Headline results
Single-stream (c=1), temp 0, GB10. The comparison row uses @albond’s exact e2e
harness (completion_tokens / total wall-clock incl. prefill, cross-prompt mean of
5 prompts, run-1 discarded as warmup) so it’s apples-to-apples:
| Config | tok/s | Notes |
|---|---|---|
| Plain INT4, no spec (baseline) | 28.2 | airtight |
| @albond’s full patched MTP-2 stack (published) | 51.58 | INT4+FP8 + int8-lmhead + MTP-2 |
| DFlash n=12, unpatched (albond harness) | 53.7 | accept len 6.5, no extra patches |
| DFlash n=12 + dense levers (albond harness) | 59.0 | + hybrid FP8 + int8 lm-head |
| DFlash n=12, real agent turns (Hermes) | ~81 | 73% tool-calls, accept len ~8.3 |
Why DFlash wins on agent traffic: it block-drafts ~12 tokens in one parallel
forward, where MTP-N needs N sequential head passes (so MTP-2 saturates at accept
length ~3). On predictable tool-call/code output, DFlash acceptance climbs to 8–12,
which MTP structurally can’t match without paying N forwards.
What’s in the stack
- Target:
Intel/Qwen3.5-122B-A10B-int4-AutoRound(INT4, bf16 KV) - Drafter:
z-lab/Qwen3.5-122B-A10B-DFlash(0.8B block-diffusion, non-causal → FA2) - Engine: vLLM 0.23 on AEON-7’s sm121 DFlash image
(ghcr.io/aeon-7/aeon-vllm-ultimate) - Optional dense levers (default profile): hybrid INT4 routed + FP8 shared
experts and an int8 W8A16 lm-head GEMV — prebuilt hybrid checkpoint hosted at
bleysg/Qwen3.5-122B-A10B-int4-fp8-hybridso there’s no per-user rebuild.
The key DFlash-on-122B fix is a scale-block KV unify patch + prefix-caching off
(the hybrid GDN/mamba + attention KV page geometry otherwise rejects the drafter’s
attention spec). Details in docs/FINDINGS.md.
Memory & context
Tuned to reserve ~14 GB for the OS even under load:
| value | |
|---|---|
| gpu-memory-utilization | 0.82 (0.87+ over-subscribes → swaps → stalls) |
| max context | 262,144 (full native), single stream can reach it |
| KV pool (dense) | 426,610 tokens — 1.63× a full 262k context |
| KV pool (dflash) | 456,664 tokens |
| free under peak 3-stream load | ~15 GiB |
Comfortably fits a realistic agent layout: one ~100k main thread + up to ~3
subagents <100k each.
Concurrency is additive (dense, decode-only tok/s per stream):
| Workload | 1 | 2 | 3 |
|---|---|---|---|
| prose | 48.8 | 38.4 | 30.8 |
| code | 66.9 | 55.3 | 43.0 |
| agentic (6k tool-call ctx) | 121.8 | 98.9 | 66.7 |
Startup
Time-to-READY is ~3 min (vs ~12). --load-format fastsafetensors reads the
62–71 GiB straight to device in ~32 s instead of ~8 min — the default
mmap+per-tensor read + redundant CPU→GPU copy is pathologically slow on GB10’s
current kernel (and the copy is pointless on unified memory). Auto-falls-back to
nogds; no GPUDirect hardware needed.
Quick start
git clone https://github.com/Entrpi/qwen3.5-122B-A10B-on-spark
cd qwen3.5-122B-A10B-on-spark
./install.sh --start # dense profile by default; downloads the hosted hybrid
OpenAI-compatible server on :8000 (model name qwen), tool-calls + SSE.
Profiles: dense (default), dflash, base, mtp.
Caveats
- The always-on dense levers (FP8 shared experts, int8 lm-head) give ~+28% at
base decode but amortize to ~null on high-acceptance agent traffic (the verify
forward batches the dense reads across the accepted block). So for a pure agent
workload, plaindflashis essentially equal and gives a larger KV pool — the
dense profile is the better default for mixed/low-acceptance use.
Huge thanks to @albond (the recipe this builds on), z-lab (DFlash),
AEON-7 (sm121 DFlash vLLM build), Intel (AutoRound INT4), and Qwen.
Feedback and numbers from other Spark units very welcome.
