Think you might want to check out myllmbox v5.2 — they seem to be onto something. That’s our highest score to date, with the fewest fails to date: 96/100, held across two independent full runs
The v30 vLLM in their image myllmbox/qwen38-flash-next-vllm:v5.2 (22.2 GB) is a little lighter than @eugr b12x build. I ran a clean pure-docker-to-docker test, matched line-for-line to our live production config (recipe set to MTP=3, same pinned weights, no re-download):
- 96/100, same median turn as production, and notably: zero safety-critical failures in both runs — a tool-sequencing scenario that fails on our current stack passed on theirs, twice.
- The whole prefill path jumped: first-token latency ~20–25% better on short prompts, and long-prompt TTFT (4k/8k context) improved 20–28% — consistent with their advertised ~2,700 tok/s prefill claim. Decode throughput ~3% lower — a fair trade for the faster first token, especially on long-context work.
I also ran his full recommendation (MTP=7 dynamic draft depth + everything else as shipped): results came in off (92/100, slower). Re-tested with just the engine/method, fixed depth in — nearly identical to the runs above. So the win here is v30 + their engine patches, not the dynamic-depth feature. Worth a look…
Full recipe we ran (recipe.yaml, everything else exactly as shipped in v5.2 — our edits are in the vllm: section, marked ✏️):
server:
image: myllmbox/qwen38-flash-next-vllm:v5.2 # vLLM 0.30.0 + their patches (RecoverSSM, table reader, MTP draft)
docker_args: --security-opt seccomp=unconfined
cpuset: "5-9,15-19" # GB10 X925 cores only
model: azampatti/Qwen3.8-Flash-Next-125B-A5B-INT4-AutoRound # pinned to revision 14642741…
port: 8000
models_dir: ./models
cache_dir: ./cache # ~52G table map on first boot
env:
MBX_PLE_MMAP: "1"
MBX_PLE_MMAP_MODE: "nvme"
MBX_PLE_MMAP_PREWARM: "0"
VLLM_USE_BREAKABLE_CUDAGRAPH: "1"
PYTHONUNBUFFERED: "1"
VLLM_MARLIN_USE_ATOMIC_ADD: "1"
TORCH_CUDA_ARCH_LIST: "12.1a"
FLASHINFER_CUDA_ARCH_LIST: "12.1a"
MALLOC_MMAP_THRESHOLD_: "65536"
MALLOC_TRIM_THRESHOLD_: "131072"
vllm:
served-model-name: qwen3.8-flash-next # ✏️ was Qwen/Qwen3.8-Flash-Next — set this to match YOUR client id or it breaks
engram-config: '{"cpu_offload": false}'
gdn-prefill-backend: triton
moe-backend: marlin # (run.sh drops this automatically for INT4-AutoRound)
distributed-executor-backend: mp
gpu-memory-utilization: 0.70
kv-cache-memory: "27000000000" # INT4-AutoRound profile: run.sh rewrites to 24G
load-format: fastsafetensors
max-model-len: 262144
block-size: 16 # ✏️ was 1632 (that's their K=7 sizing; correct for fixed K=3)
compilation-config: '{"cudagraph_mode":"PIECEWISE","cudagraph_capture_sizes":[1,2,3,4,8,12,16,20,24,28,32,36,40,44,48,52,56,60,64],"compile_ranges_endpoints":[32]}'
max-num-seqs: 16
max-num-batched-tokens: 8192
enable-prefix-caching: true
enable-auto-tool-choice: true
tool-call-parser: qwen3_coder # ✏️ was qwen3_xml — matched to our fleet parser
reasoning-parser: qwen3
chat-template: ".../medium_chat_template.jinja" # ✏️ pinned our production template
speculative-config: '{"method":"mtp","num_speculative_tokens":3,"rejection_sample_method":"block","draft_sample_method":"probabilistic"}'
# ✏️ fixed K=3, not dynamic 3..7
async-scheduling: true
use-replayssm: true
mtp_depth:
mode: off # ✏️ was dynamic — this is the change vs his shipped default
Launch is just:
git clone https://github.com/myllmbox/qwen38-flash-next-recipe.git && cd qwen38-flash-next-recipe
# apply the edits above, then:
./run.sh # first boot builds the ~52G table map (few extra minutes, once); healthy ~4 min
Tip if you run it: set served-model-name to whatever id your clients expect (see above — ours broke until we aligned it).
