Hy3 (Tencent Hunyuan 3, 295B MoE / 21B active) is up on 2× DGX Spark — sharing the full recipe as a starting point**
Got the final Hy3 release serving on two GB10 Sparks (TP=2 over the 200GbE RoCE fabric) with its native MTP speculative decoding live, and — after some fighting — the hy_v3 tool-call + reasoning parsers working too. Posting everything so others can skip the landmines.
**Setup**
- Weights: kodelow/Hy3-NVFP4-W4A16 (181GB, final-base, MTP layer preserved, MARLIN-only by design)
- vLLM 0.23.1, Ray 2-node TP=2, NCCL over ConnectX-7 (MTU 9000)
- ~90GB weights/node + FP8 KV → ~180K-token pool, 128K max ctx, 6 seqs
- GB10 flags per LibertAI’s verified bring-up: `–moe-backend marlin --enforce-eager` (FlashInfer’s native-FP4 path freezes GB10s)
**Benchmarks** (512-tok gen, temp 0.9/top_p 1.0):
- Single stream: **21.8 tok/s**
- 6-way concurrent: **59.7 tok/s aggregate** (~10/stream)
- MTP draft acceptance: pos-1 62–76%, pos-2 ~20%
**Two findings worth knowing before you tune:**
1. **Run `num_speculative_tokens: 1`, not 2.** Pos-2 MTP acceptance is only ~20% on GB10 — spec-2 is a net ~30% LOSS. (The official recipe’s “2” is tuned for H200/GB300.)
2. **`–enforce-eager` WINS.** Turning on CUDA graphs / inductor compile cost ~25% throughput with the marlin W4A16 decode path on sm121/vLLM 0.23. Counter-intuitive, measured multiple times.
**Gotchas that cost us hours (all fixed):**
- Multi-node needs explicit `–distributed-executor-backend ray`
- GMU must be **0.90** — a GB10 only exposes ~111.4GiB free of 121.69 at boot, 0.92 fails
- **Tool-call/reasoning parser crash:** this checkpoint’s special tokens carry a `:opensource` suffix (`think:opensource`, `<tool_call:opensource>`, etc. — chat template sets HYTK=‘:opensource’), but vLLM’s hy_v3 parsers hardcode the bare forms → startup crash with parsers on, fake plain-text tool calls with them off. Fix: sed the two parser files to the suffixed tokens. Structured `tool_calls` verified working after (finish_reason: tool_calls).
- TP=3 is impossible (8 KV heads don’t divide by 3) — it’s a 2-Spark or 4-Spark model.
**Full recipe, scripts, and all the bugs documented:**
Meant as a starting point — take it, tune it, PR it. Next thing we’re chasing is NVIDIA’s vLLM 26.06 container (NVFP4 paged-KV) for the KV headroom. Credit to kodelow (quant) and LibertAI (the GB10 flags). Happy to answer questions.
