Huge thanks to @a3refaat for the b12x stack + the Sebesky GPTQ/EAGLE3 checkpoints — reproduced the **36 tok/s** text number *exactly* on our 2× DGX Spark (GB10) pair. The recipe notes say *“Vision is not tested or validated but should work in theory.”* We tested it — **it works, and it’s OCR-grade.** Sharing the result and the exact steps in case others want multimodal.
TL;DR
Dropping `–language-model-only` loads the full 32-layer ViT and the b12x backend handles the VL forward path correctly. On our pair:
| Config | Decode tok/s | Context | Vision |
|---|---|---|---|
| b12x text-only (a3refaat’s recipe) | **36.3** tg32 | up to ~196k | — |
| **b12x + vision (this post)** | **32.9** tg32 (peak 35.7) | ~98k (106k-tok pool) | ✅ OCR-grade |
**Vision costs only ~3 tok/s.** Prefill stays ~1000 tok/s. Coherent, consistent, no VL-specific crashes.
Quality — it’s not just “sees shapes”
Synthetic test (red square / blue circle / “SPARK” text): read all three, position + color correct, 3× consistent.
Then a text-heavy report image (title bar, three `$1,240,000`-style figures, a small bar chart, a footer). Asked for the title, the three values, and who prepared it:
```
Title: Quarterly Sales Report
Q1: $1,240,000 Q2: $1,890,000 Q3: $1,050,000
Prepared by: Finance Dept
```
Every string and every digit exact. The GPTQ W4A16 quantization does not visibly degrade vision.
How to enable it (deltas from the nvfp4-eagle3 recipe)
- **Remove `–language-model-only`** — that’s the whole switch; the ViT + patch-merge projector are in the Sebesky checkpoint (519 vision weight keys, 32-layer ViT, hidden 1280).
- **Add `–limit-mm-per-prompt ‘{“image”: 4, “video”: 0}’`.** The multimodal profiler reserves a *video* budget at load (~6 GB here) that you never use — turning video off gives it back to the KV pool. Without this, KV init fails (“No available memory for the cache blocks”).
- **Budget for the ViT (~4 GB).** With the tower loaded we run `gpu_memory_utilization 0.92` + `max_model_len 98304` → a **106,112-token pool** with vision. (Text-only can push util higher / context longer; the ViT is the tradeoff.)
Everything else is a3refaat’s recipe unchanged: `–attention-backend b12x`, `–kv-cache-dtype nvfp4`, EAGLE3 drafter (`Sebesky/MiniMax-M3-EAGLE3-RTN-INT4`, `num_speculative_tokens 3`), cudagraph `FULL_DECODE_ONLY`.
Two gotchas that bit us hard (both cross-node, both software)
Neither is hardware-specific — same GB10s as everyone — but they cost us hours, so:
-
**Warm the Triton cache on BOTH nodes, or the cross-node cudagraph deadlocks.** The b12x triton-prewarm mod didn’t cover every serving-shape kernel on our build (its exception is swallowed unless `B12X_TRITON_PREWARM_STRICT=1`), so `eagle_prepare_next`, `_nvfp4_write`, `update_regular_decode`, `rejection_sample` JIT-compiled *during* the first cudagraph replay. On a no-GPUDirect cross-node setup the worker’s cache started nearly empty while the head’s had them → the ranks desync mid-replay → `shm_broadcast: No available shared memory broadcast block found in 60 seconds` → EngineDead / garbage output. Fix: `rsync -a ~/.triton/ :~/.triton/` (identical sm_121 nodes → portable kernels). `/root/.triton` is a host mount, so once both are warm it stays fixed. **This was the source of every “garbage output” symptom we chased** — the model was fine.
-
**Check GPU clocks on BOTH nodes before trusting a low number.** We got 20 tok/s and nearly blamed the config — the head GPU was **power-wedged** (pinned 611 MHz / ~10 W under 95% load, *zero* active throttle reasons in `nvidia-smi -q -d PERFORMANCE`). On TP=2 the lockstep pair drags to the wedged clock. A **reboot does not clear it — a full AC power-cycle does** (unplug the USB-C PSU). After that: head back to ~2400 MHz, 20 → 36 tok/s instantly. If your cross-node number is suspiciously low, `nvidia-smi --query-gpu=clocks.sm --format=csv` on both nodes first.
(Also: the `fix-prometheus-fastapi-routing` mod hard-failed the launch on our image — package version differs; making its “block not found” branch a graceful skip instead of `SystemExit` fixed it. And on a head running a desktop, the util-0.93 free-mem check fails by ~0.5 GB — we set the nodes headless (`systemctl set-default multi-user.target`) to reclaim it.)
Net
For anyone who needs multimodal M3 on 2× Spark: this stack gives **~33 tok/s with OCR-grade vision at ~98k context**, which as far as we can tell is the fastest coherent M3-vision on a Spark pair by a wide margin. Thanks again @a3refaat — the vision was one flag away the whole time.
Happy to share our recipe/config diffs.
