Running Thinking Machines Inkling model NVfp4

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Inkling Build

-I haven’t seen this anywhere else yet so I’ll start a thread. Took a little bit but we now have this running on a cluster of 8 sparks. This was designed for the B300 etc.. but it looks like there was some thought of it running on sm12x in the notes (the kernel comments name DGX Spark explicitly). Issues we found so far, will update as we go:

  1. scipy missing — new vLLM-main dependency, not in older CUDA-13 base images. FIX pip install scipy.
  2. Inkling’s Lamport collectives require MNNVL (NVLink fabric) — hard error on RoCE clusters. FIX TML shipped the escape hatch: LAMPORT_RS_SCONV=0
  3. tml_fa4’s Sm120 path has no paged-KV support yet — vLLM’s paged cache can’t feed it. We patch fa4_rel_attention to gather paged KV → contiguous per call , TEMP FIX
  4. Don’t route sm12x to the sheared/tml_fa4 rel-bias path — the Sm120 kernel there discards the relative-position bias (no bias parameter in the Sm80-inherited call). You’d get plausible-but-wrong outputs on every layer. FIX The score-mod vllm_flash_attn/cute path is the intended sm12x route.
  5. Two one-line bugs in the (never-before-compiled) cute Sm80/Sm120 base: flash_fwd.py references mDynamicCausal which isn’t a kernel parameter (→ NameError during DSL tracing; safe fix psc = None, dynamic_causal is SM90-only anyway), and self.is_split_kv is referenced but never assigned (→ = False in init).
  6. Phantom varlen work tiles → cudaErrorIllegalAddress. The SingleTileVarlenScheduler launches an upper-bound grid; Sm90 kernels check work_tile.is_valid_tile, the Sm80/Sm120 base doesn’t, so phantom tiles read cu_seqlens[num_batch+1] out of bounds. Verified with compute-sanitizer (invalid read decoded exactly to the first phantom block). One-line fix: remap phantom tiles to a real tile (duplicate compute is benign, is_valid stays false so Sm90+ unaffected).
  7. Warmup tracing: vLLM’s V2 model runner traces attention under FakeTensorMode — any .item() in your patch path needs a fake-tensor early-return.
  8. One remaining timing-dependent race, still unlocated: the cluster faults under deep async pipelines but runs clean when CUDA_ENABLE_COREDUMP_ON_EXCEPTION=1 (+ coredump flags) perturbs driver timing. We ship with the instrumentation on as a stabilizer (small overhead) until it’s found. The vLLM CUDA debugging blog ( CUDA Core Dump: An Effective Tool to Debug Memory Access Issues and Beyond | vLLM Blog ) workflow is what got us this far.

Lots of help from Claude/M3 etc… we are funnign though MTP1 draft acceptance is 60%

2

Thanks! Can you share your recipe and patches if any?

3

I will soon, — it’s been buggy but big progress: just got CUDA graphs working (was stuck on eager, root-caused the bug — a boundary bug in the attention kernel, found it via GPU coredump). NVFP4 running clean with no dtype fallbacks. Still stuck at MTP k=1. Numbers: c1 decode ~25–27 tok/s on the graphs config; prefill up to 2,711 tok/s on the throughput config (higher than we ever got M3); high concurrency much more stable than early runs.

4

Inkling-NVFP4 on our 8x DGX Spark cluster (GB10, sm_121a). We got it running clean and stable and fixed a couple of real kernel bugs, but decode speed falls off hard as context grows, so we’re parking it and staying on our M3 build. Full recipe, patches, and bug findings: GitHub - blockmos/inkling-sparks-gb10 · GitHub

Single-user decode holds up great on short prompts, then drops as context fills:

Full numbers (decode t/s):

┌──────────────────┬────────────────────┬────────────┬─────────────┐
│ context │ c1 │ c8 (total) │ c32 (total) │
├──────────────────┼────────────────────┼────────────┼─────────────┤
│ short (~100 tok) │ 25 (27 w/ MTP k=2) │ 80 │ 193 │
├──────────────────┼────────────────────┼────────────┼─────────────┤
│ 2048 tokens │ 13.5 │ 25 │ 24 │
└──────────────────┴────────────────────┴────────────┴─────────────┘

Prefill ~1,400 t/s at 2048. The cliff is because the sm_121a cute kernels have no paged-KV, so our workaround re-gathers the whole KV history every decode step — the engine caps ~24 t/s aggregate at real context no matter how many users. (Short numbers are our own bench, 2048 is llama-benchy.)

Recipe gotchas that cost us hours:

  • Pin “mode”: 0 in --compilation-config. Drop --enforce-eager without it and vllm goes full torch.compile and corrupts output.
  • gpu-memory-utilization 0.70 max. The missing 30% is page cache + ray + k8s + cuda runtime, not waste. 0.78 wedges the node.

Bugs (most useful part):

  1. Cute FlashAttention Sm120 kernel doesn’t clamp the rel-bias q-row index — illegal memory access at some shapes. Caught with GPU coredumps, byte-identical fault on 2 ranks = deterministic. Filed vllm#49049, one-line fix.
  2. KV write in fused_qkvr_prep races the attention read on a side stream. Move to main stream.
  3. On unified memory, out-of-range indexes usually DON’T crash — they land in another allocation and silently corrupt. Looked like a race for days.

Debug tip: CUDA_ENABLE_COREDUMP_ON_EXCEPTION + host-mounted CUDA_COREDUMP_FILE, then cuda-gdb. Names the exact faulting instruction.

nvfp4 itself is fine — no dtype fallbacks, clean output. It’s kernel maturity (paged KV, long-context, multi-depth MTP) that’s not there yet on sm_121a. M3 (42 t/s single user, scales with concurrency)

result: Public repo live at GitHub - blockmos/inkling-sparks-gb10 · GitHub (recipe + 12 patches + coredump bug report, scanned clean of all internal/business data), shareable write-up delivered with chart + short-token speeds, and M3 restored as the serving daily driver. Hopefully someone else can get us a little farther on this one