Inkling-NVFP4 (975B) on 8× DGX Spark

We deployed the thinkingmachines/Inkling-NVFP4 (975B MoE, 41B active) on our 8× DGX Spark cluster (TP=8). The model now serves stably.

The model crashed under inference load with CUBLAS_STATUS_INTERNAL_ERROR. Root cause and validated fixes are documented in vLLM Issue #49049 and PR #49681. We applied the PR #49681 fixes to our Docker image and the crashes are resolved.

Stack

  • 8× DGX Spark (GB10, sm_121a, 128 GB UMA each), CX-7 interconnect

  • sparkrun v0.3.4

  • eugr/spark-vllm-docker nightly-20260810 — vLLM 0.26.1rc1.dev553, CUDA 13.0, FlashInfer 0.6.18

  • eugr inkling-sm12-paged-kv mod + instanttensor-hybrid-draft-loader mod

Deployment

  • 8× DGX Spark, driver 580.173.02, kernel 6.17.0-1029-nvidia

  • TP=8 / Ray, gpu_memory_utilization 0.80, max_model_len 262144, max_num_seqs 4, MTP k=1

  • KV cache: BF16

  • KV cache: 2,146,384 tokens → 8.19× max concurrency at 262K context

Results

Benchmark (CordatusAI LLM Benchmark Tool, 128 in / 128 out, 10 rounds per level):

Concurrency TTFT avg (ms) TPS avg (tok/s)
1 412 27.3
2 557 21.0
4 778 15.5

≈ 62 tok/s aggregate at C=4.

Recipe

recipe_version: "1"
name: Inkling-NVFP4-TP8
description: thinkingmachines/Inkling-NVFP4 (975B MoE, 41B active) on 8x DGX Spark

model: thinkingmachines/Inkling-NVFP4
cluster_only: true
min_nodes: 8
max_nodes: 8
container: registry.cordata.ai/spark-cluster/vllm-node-inkling-fixed:latest

mods:
  - mods/drop-caches

defaults:
  port: 8000
  host: 0.0.0.0
  tensor_parallel: 8
  gpu_memory_utilization: 0.80
  max_model_len: 262144
  max_num_seqs: 4
  num_speculative_tokens: 1

env:
  VLLM_USE_V2_MODEL_RUNNER: "1"
  FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED: "1"
  LAMPORT_RS_SCONV: "0"
  VLLM_USE_BREAKABLE_CUDAGRAPH: "0"
  VLLM_USE_AOT_COMPILE: "1"
  VLLM_WORKER_MULTIPROC_METHOD: "spawn"
  RAY_memory_usage_threshold: "0.99"
  RAY_memory_monitor_refresh_ms: "0"

command: |
  vllm serve thinkingmachines/Inkling-NVFP4 \
    --host {host} \
    --port {port} \
    --trust-remote-code \
    --tokenizer-mode inkling \
    --tensor-parallel-size {tensor_parallel} \
    --distributed-executor-backend ray \
    --enable-auto-tool-choice \
    --tool-call-parser inkling \
    --reasoning-parser inkling \
    --gpu-memory-utilization {gpu_memory_utilization} \
    --max-model-len {max_model_len} \
    --max-num-seqs {max_num_seqs} \
    --load-format auto \
    --quantization nvfp4 \
    --kernel-config.enable_flashinfer_autotune=False \
    --speculative-config '{"method":"mtp","num_speculative_tokens":{num_speculative_tokens}}'

Launch:

sparkrun run ~/inkling-nvfp4-tp8-fixed.yaml --cluster default --tp 8 --no-follow

Summary

Inkling-NVFP4 boots cleanly on 8× DGX Spark but crashes under inference load with CUBLAS_STATUS_INTERNAL_ERROR. Fixes from PR #49681 (closed unmerged) resolve the crashes.

Thanks to the Issue #49049 reporter, PR #49681 author, eugr_nv for the nightly images + mods, and the sparkrun crew.

Have you measured any use case where the large Inkling model is worth the extra size compared to Inkling-Small? They are surprisingly close in almost all benchmarks I’ve seen.

We’ve run both Inkling-Small on 2 nodes and the full model on 8 nodes. We haven’t done a controlled capability comparison between the two yet, so I can’t point to specific use cases that justify the additional compute required by the large model. That said, with other model families we’ve tested and used extensively, we’ve found that relatively small benchmark differences do not always translate to equally small differences in real-world workflows. The gap can become more apparent in longer, multi-step tasks involving extended context, iterative reasoning, or chained tool calls, where small differences in reliability or quality seem to accumulate.