We deployed the thinkingmachines/Inkling-NVFP4 (975B MoE, 41B active) on our 8× DGX Spark cluster (TP=8). The model now serves stably.
The model crashed under inference load with CUBLAS_STATUS_INTERNAL_ERROR. Root cause and validated fixes are documented in vLLM Issue #49049 and PR #49681. We applied the PR #49681 fixes to our Docker image and the crashes are resolved.
Stack
-
8× DGX Spark (GB10, sm_121a, 128 GB UMA each), CX-7 interconnect
-
sparkrun v0.3.4
-
eugr/spark-vllm-docker nightly-20260810 — vLLM 0.26.1rc1.dev553, CUDA 13.0, FlashInfer 0.6.18
-
eugr
inkling-sm12-paged-kvmod +instanttensor-hybrid-draft-loadermod
Deployment
-
8× DGX Spark, driver 580.173.02, kernel 6.17.0-1029-nvidia
-
TP=8 / Ray,
gpu_memory_utilization 0.80,max_model_len 262144,max_num_seqs 4, MTP k=1 -
KV cache: BF16
-
KV cache: 2,146,384 tokens → 8.19× max concurrency at 262K context
Results
Benchmark (CordatusAI LLM Benchmark Tool, 128 in / 128 out, 10 rounds per level):
| Concurrency TTFT avg (ms) TPS avg (tok/s) | ||
|---|---|---|
| 1 | 412 | 27.3 |
| 2 | 557 | 21.0 |
| 4 | 778 | 15.5 |
≈ 62 tok/s aggregate at C=4.
Recipe
recipe_version: "1"
name: Inkling-NVFP4-TP8
description: thinkingmachines/Inkling-NVFP4 (975B MoE, 41B active) on 8x DGX Spark
model: thinkingmachines/Inkling-NVFP4
cluster_only: true
min_nodes: 8
max_nodes: 8
container: registry.cordata.ai/spark-cluster/vllm-node-inkling-fixed:latest
mods:
- mods/drop-caches
defaults:
port: 8000
host: 0.0.0.0
tensor_parallel: 8
gpu_memory_utilization: 0.80
max_model_len: 262144
max_num_seqs: 4
num_speculative_tokens: 1
env:
VLLM_USE_V2_MODEL_RUNNER: "1"
FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED: "1"
LAMPORT_RS_SCONV: "0"
VLLM_USE_BREAKABLE_CUDAGRAPH: "0"
VLLM_USE_AOT_COMPILE: "1"
VLLM_WORKER_MULTIPROC_METHOD: "spawn"
RAY_memory_usage_threshold: "0.99"
RAY_memory_monitor_refresh_ms: "0"
command: |
vllm serve thinkingmachines/Inkling-NVFP4 \
--host {host} \
--port {port} \
--trust-remote-code \
--tokenizer-mode inkling \
--tensor-parallel-size {tensor_parallel} \
--distributed-executor-backend ray \
--enable-auto-tool-choice \
--tool-call-parser inkling \
--reasoning-parser inkling \
--gpu-memory-utilization {gpu_memory_utilization} \
--max-model-len {max_model_len} \
--max-num-seqs {max_num_seqs} \
--load-format auto \
--quantization nvfp4 \
--kernel-config.enable_flashinfer_autotune=False \
--speculative-config '{"method":"mtp","num_speculative_tokens":{num_speculative_tokens}}'
Launch:
sparkrun run ~/inkling-nvfp4-tp8-fixed.yaml --cluster default --tp 8 --no-follow
Summary
Inkling-NVFP4 boots cleanly on 8× DGX Spark but crashes under inference load with CUBLAS_STATUS_INTERNAL_ERROR. Fixes from PR #49681 (closed unmerged) resolve the crashes.
Thanks to the Issue #49049 reporter, PR #49681 author, eugr_nv for the nightly images + mods, and the sparkrun crew.