Running Mistral Small 4 119B NVFP4 on NVIDIA DGX Spark (GB10)
Status: Working as of 2026-03-17
Hardware: ASUS Ascent GX10 / NVIDIA GB10 Grace Blackwell Superchip
GPU: Blackwell SM 12.1a (NVFP4 capable)
RAM: 128GB LPDDR5x unified memory
OS: Ubuntu 24.04 ARM64
To my knowledge, this is the first confirmed working configuration for Mistral Small 4 119B NVFP4 on the DGX Spark platform. The model was released March 16, 2026 and was confirmed running the next day. Community reports at the time indicated nobody else had it working, and the model author (eugr) himself said “hope vLLM will run it soon.”
The Problem
mistralai/Mistral-Small-4-119B-2603-NVFP4 uses a Multi-head Latent Attention (MLA) architecture with:
-
kv_lora_rank=256 -
qk_nope_head_dim=64 -
qk_rope_head_dim=64 -
v_head_dim=128 -
Effective KV head size = kv_lora_rank + qk_rope_head_dim = 320
Standard vLLM MLA backends (Flash Attention, XFormers, etc.) reject head_size=320. The Triton MLA backend claims to support all compute capabilities but in practice still rejects this head size. This means native MLA cannot be used on SM 12.1a as of this writing.
Failed Approaches
| Attempt | Result |
|---|---|
| avarok v23 (stock) | FAIL — tekken.json tokenizer v15 not in mistral_common (tops out at v13) |
avarok v23 + pip install --upgrade mistral_common |
FAIL — head_size=320, use_mla=True → no valid MLA backend on SM121 |
pip install triton (3.6.0 already present) |
FAIL — Triton MLA still rejects head_size=320 |
| hellohal2064/vllm-dgx-spark-gb10:latest | FAIL — glibc TLS crash in subprocess (dl-tls.c: _dl_add_to_slotinfo: Assertion 'idx == 0' failed), incompatible with Ubuntu 24.04 ARM host |
| vLLM 0.17.2rc1 built from source | FAIL — host torch (custom NVIDIA build) lacks cutlass_scaled_mm_supports_fp8 |
| max_model_len=65536 with VLLM_MLA_DISABLE=1 | FAIL — KV cache requires ~36GB, only ~23GB available after model weights |
| max_model_len=42000 with VLLM_MLA_DISABLE=1 | FAIL — ~100MB too short (23.07GB needed, 22.96GB available) |
The Solution
Disable MLA entirely (VLLM_MLA_DISABLE=1) and use standard attention with FLASH_ATTN + MARLIN NvFp4 MoE backend. This bypasses the head_size=320 problem by falling back to standard attention (head_size=128). The tradeoff is a larger KV cache footprint, which is why --max-model-len must be capped at 40000.
You also need to upgrade mistral_common inside the avarok v23 container to add tokenizer v15 support.
Prerequisites
-
NVIDIA DGX Spark / GB10 with 128GB unified memory
-
Docker with NVIDIA runtime
-
avarok/dgx-vllm-nvfp4-kernel:v23image pulled -
Model downloaded to a local path (e.g.
~/models/mistralai/Mistral-Small-4-119B-2603-NVFP4/) -
Qwen3-80B (or other large model) stopped — not enough RAM to run both simultaneously
Download the model
pip install huggingface_hub
huggingface-cli download mistralai/Mistral-Small-4-119B-2603-NVFP4 \
--local-dir ~/models/mistralai/Mistral-Small-4-119B-2603-NVFP4
Build the custom image
avarok v23 needs mistral_common upgraded for tokenizer v15 support:
FROM avarok/dgx-vllm-nvfp4-kernel:v23
RUN /opt/venv/bin/pip install --upgrade mistral_common
docker build -t avarok-mistral-small4:v1 - < Dockerfile
Working Docker Run Command
docker run -d \
--name mistral-small-4 \
--gpus all \
--runtime nvidia \
--shm-size=16g \
--ipc=host \
--entrypoint '' \
-v ~/models/mistralai/Mistral-Small-4-119B-2603-NVFP4:/model \
-e VLLM_MLA_DISABLE=1 \
-e VLLM_NVFP4_GEMM_BACKEND=marlin \
-e VLLM_USE_FLASHINFER_MOE_FP4=0 \
-e VLLM_TEST_FORCE_FP8_MARLIN=1 \
-e VLLM_ENGINE_CORE_STARTUP_TIMEOUT=300 \
-p 8002:8002 \
avarok-mistral-small4:v1 \
/bin/bash -c 'vllm serve --model /model \
--served-model-name mistral-small-4 \
--tokenizer-mode mistral \
--config-format mistral \
--load-format mistral \
--host 0.0.0.0 --port 8002 \
--tensor-parallel-size 1 \
--gpu-memory-utilization 0.75 \
--max-model-len 40000 \
--tool-call-parser mistral \
--enable-auto-tool-choice 2>&1'
Key Parameters Explained
| Parameter | Value | Why |
|---|---|---|
VLLM_MLA_DISABLE=1 |
1 | Critical — disables MLA, falls back to standard attention. Without this, all MLA backends reject head_size=320. |
VLLM_NVFP4_GEMM_BACKEND=marlin |
marlin | Use MARLIN NvFp4 MoE GEMM kernel — required for NVFP4 weight quantization |
VLLM_USE_FLASHINFER_MOE_FP4 |
0 | Disable FlashInfer MoE FP4 (use MARLIN instead) |
VLLM_TEST_FORCE_FP8_MARLIN |
1 | Force MARLIN path for FP8/NVFP4 operations |
--tokenizer-mode mistral |
mistral | Use Mistral-native tokenizer (required for tekken.json v15) |
--config-format mistral |
mistral | Use Mistral config format |
--load-format mistral |
mistral | Use Mistral weight loading |
--max-model-len 40000 |
40000 | Maximum safe value — 42000+ causes OOM on KV cache init with 128GB unified memory and model loaded |
--gpu-memory-utilization 0.75 |
0.75 | Leaves headroom for system + model weights |
--tool-call-parser mistral |
mistral | Enable Mistral-format tool calling |
--enable-auto-tool-choice |
— | Required for function/tool calling |
Startup Time & RAM Usage
-
Weight loading: ~7-8 minutes (13 safetensors shards, ~34s each)
-
RAM after startup: ~99GB used (model weights ~66GB + KV cache + system)
-
Free RAM: ~22GB with model loaded at 40k context
-
Throughput: ~27 tok/s sustained
Verification
# Check startup complete
docker logs mistral-small-4 2>&1 | grep "startup complete"
# Test API
curl -s http://localhost:8002/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "mistral-small-4",
"messages": [{"role": "user", "content": "Say hello."}],
"max_tokens": 50
}' | python3 -c "import sys,json; print(json.load(sys.stdin)['choices'][0]['message']['content'])"
Limitations
-
Context window hard-capped at 40,000 tokens — cannot be increased without exceeding available KV cache RAM
-
Max input ~28,000 tokens (40k minus 12k output reserve)
-
Cannot run simultaneously with Qwen3-80B — not enough unified memory for both
-
MLA disabled — running standard attention, not the intended MLA mode. Performance/quality may differ slightly from the model’s designed operation. Proper MLA support awaiting upstream vLLM kernel work.
-
Health check may show “unhealthy” — Docker health check can be overly strict; model responds correctly regardless
Notes on MLA
The root cause of MLA failure was investigated thoroughly. PixtralForConditionalGeneration in vLLM’s pixtral.py (used for both Pixtral vision models and Mistral Small 4) has no MLA wiring — it delegates to MistralAttention which uses standard head sizes. The DeepSeek V3 MLA delegation path that avarok uses was architecturally correct, but the Triton MLA backend’s support for head_size=320 on SM 12.1a was not functional at time of writing.
Once vLLM adds proper head_size=320 MLA support, VLLM_MLA_DISABLE=1 can be removed and --max-model-len can potentially be increased, since MLA significantly reduces KV cache size.
Tested on ASUS Ascent GX10 (GB10 Grace Blackwell), Ubuntu 24.04 ARM64, 2026-03-17