Hello community,
Sharing my complete setup for running Mistral-Small-4-119B-2603-NVFP4 on a DGX Spark (GB10 / Blackwell) using SGLang with EAGLE speculative decoding. Several non-obvious GB10-specific pitfalls are documented here, including why the stable SGLang image fails and how memory constraints drive every parameter choice.
References
-
sgl-project/sglang#20708 — SM121A fix (merged in nightly)
Model Overview
Mistral Small 4 is a hybrid MoE model that unifies three capabilities in one checkpoint:
| Property | Value |
|---|---|
| Total parameters | 119B |
| MoE experts | 128 total, 4 active per token |
| Activated parameters per token | ~6.5B |
| Native context window | 256K tokens |
| Modality | Text + Image (multimodal) |
| Quantization used here | NVFP4 (~66 GB on-disk) |
| License | Apache 2.0 |
Why NVFP4? This quantization is Blackwell-native (B200/GB10) and fits in a single unified 128 GB memory pool — no tensor parallelism needed, unlike the FP8 version (~119 GB) which requires
tp=2on dual-H100 setups.
Prerequisites
Why the nightly Docker image?
The stable lmsysorg/sglang:mistral-small-4 image crashes on GB10 with a ptxas error on sm_121a. The fix landed in PR #20708 and is only available in CUDA 13 nightly builds compiled for Blackwell.
docker pull lmsysorg/sglang:nightly-dev-cu13-20260323-999bad5a
Model weights
hf download mistralai/Mistral-Small-4-119B-2603-NVFP4
hf download mistralai/Mistral-Small-4-119B-2603-eagle
The EAGLE draft model is only 390 MB (fp8) — negligible overhead for the latency gains it provides.
Memory Budget
Before choosing --mem-fraction-static and --context-length, the math:
| Component | Size |
|---|---|
| Main model weights (NVFP4) | ~66 GB |
| EAGLE draft model (fp8) | ~0.4 GB |
| CUDA graphs (bs 1–48) | ~1.95 GB |
| OS + ComfyUI headroom | ~32 GB |
| KV cache pool (0.75 × remainder) | ~29 GB |
With 29 GB of KV cache at context-length 65536, SGLang allocates ~910K total tokens across the pool. The model supports 256K natively, but that would require either sacrificing KV cache capacity or running out of memory — 65536 is the practical ceiling for this memory budget.
Launch Script
Parameter rationale — DGX Spark vs cookbook defaults
| Parameter | Cookbook (dual H100) | DGX Spark (GB10) | Reason |
|---|---|---|---|
--tp |
2 | 1 | Single unified 128 GB memory pool |
--attention-backend |
flashinfer | triton | Avoids flashinfer attention bugs on aarch64/SM121 |
--moe-runner-backend |
— | flashinfer_cutlass | Optimal sparse MoE routing on Blackwell |
--context-length |
131072 | 65536 | Memory constraint — see budget above |
load_format |
modelopt_fp4 | auto (mistral) | Compressed-tensors format auto-detected from checkpoint |
EAGLE speculative decoding settings:
| Parameter | Value | Effect |
|---|---|---|
--speculative-num-steps |
3 | Draft cycles per verification step |
--speculative-eagle-topk |
1 | Greedy top-1 — best latency, lowest overhead |
--speculative-num-draft-tokens |
4 | Tokens predicted ahead per step |
#!/bin/bash
docker run --gpus all -d --rm \
--name sglang-mistral-small-4 \
-p 30000:30000 \
-v /tmp:/tmp \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--ipc=host \
lmsysorg/sglang:nightly-dev-cu13-20260323-999bad5a \
python3 -m sglang.launch_server \
--model-path mistralai/Mistral-Small-4-119B-2603-NVFP4 \
--host 0.0.0.0 \
--port 30000 \
--tp 1 \
--attention-backend triton \
--moe-runner-backend flashinfer_cutlass \
--reasoning-parser mistral \
--tool-call-parser mistral \
--speculative-algorithm EAGLE \
--speculative-draft-model-path mistralai/Mistral-Small-4-119B-2603-eagle \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--mem-fraction-static 0.75 \
--context-length 65536 \
--served-model-name Mistral-Small-4
Boot Log — Key Events
Full boot log
========== == CUDA == ==========
CUDA Version 13.0.1
...
[2026-03-24 22:07:58] INFO model_config.py:1247: Downcasting torch.float32 to torch.float16.
[2026-03-24 22:07:58] WARNING model_config.py:1028: DeepGemm is enabled but the scale_fmt
of checkpoint is not ue8m0. This might cause accuracy degradation on Blackwell.
[2026-03-24 22:07:58] WARNING server_args.py:2938: Max running requests is reset to 48
for speculative decoding.
[2026-03-24 22:07:58] WARNING server_args.py:2959: Overlap scheduler is disabled when
spec v2 is off or using unsupported speculative algorithm.
[2026-03-24 22:07:59] INFO server_args.py:3084: Detected Mistral native format checkpoint,
setting load_format='mistral'
[2026-03-24 22:08:08] SM120 (Blackwell) detected: auto-selecting fp4-gemm-backend=flashinfer_cudnn
[2026-03-24 22:08:09] Using CompressedTensorsW4A4Nvfp4MoE
Loading safetensors checkpoint shards: 100% | 13/13 [01:12<00:00, 5.56s/it]
[2026-03-24 22:12:00] Load weight end. elapsed=230.84 s, type=PixtralForConditionalGeneration,
quant=compressed-tensors, avail mem=48.02 GB, mem usage=65.80 GB.
[2026-03-24 22:12:02] KV Cache is allocated. #tokens: 910897, KV size: 19.55 GB
[2026-03-24 22:12:03] Capture cuda graph begin. avail mem=28.02 GB
Capturing batches (bs=48): 23/23 [01:03<00:00, 2.77s/it]
[2026-03-24 22:13:08] Capture cuda graph end. Time elapsed: 65.47 s. mem usage=1.95 GB.
[2026-03-24 22:13:19] Load weight end (EAGLE draft). elapsed=5.67 s,
type=MistralLarge3ForCausalLMEagle, quant=fp8, mem usage=4.94 GB.
[2026-03-24 22:13:31] max_total_num_tokens=910897, context_len=65536,
available_gpu_mem=22.77 GB
[2026-03-24 22:13:36] INFO: Application startup complete.
[2026-03-24 22:13:36] INFO: Uvicorn running on http://0.0.0.0:30000
Warnings worth knowing about
⚠️ DeepGemm scale format mismatch
WARNING: DeepGemm is enabled but the scale_fmt of checkpoint is not ue8m0.
This might cause accuracy degradation on Blackwell.
The NVFP4 checkpoint uses a scale format different from what DeepGemm expects on Blackwell (ue8m0). This is not cosmetic — it can affect output quality. No workaround available yet; tracked upstream.
⚠️ Tokenizer incorrect regex
The tokenizer you are loading from 'mistralai/Mistral-Small-4-119B-2603-NVFP4'
with an incorrect regex pattern. Set fix_mistral_regex=True to fix this issue.
This warning appears 4 times during startup. It leads to incorrect tokenization. Fix by adding fix_mistral_regex=True when loading the tokenizer in client code.
⚠️ Overlap scheduler disabled
WARNING: Overlap scheduler is disabled when spec v2 is off or using unsupported
speculative algorithm. Set env SGLANG_ENABLE_SPEC_V2=True to enable.
Enabling SGLANG_ENABLE_SPEC_V2=True activates the experimental overlap scheduler, which can improve throughput further. Not tested in this setup.
✅ FP4 GEMM backend auto-selected
SM120 (Blackwell) detected: auto-selecting fp4-gemm-backend=flashinfer_cudnn
SGLang correctly identifies the GB10 as Blackwell and selects the optimal low-level GEMM kernel automatically.
✅ Multimodal architecture confirmed
type=PixtralForConditionalGeneration
The model loaded as a multimodal (text + image) model — Mistral Small 4 accepts image inputs natively through this architecture.
Performance Benchmarks
Metrics:
-
TTFT — Time To First Token (latency to first generated token)
-
TPOT — Time Per Output Token (average per-token generation time)
-
ITL — Inter-Token Latency (latency between consecutive tokens)
-
E2E — End-to-end request latency
-
Accept length — Average tokens accepted per EAGLE draft cycle (higher = more effective speculative decoding)
Single-user latency (concurrency=1)
python3 -m sglang.bench_serving \
--backend sglang \
--tokenizer mistralai/Mistral-Small-4-119B-2603-NVFP4 \
--max-concurrency 1 \
--num-prompts 20 \
--random-input-len 256 \
--random-output-len 128 \
--dataset-name random
Full results
============ Serving Benchmark Result ============
Backend: sglang
Max request concurrency: 1
Successful requests: 20
Benchmark duration (s): 58.48
Total generated tokens: 1566
Output token throughput (tok/s): 26.78
Concurrency: 1.00
Accept length: 2.30
--- End-to-End Latency ---
Mean E2E Latency (ms): 2923.17
Median E2E Latency (ms): 2825.40
P99 E2E Latency (ms): 5582.51
--- Time to First Token ---
Mean TTFT (ms): 73.08
Median TTFT (ms): 60.33
P99 TTFT (ms): 136.92
--- Time per Output Token ---
Mean TPOT (ms): 37.19
Median TPOT (ms): 38.06
P99 TPOT (ms): 51.02
--- Inter-Token Latency ---
Mean ITL (ms): 36.94
Median ITL (ms): 26.22
P95 ITL (ms): 78.42
P99 ITL (ms): 86.16
Max ITL (ms): 94.76
==================================================
73ms median TTFT for a 119B model is excellent. The
accept lengthof 2.30 means EAGLE is effectively reducing actual forward passes by ~2.3× at low concurrency — validating the speculative decoding configuration.
Scalability sweep (concurrency 1 → 16)
for CONCURRENCY in 1 2 4 8 16; do
echo ">>> max-concurrency=$CONCURRENCY"
python3 -m sglang.bench_serving \
--backend sglang \
--tokenizer mistralai/Mistral-Small-4-119B-2603-NVFP4 \
--max-concurrency $CONCURRENCY \
--num-prompts $((CONCURRENCY * 10)) \
--random-input-len 256 \
--random-output-len 128 \
--dataset-name random 2>/dev/null \
| awk '/={10,}/{found=1} found{print} /={10,}/{count++; if(count==2) exit}'
echo ""
done | tee scalability_results.log
| Concurrency | Output tok/s | Mean TTFT (ms) | Mean TPOT (ms) | Mean E2E (ms) | Accept length |
|---|---|---|---|---|---|
| 1 | 23.4 | 291 | 30.3 | 2165 | 2.02 |
| 2 | 36.5 | 244 | 51.6 | 4151 | 2.01 |
| 4 | 45.8 | 328 | 86.4 | 4691 | 1.96 |
| 8 | 59.9 | 379 | 129.3 | 7649 | 1.89 |
| 16 | 79.4 | 427 | 198.4 | 12345 | 1.81 |
Full raw results per concurrency level
>>> max-concurrency=1
============ Serving Benchmark Result ============
Output token throughput (tok/s): 23.44 | Accept length: 2.02
Mean TTFT: 290.88ms | Mean TPOT: 30.27ms | Mean E2E: 2164.87ms
==================================================
>>> max-concurrency=2
============ Serving Benchmark Result ============
Output token throughput (tok/s): 36.52 | Accept length: 2.01
Mean TTFT: 243.69ms | Mean TPOT: 51.63ms | Mean E2E: 4151.22ms
==================================================
>>> max-concurrency=4
============ Serving Benchmark Result ============
Output token throughput (tok/s): 45.81 | Accept length: 1.96
Mean TTFT: 327.63ms | Mean TPOT: 86.39ms | Mean E2E: 4690.99ms
==================================================
>>> max-concurrency=8
============ Serving Benchmark Result ============
Output token throughput (tok/s): 59.90 | Accept length: 1.89
Mean TTFT: 378.83ms | Mean TPOT: 129.34ms | Mean E2E: 7649.27ms
==================================================
>>> max-concurrency=16
============ Serving Benchmark Result ============
Output token throughput (tok/s): 79.36 | Accept length: 1.81
Mean TTFT: 427.12ms | Mean TPOT: 198.39ms | Mean E2E: 12345.11ms
==================================================
Observations:
-
Throughput scales from 23 → 79 tok/s (×3.4) as concurrency goes 1 → 16, but gains diminish: each doubling adds less than the previous.
-
TTFT stays remarkably stable (290–427ms) across the full range — the KV cache pool and CUDA graph captures handle batching well.
-
EAGLE
accept lengthdegrades gracefully under load (2.02 → 1.81), still meaningful at concurrency=16. -
TPOT grows nearly linearly with concurrency, which is expected for a single-GPU setup without pipeline parallelism.
Steady-state throughput (concurrency=8, longer output)
python3 -m sglang.bench_serving \
--backend sglang \
--tokenizer mistralai/Mistral-Small-4-119B-2603-NVFP4 \
--max-concurrency 8 \
--num-prompts 80 \
--random-input-len 256 \
--random-output-len 32 \
--dataset-name random
Full results
============ Serving Benchmark Result ============
Backend: sglang
Max request concurrency: 8
Successful requests: 80
Benchmark duration (s): 39.83
Output token throughput (tok/s): 30.65
Peak output token throughput: 49.00
Accept length: 2.31
--- Time to First Token ---
Mean TTFT (ms): 640.58
Median TTFT (ms): 550.42
P99 TTFT (ms): 3080.02
--- Time per Output Token ---
Mean TPOT (ms): 225.56
P99 TPOT (ms): 640.33
--- Inter-Token Latency ---
Mean ITL (ms): 226.13
P99 ITL (ms): 952.05
Max ITL (ms): 2445.91
==================================================
Note: shorter
--random-output-len 32with the same concurrency=8 shows higher TTFT variance (P99: 3080ms) compared to the scalability sweep (P99: 852ms at output-len 128). Shorter outputs mean more scheduling churn — the server spends proportionally more time on prefill overhead.
Bug: Triton JIT Crash at Concurrency ≥ 32
At concurrency=32, the server crashes with:
RuntimeError: Triton Error [CUDA]: operation not permitted
eagle_worker.py → draft() → _draft_preprocess_decode()
→ assign_draft_cache_locs (Triton kernel)
→ triton compiler → load_binary()
This is a race condition between the Triton JIT compiler and the CUDA context under extreme load. Three mitigations, ordered by impact on performance:
Option 1 — Enable SGLANG_ENABLE_SPEC_V2 (recommended first try)
The experimental spec v2 overlap scheduler may handle the concurrency better. Add to your docker run env:
-e SGLANG_ENABLE_SPEC_V2=True
Option 2 — Cap concurrency at 16
The crash is consistently reproducible at 32. Concurrency=16 is stable and delivers 79 tok/s output throughput — a reasonable ceiling for a single-GPU agentic workload.
Option 3 — Disable CUDA Graph
CUDA graph captures and Triton JIT interact poorly under heavy speculative decoding load. Adds overhead but eliminates the race:
--disable-cuda-graph
Option 4 — Switch attention backend to flashinfer
Eliminates all Triton attention kernels. May introduce other issues on aarch64/SM121 (flashinfer attention has known bugs on GB10), but worth testing if spec v2 doesn’t solve it:
--attention-backend flashinfer # instead of triton
Agentic Usage Tips
Mistral Small 4 unifies instruct, reasoning, and agentic capabilities in one checkpoint. Reasoning effort is configurable per request — no server restart needed.
Recommended pattern for agentic pipelines:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
# Planning / analysis tasks — reasoning ON
response = client.chat.completions.create(
model="Mistral-Small-4",
messages=[{"role": "user", "content": "Analyze this codebase and create a refactoring plan."}],
extra_body={"reasoning_effort": "high"},
)
# Execution tasks — reasoning OFF (faster, lower TPOT)
response = client.chat.completions.create(
model="Mistral-Small-4",
messages=[{"role": "user", "content": "Call the search_files tool with pattern='*.py'."}],
extra_body={"reasoning_effort": "none"},
)
Disabling reasoning on execution/tool-call steps significantly reduces TPOT and E2E latency — the model doesn’t need to think about calling a function it’s already been instructed to call. Reserve
reasoning_effort: highfor planning, analysis, and ambiguous decisions.
Tool calling is enabled via --tool-call-parser mistral and works with the standard OpenAI function calling format.
Would you like prompt testing cases and tool/skill calling examples? Happy to share those as a follow-up.
Cheers — may the tokens be with you! 🚀
William