Running Mistral Small 4 (119B MoE) on DGX Spark with SGLang — Full Setup & Benchmarks

Hello community,

Sharing my complete setup for running Mistral-Small-4-119B-2603-NVFP4 on a DGX Spark (GB10 / Blackwell) using SGLang with EAGLE speculative decoding. Several non-obvious GB10-specific pitfalls are documented here, including why the stable SGLang image fails and how memory constraints drive every parameter choice.


References


Model Overview

Mistral Small 4 is a hybrid MoE model that unifies three capabilities in one checkpoint:

Property Value
Total parameters 119B
MoE experts 128 total, 4 active per token
Activated parameters per token ~6.5B
Native context window 256K tokens
Modality Text + Image (multimodal)
Quantization used here NVFP4 (~66 GB on-disk)
License Apache 2.0

Why NVFP4? This quantization is Blackwell-native (B200/GB10) and fits in a single unified 128 GB memory pool — no tensor parallelism needed, unlike the FP8 version (~119 GB) which requires tp=2 on dual-H100 setups.


Prerequisites

Why the nightly Docker image?

The stable lmsysorg/sglang:mistral-small-4 image crashes on GB10 with a ptxas error on sm_121a. The fix landed in PR #20708 and is only available in CUDA 13 nightly builds compiled for Blackwell.

docker pull lmsysorg/sglang:nightly-dev-cu13-20260323-999bad5a

Model weights

hf download mistralai/Mistral-Small-4-119B-2603-NVFP4
hf download mistralai/Mistral-Small-4-119B-2603-eagle

The EAGLE draft model is only 390 MB (fp8) — negligible overhead for the latency gains it provides.


Memory Budget

Before choosing --mem-fraction-static and --context-length, the math:

Component Size
Main model weights (NVFP4) ~66 GB
EAGLE draft model (fp8) ~0.4 GB
CUDA graphs (bs 1–48) ~1.95 GB
OS + ComfyUI headroom ~32 GB
KV cache pool (0.75 × remainder) ~29 GB

With 29 GB of KV cache at context-length 65536, SGLang allocates ~910K total tokens across the pool. The model supports 256K natively, but that would require either sacrificing KV cache capacity or running out of memory — 65536 is the practical ceiling for this memory budget.


Launch Script

Parameter rationale — DGX Spark vs cookbook defaults
Parameter Cookbook (dual H100) DGX Spark (GB10) Reason
--tp 2 1 Single unified 128 GB memory pool
--attention-backend flashinfer triton Avoids flashinfer attention bugs on aarch64/SM121
--moe-runner-backend flashinfer_cutlass Optimal sparse MoE routing on Blackwell
--context-length 131072 65536 Memory constraint — see budget above
load_format modelopt_fp4 auto (mistral) Compressed-tensors format auto-detected from checkpoint

EAGLE speculative decoding settings:

Parameter Value Effect
--speculative-num-steps 3 Draft cycles per verification step
--speculative-eagle-topk 1 Greedy top-1 — best latency, lowest overhead
--speculative-num-draft-tokens 4 Tokens predicted ahead per step
#!/bin/bash
docker run --gpus all -d --rm \
  --name sglang-mistral-small-4 \
  -p 30000:30000 \
  -v /tmp:/tmp \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --ipc=host \
  lmsysorg/sglang:nightly-dev-cu13-20260323-999bad5a \
  python3 -m sglang.launch_server \
    --model-path mistralai/Mistral-Small-4-119B-2603-NVFP4 \
    --host 0.0.0.0 \
    --port 30000 \
    --tp 1 \
    --attention-backend triton \
    --moe-runner-backend flashinfer_cutlass \
    --reasoning-parser mistral \
    --tool-call-parser mistral \
    --speculative-algorithm EAGLE \
    --speculative-draft-model-path mistralai/Mistral-Small-4-119B-2603-eagle \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --mem-fraction-static 0.75 \
    --context-length 65536 \
    --served-model-name Mistral-Small-4


Boot Log — Key Events

Full boot log
========== == CUDA == ==========
CUDA Version 13.0.1
...
[2026-03-24 22:07:58] INFO model_config.py:1247: Downcasting torch.float32 to torch.float16.
[2026-03-24 22:07:58] WARNING model_config.py:1028: DeepGemm is enabled but the scale_fmt
  of checkpoint is not ue8m0. This might cause accuracy degradation on Blackwell.
[2026-03-24 22:07:58] WARNING server_args.py:2938: Max running requests is reset to 48
  for speculative decoding.
[2026-03-24 22:07:58] WARNING server_args.py:2959: Overlap scheduler is disabled when
  spec v2 is off or using unsupported speculative algorithm.
[2026-03-24 22:07:59] INFO server_args.py:3084: Detected Mistral native format checkpoint,
  setting load_format='mistral'
[2026-03-24 22:08:08] SM120 (Blackwell) detected: auto-selecting fp4-gemm-backend=flashinfer_cudnn
[2026-03-24 22:08:09] Using CompressedTensorsW4A4Nvfp4MoE
Loading safetensors checkpoint shards: 100% | 13/13 [01:12<00:00,  5.56s/it]
[2026-03-24 22:12:00] Load weight end. elapsed=230.84 s, type=PixtralForConditionalGeneration,
  quant=compressed-tensors, avail mem=48.02 GB, mem usage=65.80 GB.
[2026-03-24 22:12:02] KV Cache is allocated. #tokens: 910897, KV size: 19.55 GB
[2026-03-24 22:12:03] Capture cuda graph begin. avail mem=28.02 GB
Capturing batches (bs=48): 23/23 [01:03<00:00,  2.77s/it]
[2026-03-24 22:13:08] Capture cuda graph end. Time elapsed: 65.47 s. mem usage=1.95 GB.
[2026-03-24 22:13:19] Load weight end (EAGLE draft). elapsed=5.67 s,
  type=MistralLarge3ForCausalLMEagle, quant=fp8, mem usage=4.94 GB.
[2026-03-24 22:13:31] max_total_num_tokens=910897, context_len=65536,
  available_gpu_mem=22.77 GB
[2026-03-24 22:13:36] INFO: Application startup complete.
[2026-03-24 22:13:36] INFO: Uvicorn running on http://0.0.0.0:30000

Warnings worth knowing about

⚠️ DeepGemm scale format mismatch

WARNING: DeepGemm is enabled but the scale_fmt of checkpoint is not ue8m0.
This might cause accuracy degradation on Blackwell.

The NVFP4 checkpoint uses a scale format different from what DeepGemm expects on Blackwell (ue8m0). This is not cosmetic — it can affect output quality. No workaround available yet; tracked upstream.

⚠️ Tokenizer incorrect regex

The tokenizer you are loading from 'mistralai/Mistral-Small-4-119B-2603-NVFP4'
with an incorrect regex pattern. Set fix_mistral_regex=True to fix this issue.

This warning appears 4 times during startup. It leads to incorrect tokenization. Fix by adding fix_mistral_regex=True when loading the tokenizer in client code.

⚠️ Overlap scheduler disabled

WARNING: Overlap scheduler is disabled when spec v2 is off or using unsupported
speculative algorithm. Set env SGLANG_ENABLE_SPEC_V2=True to enable.

Enabling SGLANG_ENABLE_SPEC_V2=True activates the experimental overlap scheduler, which can improve throughput further. Not tested in this setup.

✅ FP4 GEMM backend auto-selected

SM120 (Blackwell) detected: auto-selecting fp4-gemm-backend=flashinfer_cudnn

SGLang correctly identifies the GB10 as Blackwell and selects the optimal low-level GEMM kernel automatically.

✅ Multimodal architecture confirmed

type=PixtralForConditionalGeneration

The model loaded as a multimodal (text + image) model — Mistral Small 4 accepts image inputs natively through this architecture.


Performance Benchmarks

Metrics:

  • TTFT — Time To First Token (latency to first generated token)

  • TPOT — Time Per Output Token (average per-token generation time)

  • ITL — Inter-Token Latency (latency between consecutive tokens)

  • E2E — End-to-end request latency

  • Accept length — Average tokens accepted per EAGLE draft cycle (higher = more effective speculative decoding)

Single-user latency (concurrency=1)

python3 -m sglang.bench_serving \
  --backend sglang \
  --tokenizer mistralai/Mistral-Small-4-119B-2603-NVFP4 \
  --max-concurrency 1 \
  --num-prompts 20 \
  --random-input-len 256 \
  --random-output-len 128 \
  --dataset-name random

Full results
============ Serving Benchmark Result ============
Backend:                         sglang
Max request concurrency:         1
Successful requests:             20
Benchmark duration (s):          58.48
Total generated tokens:          1566
Output token throughput (tok/s): 26.78
Concurrency:                     1.00
Accept length:                   2.30

--- End-to-End Latency ---
Mean E2E Latency (ms):    2923.17
Median E2E Latency (ms):  2825.40
P99 E2E Latency (ms):     5582.51

--- Time to First Token ---
Mean TTFT (ms):    73.08
Median TTFT (ms):  60.33
P99 TTFT (ms):    136.92

--- Time per Output Token ---
Mean TPOT (ms):    37.19
Median TPOT (ms):  38.06
P99 TPOT (ms):     51.02

--- Inter-Token Latency ---
Mean ITL (ms):    36.94
Median ITL (ms):  26.22
P95 ITL (ms):     78.42
P99 ITL (ms):     86.16
Max ITL (ms):     94.76
==================================================

73ms median TTFT for a 119B model is excellent. The accept length of 2.30 means EAGLE is effectively reducing actual forward passes by ~2.3× at low concurrency — validating the speculative decoding configuration.


Scalability sweep (concurrency 1 → 16)

for CONCURRENCY in 1 2 4 8 16; do
  echo ">>> max-concurrency=$CONCURRENCY"
  python3 -m sglang.bench_serving \
    --backend sglang \
    --tokenizer mistralai/Mistral-Small-4-119B-2603-NVFP4 \
    --max-concurrency $CONCURRENCY \
    --num-prompts $((CONCURRENCY * 10)) \
    --random-input-len 256 \
    --random-output-len 128 \
    --dataset-name random 2>/dev/null \
  | awk '/={10,}/{found=1} found{print} /={10,}/{count++; if(count==2) exit}'
  echo ""
done | tee scalability_results.log

Concurrency Output tok/s Mean TTFT (ms) Mean TPOT (ms) Mean E2E (ms) Accept length
1 23.4 291 30.3 2165 2.02
2 36.5 244 51.6 4151 2.01
4 45.8 328 86.4 4691 1.96
8 59.9 379 129.3 7649 1.89
16 79.4 427 198.4 12345 1.81
Full raw results per concurrency level
>>> max-concurrency=1
============ Serving Benchmark Result ============
Output token throughput (tok/s): 23.44 | Accept length: 2.02
Mean TTFT: 290.88ms | Mean TPOT: 30.27ms | Mean E2E: 2164.87ms
==================================================

>>> max-concurrency=2
============ Serving Benchmark Result ============
Output token throughput (tok/s): 36.52 | Accept length: 2.01
Mean TTFT: 243.69ms | Mean TPOT: 51.63ms | Mean E2E: 4151.22ms
==================================================

>>> max-concurrency=4
============ Serving Benchmark Result ============
Output token throughput (tok/s): 45.81 | Accept length: 1.96
Mean TTFT: 327.63ms | Mean TPOT: 86.39ms | Mean E2E: 4690.99ms
==================================================

>>> max-concurrency=8
============ Serving Benchmark Result ============
Output token throughput (tok/s): 59.90 | Accept length: 1.89
Mean TTFT: 378.83ms | Mean TPOT: 129.34ms | Mean E2E: 7649.27ms
==================================================

>>> max-concurrency=16
============ Serving Benchmark Result ============
Output token throughput (tok/s): 79.36 | Accept length: 1.81
Mean TTFT: 427.12ms | Mean TPOT: 198.39ms | Mean E2E: 12345.11ms
==================================================

Observations:

  • Throughput scales from 23 → 79 tok/s (×3.4) as concurrency goes 1 → 16, but gains diminish: each doubling adds less than the previous.

  • TTFT stays remarkably stable (290–427ms) across the full range — the KV cache pool and CUDA graph captures handle batching well.

  • EAGLE accept length degrades gracefully under load (2.02 → 1.81), still meaningful at concurrency=16.

  • TPOT grows nearly linearly with concurrency, which is expected for a single-GPU setup without pipeline parallelism.


Steady-state throughput (concurrency=8, longer output)

python3 -m sglang.bench_serving \
  --backend sglang \
  --tokenizer mistralai/Mistral-Small-4-119B-2603-NVFP4 \
  --max-concurrency 8 \
  --num-prompts 80 \
  --random-input-len 256 \
  --random-output-len 32 \
  --dataset-name random

Full results
============ Serving Benchmark Result ============
Backend:                         sglang
Max request concurrency:         8
Successful requests:             80
Benchmark duration (s):          39.83
Output token throughput (tok/s): 30.65
Peak output token throughput:    49.00
Accept length:                   2.31

--- Time to First Token ---
Mean TTFT (ms):    640.58
Median TTFT (ms):  550.42
P99 TTFT (ms):    3080.02

--- Time per Output Token ---
Mean TPOT (ms):    225.56
P99 TPOT (ms):     640.33

--- Inter-Token Latency ---
Mean ITL (ms):    226.13
P99 ITL (ms):     952.05
Max ITL (ms):    2445.91
==================================================

Note: shorter --random-output-len 32 with the same concurrency=8 shows higher TTFT variance (P99: 3080ms) compared to the scalability sweep (P99: 852ms at output-len 128). Shorter outputs mean more scheduling churn — the server spends proportionally more time on prefill overhead.


Bug: Triton JIT Crash at Concurrency ≥ 32

At concurrency=32, the server crashes with:

RuntimeError: Triton Error [CUDA]: operation not permitted
  eagle_worker.py → draft() → _draft_preprocess_decode()
  → assign_draft_cache_locs (Triton kernel)
  → triton compiler → load_binary()

This is a race condition between the Triton JIT compiler and the CUDA context under extreme load. Three mitigations, ordered by impact on performance:

Option 1 — Enable SGLANG_ENABLE_SPEC_V2 (recommended first try)

The experimental spec v2 overlap scheduler may handle the concurrency better. Add to your docker run env:

-e SGLANG_ENABLE_SPEC_V2=True

Option 2 — Cap concurrency at 16

The crash is consistently reproducible at 32. Concurrency=16 is stable and delivers 79 tok/s output throughput — a reasonable ceiling for a single-GPU agentic workload.

Option 3 — Disable CUDA Graph

CUDA graph captures and Triton JIT interact poorly under heavy speculative decoding load. Adds overhead but eliminates the race:

--disable-cuda-graph

Option 4 — Switch attention backend to flashinfer

Eliminates all Triton attention kernels. May introduce other issues on aarch64/SM121 (flashinfer attention has known bugs on GB10), but worth testing if spec v2 doesn’t solve it:

--attention-backend flashinfer   # instead of triton


Agentic Usage Tips

Mistral Small 4 unifies instruct, reasoning, and agentic capabilities in one checkpoint. Reasoning effort is configurable per request — no server restart needed.

Recommended pattern for agentic pipelines:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

# Planning / analysis tasks — reasoning ON
response = client.chat.completions.create(
    model="Mistral-Small-4",
    messages=[{"role": "user", "content": "Analyze this codebase and create a refactoring plan."}],
    extra_body={"reasoning_effort": "high"},
)

# Execution tasks — reasoning OFF (faster, lower TPOT)
response = client.chat.completions.create(
    model="Mistral-Small-4",
    messages=[{"role": "user", "content": "Call the search_files tool with pattern='*.py'."}],
    extra_body={"reasoning_effort": "none"},
)

Disabling reasoning on execution/tool-call steps significantly reduces TPOT and E2E latency — the model doesn’t need to think about calling a function it’s already been instructed to call. Reserve reasoning_effort: high for planning, analysis, and ambiguous decisions.

Tool calling is enabled via --tool-call-parser mistral and works with the standard OpenAI function calling format.


Would you like prompt testing cases and tool/skill calling examples? Happy to share those as a follow-up.

Cheers — may the tokens be with you! 🚀

William

Update — Improved launch config + new benchmarks

Following up on my initial setup post. Two meaningful changes to the launch script, one confirmed improvement and one fix.


Changes

1. SGLANG_ENABLE_SPEC_V2=True — overlap scheduler enabled

Added as a Docker env variable:

docker run ... \
  -e SGLANG_ENABLE_SPEC_V2=True \
  ...

Before:

WARNING: Overlap scheduler is disabled when spec v2 is off

After:

INFO: Spec v2 is enabled for eagle/eagle3 speculative decoding and overlap schedule is turned on.

2. Tokenizer regex fix — patched directly in the HF snapshot

The warning fix_mistral_regex=True appearing 4x at boot is a real issue — incorrect pre-tokenizer regex affecting uppercase, accents, and Unicode tokenization. --tokenizer-path doesn’t work for Pixtral-based multimodal models in this SGLang version (crashes with HFValidationError), so the fix is applied directly to the snapshot:

python3 - <<'EOF'
import tokenizers

SNAPSHOT = "/root/.cache/huggingface/hub/models--mistralai--Mistral-Small-4-119B-2603-NVFP4/snapshots/043f75a201a226d8e9cbbc3316af437ea25d3912"
path = f"{SNAPSHOT}/tokenizer.json"

tok = tokenizers.Tokenizer.from_file(path)
split = tokenizers.pre_tokenizers.Split(
    pattern=tokenizers.Regex(
        r"[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+|[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+[\p{Ll}\p{Lm}\p{Lo}\p{M}]*|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n/]*|\s*[\r\n]+|\s+(?!\S)|\s+"
    ),
    behavior="isolated",
)
current = tok.pre_tokenizer
if isinstance(current, tokenizers.pre_tokenizers.Sequence):
    tok.pre_tokenizer[0] = split
else:
    tok.pre_tokenizer = split
tok.save(path)
print("Done.")
EOF

Run once inside the container. The HF cache is on a mounted volume so the patch persists across restarts.


Updated launch script

docker run --gpus all -d --rm \
  --name sglang-jane \
  -p 30000:30000 \
  -v /tmp:/tmp \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --ipc=host \
  -e SGLANG_ENABLE_SPEC_V2=True \
  lmsysorg/sglang:nightly-dev-cu13-20260323-999bad5a \
  sglang serve \
    --model-path mistralai/Mistral-Small-4-119B-2603-NVFP4 \
    --host 0.0.0.0 \
    --port 30000 \
    --tp 1 \
    --attention-backend triton \
    --moe-runner-backend flashinfer_cutlass \
    --reasoning-parser mistral \
    --tool-call-parser mistral \
    --speculative-algorithm EAGLE \
    --speculative-draft-model-path mistralai/Mistral-Small-4-119B-2603-eagle \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --mem-fraction-static 0.75 \
    --context-length 65536 \
    --served-model-name Mistral-Small-4

Note: sglang serve replaces python3 -m sglang.launch_server (both work, sglang serve is the recommended entrypoint per upstream).


Benchmark comparison

Steady-state throughput (concurrency=8, output-len=32)

Metric v1 v2 (Spec V2) Delta
Output tok/s 30.65 59.03 +93%
Mean E2E (ms) 3845 1971 -49%
Mean TTFT (ms) 641 357 -44%
Mean TPOT (ms) 226 112 -50%
Benchmark duration (s) 39.83 20.69 -52%

Scalability sweep — throughput vs TTFT trade-off

Concurrency tok/s v1 tok/s v2 TTFT v1 TTFT v2
1 23.4 27.7 291ms 140ms
2 36.5 39.4 244ms 231ms
4 45.8 49.3 328ms 473ms ⚠️
8 59.9 65.9 379ms 555ms ⚠️
16 79.4 92.6 427ms 702ms ⚠️

Key observation: Spec V2 overlap scheduler doubles steady-state throughput. The trade-off is a TTFT regression at concurrency ≥ 4 (+150–275ms) — the scheduler batches prefill/decode concurrently which delays the first token under load. For single-user or low-concurrency interactive use, TTFT actually improves (-150ms at concurrency=1). For batch/pipeline workloads, the throughput gain is unambiguously worth it.

EAGLE accept length is also more stable under load with Spec V2 (2.19–2.26 vs 1.81–2.02 previously).


Note on KV cache: Spec V2 uses additional buffers for the overlap scheduler, reducing the KV cache pool from ~910K to ~637K tokens. Still comfortable for standard agentic workloads at 65536 context length.

why the short context? Isn’t the MLA architecture of Mistral Small 4 meant to help a lot with that and allow a bigger context in smaller memory? I have that model running under vllm with 256k context (well, I set the parameters but haven’t pushed it to max context so far)?

Also is reasoning fully supported? I have a hard time to get agent harnesses to actually activate reasoning and am wondering how the situation is with slang. for vllm it was related to problems with mistral_common, so no idea if sglang is facing the same issues. I am still too noob in all this stuff ;)

Hello @glbauer

Insights on KV Cache and Inference Performance

Understanding KV Cache Behavior

From what I gather about KV cache (the storage of key-value pairs for immediate predictions):

  • More context = larger KV cache (higher memory usage).
  • Trade-off: While beneficial for repetitive tasks (e.g., batch processing, document classification), it introduces slower Time-to-First-Token (TTFS) for dynamic or interactive requests (e.g., brainstorming, quick Q&A).

Use Case Impact:

  • Ideal for continuous, high-volume workloads (e.g., automation, classification).
  • Less ideal for short, exploratory tasks (e.g., tweaking prompts, rapid ideation).

Reasoning Mode Performance

Fully supported in “high” mode:

  • Handles complex queries well after adjustments.
  • Critical Limitation: “low” and “medium” modes trigger 401 Bad Request errors.
  • SGLang makes switching reasoning modes straightforward.

Performance vs. vLLM:

  • No noticeable speed advantage over vLLM.
  • Chosen for convenience—easy setup and usability over raw performance gains.

Vision Model (PDF/Image Processing)

Delivers as promised:

  • Reliable processing of PDFs and images.
  • Limitations: A lightweight MoE agentic model optimized for cost-efficiency (sacrifices raw speed for affordability).

Desired Improvement:

  • Optimize inference speed to rival GPT-OSS-120B (~50-60 tok/s).

Final Takeaways

A practical Agent, but constrained by its lightweight MoE design.

Cheers, and may your tokens be ever in your favor! 🙏

— William

Just a quick update to community: vllm 0.21 released days ago included mistral fixes pushed into main earlier and now 119b work on boilerplate vllm. 1 seq at 30 tg tps very stable without eagle, which is still not supported. Once it’s on I expect 40-45 tg tps. Tools calls and reasoning excellent. Work like a charm in open code, temp 0.6, 0.95, 20, repetition penalty 1.05

Just heads up.

Could I ask you to share an example of your launch command to achieve 30 tokens per second?

Here you go. Vanilla release vllm 0.21 container, official Mistral NVFP4 checkpoint. No gimmicks. Just works. No MTP - Eagle is still in works by vLLM.

VLLM BENCH

Serving Benchmark Result

Successful requests: 20
Failed requests: 0
Maximum request concurrency: 1
Benchmark duration (s): 89.01
Total input tokens: 10520
Total generated tokens: 2560
Request throughput (req/s): 0.22
Output token throughput (tok/s): 28.76
Peak output token throughput (tok/s): 30.00
Peak concurrent requests: 2.00
Total token throughput (tok/s): 146.96
---------------Time to First Token----------------
Mean TTFT (ms): 112.88
Median TTFT (ms): 112.60
P99 TTFT (ms): 120.71
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 34.15 ~ 29.4 tok/s
Median TPOT (ms): 34.29
P99 TPOT (ms): 34.53
---------------Inter-token Latency----------------
Mean ITL (ms): 33.88
Median ITL (ms): 33.90
P99 ITL (ms): 41.03

Serving Benchmark Result

Successful requests: 20
Failed requests: 0
Maximum request concurrency: 4
Benchmark duration (s): 34.39
Total input tokens: 10520
Total generated tokens: 2560
Request throughput (req/s): 0.58
Output token throughput (tok/s): 74.44
Peak output token throughput (tok/s): 91.00
Peak concurrent requests: 8.00
Total token throughput (tok/s): 380.35
---------------Time to First Token----------------
Mean TTFT (ms): 288.64
Median TTFT (ms): 268.56
P99 TTFT (ms): 418.84
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 51.86
Median TPOT (ms): 51.64
P99 TPOT (ms): 53.12
---------------Inter-token Latency----------------
Mean ITL (ms): 51.46
Median ITL (ms): 52.46
P99 ITL (ms): 61.46

LLAMA-BENCHY

model test t/s (total) t/s (req) peak t/s peak t/s (req) ttfr (ms) est_ppt (ms) e2e_ttft (ms)
mistral-small4 pp512 (c1) 1365.43 ± 17.73 1365.43 ± 17.73 336.25 ± 5.34 332.04 ± 5.34 336.25 ± 5.34
mistral-small4 tg128 (c1) 29.51 ± 0.04 29.51 ± 0.04 30.00 ± 0.00 30.00 ± 0.00
mistral-small4 pp512 (c4) 2444.88 ± 44.02 706.80 ± 140.56 699.18 ± 116.06 694.97 ± 116.06 699.18 ± 116.06
mistral-small4 tg128 (c4) 72.03 ± 2.42 18.98 ± 0.35 82.67 ± 1.89 21.08 ± 0.76

My own benchmarks based on real working tasks, tricky questions and agentic workload. Analyzed by my Agent, verified by me. Grades are based on a baseline of GPT 5.4 Thinking - it is not score 100, as while answered all questions and researched them well it butchered the bencher code a bit, so my agent punished his score to around 95.


Executive Summary

After extensive testing across 7 model configurations on DGX Spark hardware, here’s the definitive ranking for our sovereign AI stack:

Rank Model Grade Speed (single) Speed (concurrent) Latency Weights 1M Tokens Context Token Size Verdict
1 Mistral Small 4 91.38 30.7 tok/s N/A 7.4s ~75GB (nvFP4) ~10GB 256K Small Quality leader, fast
2 Qwen3.6-40B Deckard FP8 89.06 11.8 tok/s N/A 177.1s ~60GB (FP8) ~51GB 256K Massive 40GB/M Big brain, uncensored
3 Qwen 3.6 27B 87.83 21.8 tok/s N/A 14.1s ~22GB (nvFP4) ~36GB 1M (YaRN) Big 36GB/M Balanced choice
4 Nemotron Cascade 2 85.78 59.5 tok/s N/A 5.1s ~22GB (nvFP4) ~14GB 1M native Small Speed champion
5 Nemotron Omni 3 85.06 36.6 tok/s N/A 55.7s ~22GB (nvFP4) ~14GB 256K Small Solid middle ground
6 Nemotron 3 Super NVFP4 88.56 20.4 tok/s 41.3 tok/s ~80GB (nvFP4) ~12GB 1M Small Heavy lifter
7 Qwen 3.5 122B Intel AutoRound INT4 81.56 27.1 tok/s N/A 127.1s ~67GB (INT4) N/A 1M (YaRN) Small

mistral-4-small.md.txt (6.0 KB)

Thanks a lot! This is super interesting. And the first post that highlights Mistral Small 4 as quality leader. Most benchmarks imply it is not that good.

In my use cases (non coding, European languages), I began to prefer Qwen3.6 35b. Have you tried it? It is twice as fast as Mistral Small 4, but thinks/reasons very very long, which makes time to first token not that good.

Will try your setup tomorrow.

I am in Europe too, Czech Republic. I used to use mostly Qwens, 3.5 9B, used LFM 2 models, Mistral Nemo 14B, Mistral 7B. Depending on task. Then Qwen 3.6 got released and I switched to it. 35b is very very lightweight but struggles a lot on large context. Not suitable - I work with massive data corpusi, large codebases. 200k and it loops like a madman. In contrast, same 30B A3B Cascade 2 works fine with large data and has native 1M tokens. Massive density of 27b works with YaRM well past native 256k - I constantly push to 500k (not at once but during the session, sequentially).
But honestly Mistral 4 Small is just amazing. It is so good balance of speed and smartness, handles all well: chat, research, docs, code. It’s a mixture of 3 models as they say: Baguette, Champagne and Fromage :) LOL, No, Codestral, Ministral and something else. But it just did not work in GB10 constrainst on backend and RAM (FP8 would not fit and would be woefully slow), until few days ago when 0.21 was released with specifically Mistral fixes. Once they fix Eagle it would be awesome.

Yes, Qwen3.6 sometimes runs in circles, unfortunately. Downloading 0.21 and Mistral Small 4 at the moment. Might take some hours. Can’t wait to try :-). The Mixtrals, esp. 8x22b, were the reason I started local LLMs.