Qwen3.8-27B on one DGX Spark, single stream, thinking ON: 8 serving configs, per-question correctness on hard code/reasoning tasks

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
,
1

Qwen3.8-27B on one DGX Spark, single stream, thinking ON: 8 serving configs, per-question correctness on hard code/reasoning tasks

Category: DGX Spark / GB10 Projects

TL;DR

  • With thinking on (template default, ~10K output tokens per answer) and per-question grading, TensorFold 0.6.0 + nvidia/Qwen3.8-27B-NVFP4 + DFlash2 gave complete answers ~2.3x faster than vLLM NVFP4 + MTP3 (median 184 s vs 469 s; 24 prompts in 1.73 h vs 4.03 h) with the same number of correct answers (18/24).
  • Inside vLLM, MTP5 was the best depth: 1.24x vs MTP3, 19/24. MTP4 was not better than MTP5 on either axis.
  • TensorFold + MLX 4-bit was the fastest (68 tok/s) but passed only 14/24; switching only the weights to NVIDIA NVFP4 (same engine, drafter, FP8 KV/prefill) brought it back to 18/24.
  • Speculative decoding itself showed no quality loss in the two clean pairs (same engine and weights): NVFP4 without speculation vs MTP3 were identical per question; FP8 MTP3 vs DFlash2 was -3/+4.
  • Caveat for anyone benchmarking this model: across the 8 configs, 9 prompts always pass, 4 never finish within 24,576 tokens, and 11 flip between configs. One-launch differences of 1-4 questions are mostly these fragile prompts, not real quality differences.
  • Everything here is one launch per configuration on 24 prompts. Treat it as preliminary.
  • The repository has the results and a step-by-step setup (pinned images, checkpoint revisions, exact launch commands, measurement clients), so you can build the same engines on your own Spark and test them with your own prompts: qwen3.8_study/qwen38-spark-speed-comparison-display-on at main · NautilusWu/qwen3.8_study · GitHub

speed_vs_correctness
speed_vs_correctness1600×960 78.6 KB

Why this test

Most public Qwen3.8-27B numbers on the Spark are short outputs (a few hundred tokens), thinking off, and speed only. I wanted the number a single user actually feels: time to a complete, correct answer with thinking on, and whether any of the speed-ups cost correctness on tasks harder than GSM8K.

Setup (identical for every row unless noted)

  • One DGX Spark (GB10, 128 GB unified memory, ~273 GB/s), DGX OS 7.2.3, driver 580.173.02.
  • Display on for every run except the second TensorFold MLX 4-bit launch (20 of its 24 prompts), where it was reportedly off but not verified. An active display costs ~12-22% decode speed on this machine, so absolute speeds are below a headless box.
  • Single user, one request at a time (n=1); the next request starts only after the server is idle, and server counters confirm that no other client generated during a measurement.
  • Prompts rendered with the official Qwen tokenizer + chat template (Qwen/Qwen3.8-27B@1d4bf0f, byte-identical to the files in NVIDIA’s repo); prompt token IDs verified identical across all engines (sha256).
  • Thinking on (template default reasoning effort), temperature 0, max_tokens 24,576, context 32,768, system prompt “You are a helpful assistant.”
  • Checkpoints: nvidia/Qwen3.8-27B-NVFP4@482ca0f, Qwen/Qwen3.8-27B-FP8@017b9c7, TensorFold/Qwen3.8-27B-MLX-4bit@22d8d53; drafter z-lab/Qwen3.8-27B-DFlash2@50307d4.
  • vLLM 0.27.1 = NGC nvcr.io/nvidia/vllm:26.08-py3; vLLM 0.29.0 = NGC 26.09-py3 (needed for DFlash2; 0.27.1 has no DFlash2 support). vLLM flags: BF16 KV, prefix caching off, chunked prefill, --max-num-seqs 4, --gpu-memory-utilization 0.75, --language-model-only.
  • TensorFold v0.6.0 (c464617) with MiaAI-Lab’s recipe patches (c11ff13): --parallel 1 --context 32768 --thinking --prefill-fp8, TENSORFOLD_KV_DTYPE=fp8. Note: FP8 KV and FP8 prefill on TensorFold rows, BF16 KV on vLLM rows.

Tasks and grading

24 tasks (12 coding, 12 reasoning) that I wrote for this test, with answers computed by code:

  • coding: algorithms with performance-bound hidden tests, specification edge cases (cron, SemVer, glob…), bug fixing, predicting exact program output (Python/C++), C++ with hidden tests, a 2K-token module where a method must follow cross-function conventions;
  • reasoning: multi-step calculations, combinatorics/probability (brute-force verified), number theory, logic puzzles whose uniqueness was proven by exhaustive search, constructions/counterexamples checked by a verifier, derivations from long documents (tax rules, shipping tariff, timetable), strict-format instructions.

Only the text after </think> is graded; no final answer = fail. Every grader was checked to accept a reference answer and reject known-wrong answers. I am not publishing the prompts; the repository lets you run the same engines on your own tests.

Metrics

  • Output speed: committed tokens after the first token-carrying stream event / time between the first and last such event (median over prompts). Rejected drafts are never counted.
  • Total time: request sent → stream finished. This is what a user waits for.
  • Ratios are per-prompt paired geometric means vs vLLM NVFP4 MTP3.

Results

Engine Weights Speculation Output tok/s Median total s 24 prompts (h) Total-time vs NVFP4 MTP3 Passed (code/reasoning) vs NVFP4 MTP3 per question
vLLM 0.27.1 NVIDIA NVFP4 none 8.8 1233 9.44 0.42x 18 (7/11) identical
vLLM 0.27.1 NVIDIA NVFP4 MTP3 21.9 469 4.03 ref 18 (7/11) ref
vLLM 0.27.1 NVIDIA NVFP4 MTP4 23.9 392 3.54 1.18x 16 (5/11) -3 / +1
vLLM 0.27.1 NVIDIA NVFP4 MTP5 24.8 379 3.39 1.24x 19 (7/12) -1 / +2
vLLM 0.29.0 Qwen FP8 MTP3 14.3 615 5.92 0.70x 14 (4/10) -5 / +1
vLLM 0.29.0 Qwen FP8 DFlash2 k=7 24.7 389 4.03 1.07x 15 (4/11) -3 / 0
TensorFold 0.6.0 MLX 4-bit DFlash2 68.3 198 1.64 2.53x 14 (4/10) -5 / +1
TensorFold 0.6.0 NVIDIA NVFP4 DFlash2 52.6 184 1.73 2.34x 18 (7/11) -1 / +1

pass_matrix
pass_matrix1520×1480 171 KB

What explains the speed

Single-stream decode is bandwidth-bound, so tok/s ≈ tokens accepted per verification step × steps per second, and steps/s ≈ effective bandwidth / bytes read per step. The estimates below use weight sizes read from the safetensors headers plus KV and the linear-attention state (an estimate, not a profiler measurement):

Config Tokens/step GB read/step (est.) Bandwidth used (est.)
vLLM NVFP4 none / MTP3 / MTP5 1.00 / 3.08 / 3.59 18.5 / 23.2 / 26.4 59% / 57% / 63%
vLLM FP8 MTP3 / DFlash2 3.00 / 4.27 36.9 / 34.3 61% / 63%
TensorFold MLX 4-bit / NVFP4 DFlash2 5.47 / 5.63 20.6 / 22.8 87% / 70%
  • MTP depth has diminishing returns: each extra MTP draft re-runs the MTP layer and the full LM head (~1.6 GB on NVFP4), so MTP5 reads ~14% more per step than MTP3 while accepting ~17% more tokens; net +13%.
  • FP8 is ~35% slower than NVFP4 at the same speculation: 26.9 vs 17.6 GB of weights per step, plus a BF16 LM head (2.5 GB) that MTP re-reads per draft.
  • DFlash2 beats MTP (same engine and weights, FP8: 1.73x output speed) because it drafts a whole block in one pass and reads its 3.85 GB drafter once per step.
  • TensorFold vs vLLM with the same DFlash2 drafter: TensorFold verified ~15.7 drafted rows per step (DFlash2 plus its context copies) vs 7 in vLLM, accepting 5.6 vs 4.3 tokens/step, at higher bandwidth efficiency (70-87% vs ~60%), and its FP8 KV halves KV reads.

What explains the correctness differences

  • Fragile prompts. At temperature 0, any change of kernels, quantization or verification shape changes floating-point rounding; at an early near-tie a different token is chosen and the whole reasoning path changes (NVFP4 without speculation vs MTP3: outputs diverged within the first 2-72 tokens). The flipping prompts are (a) near the output budget (they need ~20K tokens, so finishing within 24,576 depends on the path), (b) long exact hand computations (one wrong letter or digit), and (c) subtle specs (spacing, exact output format).
  • Output length varies about ±25% between configs even at temperature 0, which moves total time independently of tok/s (e.g. MTP4 wrote 26% fewer tokens than MTP3 on the 9 always-pass prompts).
  • MLX 4-bit looks like a real effect, not only noise: longest thinking (median ~16.1K output tokens vs ~9.1-10.9K for every other row), tied for the most unfinished answers (8, with FP8 DFlash2), and the pass count recovered (14 → 18) when only the weights changed to NVIDIA NVFP4. The public quality checks I found for this stack test whether speculation changes the output (token identity) or use short tasks such as GSM8K/IFEval; I did not find a long-thinking accuracy comparison of the two checkpoints. One launch, so please treat it as a lead, not a verdict.
  • FP8 rows (14, 15) are unexplained: they also changed the vLLM version (0.29.0) and the differences sit on fragile prompts. FP8 being genuinely worse than NVFP4 would be surprising.

Thermal protection and stability

  • vLLM rows: no automatic thermal stop; GPU temperature logged every 0.5 s; peaks 69-77 °C; average power ~35 W.
  • TensorFold MLX 4-bit: the first launch had an automatic stop at 80 °C held for 3 s, which triggered after 4 prompts. The other 20 prompts ran in a second launch with the automatic stop disabled (temperature still logged); peak 79 °C.
  • TensorFold NVIDIA NVFP4: automatic thermal stop disabled, temperature logged every 2 s; peak 80 °C; average power ~50 W; ~23.5 GiB model process memory.
  • A host-memory and driver-error watchdog ran during the TensorFold launches and never triggered. No NVIDIA out-of-memory or Xid errors in any of the eight rows.

Limitations

  • One launch per configuration, 24 prompts; runs on different days; no reversed-order replication yet.
  • Display state was not identical across rows (see Setup).
  • The TensorFold MLX 4-bit row merges two launches (see Thermal protection). TensorFold rows use FP8 KV/prefill, vLLM rows BF16 KV.
  • Bandwidth figures are estimates from weight sizes and server step counters.

Related public work

What this post adds: thinking on with ~10K-token answers, per-question grading with hidden tests, and the per-question pass matrix across eight configurations.

Data and reproduction

Thanks to the authors whose public work made this possible: TensorFold (Ash Hart), MiaAI-Lab’s recipe, z-lab (DFlash2), and 0xBakeer (vLLM + DFlash2 on FP8).

2

I’m just now seeing this! I ran these benchmarks and when I started early last week I didn’t see that anyone had reported these yet on this model.

It appears that our results are similar, but our setups are slightly different in either case. We scored the same on IFEval and our HumanEval and GSM8K results are different but not entirely comparable since I used different settings.

We ran into the same long-reasoning walls, though. I haven’t shared my GPQA results, but I found around a third of them never finished in time to be scored fairly, I’m tracking a few changes now that helps get answers on those more difficult problems.

I’ll just link my write-up. I didn’t see any garbled outputs running SGLang (SGLang nightly-dev-20260814 commit c4271c3) up to 6k short outputs, but I don’t have anything yet on the long-generation checks where you saw your bugs.