Qwen3.8-Flash-Next on 3 DGX Sparks — 3 Recipes, Full Debugging Journey, Final Reproducible Setup
I tested Qwen3.8-Flash-Next on three NVIDIA DGX Sparks using three different deployment approaches.
The result surprised me:
The most aggressive recipe ended up being the easiest to operate, the fastest, and the most stable in our production-style testing.
Before the technical details, one important note:
I do not know how to code.
I am not a CUDA engineer, vLLM developer, or AI infrastructure expert.
I did this with ChatGPT acting as my technical copilot. I decided what I wanted to test and whether the results were acceptable. ChatGPT helped with reading documentation, analyzing logs, writing scripts, debugging failures, changing configurations, and running benchmarks.
That is also why I am posting the full path instead of only the final tok/s number.
My goal is simple:
If another DGX Spark owner — even someone who is not a programmer — wants to reproduce this, I hope this post saves them a lot of time.
And one warning:
Please don’t ask me detailed coding questions. I probably won’t know the answer. 😄
If you hit a technical problem, I honestly recommend giving your logs, configuration and this post to your own ChatGPT. That is basically how I did this.
Hardware
All three tests were on single NVIDIA DGX Spark systems:
-
NVIDIA GB10
-
128 GB unified memory
-
NVIDIA driver 580.x
-
CUDA 13.x
We called the machines:
-
Spark 3
-
Spark 4
-
Spark 5
The purpose was not just to find the highest benchmark score.
We wanted to know:
Can this model actually work like a reliable AI employee?
So we tested:
-
startup
-
correctness
-
reasoning
-
long context
-
prefill
-
decode speed
-
concurrency
-
images
-
video
-
prefix caching
-
memory behavior
-
swap
-
multi-hour stability
The three approaches
Spark 3 — Conservative vLLM + NVIDIA NVFP4
This was our stability-first route.
Core stack:
Checkpoint:
nvidia/Qwen3.8-Flash-Next-NVFP4
Runtime:
vLLM
Quantization:
NVIDIA NVFP4
Execution philosophy:
Conservative / stability first
PLE:
SSD/offload path
MTP:
Ultimately disabled for stability
Execution:
Eager / non-aggressive configuration
This route was relatively stable and usable.
But the important lesson was that some of that stability came from turning off optimizations that were not behaving reliably in our environment.
MTP was one of them.
Instead of forcing every possible speed optimization to work, we backed off until the model behaved predictably.
The result was usable, but considerably slower than what we eventually achieved on Spark 5.
It also was not perfect from a quality perspective; we still observed isolated model-quality errors during testing.
Lesson from Spark 3
If your first priority is:
“I just want the model to come up and behave predictably.”
a conservative vLLM/NVFP4 configuration is a reasonable place to start.
But don’t assume every optimization needs to be enabled just because it exists.
Spark 4 — vLLM + NVFP4 + PLE + MTP
Spark 4 was supposed to be our compromise:
Keep the familiar NVIDIA NVFP4/vLLM stack, but add MTP and other optimizations to get more performance.
Instead, this became by far the most difficult route.
This experiment consumed many hours of debugging and multiple startup attempts.
Problem 1 — MTP FP8 weights did not load correctly
The first major issue was the MTP weight format.
We created an FP8 dequantization patch and tested it against real checkpoint tensors.
The numerical validation looked reasonable:
maximum relative error ≈ 3.8e-3
That was consistent with expected BF16 rounding behavior.
But then another problem appeared.
Problem 2 — our patch matched too many tensors
The first regex was too broad.
It intercepted not only the intended MTP tensors, but also main-model expert weights.
Those tensors did not have the scale information the patch expected.
So the loader later failed.
Fix
We narrowed the matching logic so the patch only intercepted the MTP tensors.
Then we repeated validation.
Problem 3 — very long startup cycle
There were multiple failed boot attempts.
One successful MTP=4 startup alone took roughly:
1575 seconds
≈ 26 minutes
That made iteration painful.
And after finally getting the engine up, we discovered the larger problem.
Problem 4 — the model was running, but correctness was terrible
Our first meaningful correctness run produced:
4 / 32 PASS
28 / 32 FAIL
This was much worse than expected.
A common output pattern looked like:
“The user is asking me to…”
instead of simply answering the question.
The model was entering long reasoning behavior because of its chat template and thinking configuration.
Problem 5 — disabling thinking was not enough
Turning thinking off shortened the answers.
But then we found something worse:
the same prompt at temperature=0 could produce different answers.
That meant we had a determinism problem.
Problem 6 — FlashInfer autotune
After isolating components, FlashInfer autotuning became a major suspect.
We disabled it.
Then tested a simple deterministic arithmetic case:
17 * 23
Result:
391
391
391
Three identical runs.
That was the first strong sign that we had located an important instability source.
Our working direction became approximately:
MTP = 4
FlashInfer autotune = OFF
Qwen3 reasoning parser = ON
But even then, the complete correctness suite was still only:
13 / 32 PASS
19 / 32 FAIL
Some of those failures were benchmark-format problems rather than true model failures.
For example:
Expected:
4444
Model:
44 × 101 = 4444
Semantically correct, but the validator marked it wrong.
Other failures were real:
-
reasoning exceeded token budgets
-
parser/thinking produced empty visible answers
-
some arithmetic was actually wrong
-
some coding reasoning did not finish properly
We eventually discovered that different workloads benefited from different policies.
For example:
Arithmetic:
reasoning_effort = medium
Coding:
reasoning_effort = low
+ direct system prompt
Lesson from Spark 4
This route taught us an important lesson:
Getting the server to start does not mean the model is ready to work.
Correctness, determinism, parser behavior and inference policy matter just as much as tok/s.
Spark 5 — TensorFold recipe
This was the route I originally expected to be the riskiest.
Instead, it became our best result.
The public recipe we reproduced was:
MiaAI-Lab/
Qwen3.8-Flash-Next-Single-DGX-Spark-TensorFold
Recipe revision used:
4c0dea8ebe93a1d445d6a716b5a38995ff6e9362
Runtime:
TensorFold v0.6.1
Checkpoint:
TensorFold/Qwen3.8-Flash-Next-MLX-4bit-MTP
This is important:
This is NOT the NVIDIA NVFP4 checkpoint used in the previous routes.
It is:
MLX 4-bit
group size 32
native MTP draft head retained
22 model shards
approximately 113.2 GiB
Final Spark 5 recipe
These are the important settings we successfully qualified:
PARALLEL=5
CONTEXT=262144
KV_DTYPE=int8
MTP_DRAFTS=6
MTP_CONFIDENCE=0.60
TENSORFOLD_MTP_COPY=1
PLE_ON_SSD=1
TENSORFOLD_MEMORY_RESERVE_GIB=2
VISION=1
VISION_MAX_IMAGES=50
Equivalent core launch behavior:
--parallel 5 \
--context 262144 \
--kv-dtype int8 \
--mtp-drafts 6 \
--mtp-confidence 0.60 \
--ple-on-ssd \
--vision \
--vision-max-images 50 \
--thinking
Important details:
Execution:
eager
CUDA graphs:
OFF
KV:
INT8
PLE:
TensorFold native SSD path
Parallel streams:
5
Maximum context:
262,144 per stream
We deliberately did not mix in the vLLM configuration from Spark 3 or Spark 4.
Spark 5 problem #1 — not enough free memory
Before launch, Spark 5 had:
Available memory ≈ 45 GiB
because other models/services were running.
This TensorFold recipe needed roughly:
103 GiB+
available for admission.
We stopped the competing Qwen27B and music workloads.
Afterward:
Available memory ≈ 117 GiB
Swap ≈ 0
Then the recipe had enough room to run.
Lesson
If this recipe refuses to start, check MemAvailable, not just total 128 GB.
Problem #2 — existing NVFP4 weights could not be reused
Spark 5 already had Qwen3.8-Flash-Next NVFP4 weights.
Initially I assumed we could reuse them.
We could not.
The TensorFold recipe requires its own MLX-4bit-MTP checkpoint.
So we downloaded approximately:
113.2 GiB
of new model files.
Problem #3 — Hugging Face HTTP 401
Checkpoint preparation initially failed with:
HTTP 401
We found two environmental issues:
-
an old Hugging Face credential existed on the host
-
the model namespace had moved from the older
Vontra/...location to the canonical TensorFold repository
Our working solution was:
MODEL_ID=
TensorFold/Qwen3.8-Flash-Next-MLX-4bit-MTP
with:
fresh isolated HF_HOME
anonymous download
We left the existing Hugging Face cache alone.
After that, the download completed normally.
Startup
Once the checkpoint was present, the TensorFold service became available in approximately:
~100 seconds
after launch.
Compared with our Spark 4 experience, this was remarkably straightforward.
Problem #4 — our benchmark itself was wrong
Our first correctness results showed several arithmetic failures.
The model was not actually causing all of them.
We found a bug in our own Python benchmark.
The regex had no capture group, but the code called:
group(1)
which caused:
IndexError: no such group
We fixed it to use the correct match.
Then reran the tests.
This was an important lesson:
A benchmark bug can easily look like a model-quality failure.
Final correctness result
After fixing benchmark issues:
39 / 40 PASS
One problem remained:
logic_grid
We did not hide it.
We reran the same English question multiple times.
It remained wrong.
We rewrote it in Chinese.
Still wrong.
So we classify this as a genuine model reasoning miss.
Final:
39 / 40
97.5%
Problem #5 — video looked completely broken
Our first video test produced:
8 / 8 empty visible answers
At first glance, it looked like video support was broken.
But server metrics told a different story:
video prefill: successful
prefill time: ~0.46 s
decode: ~67 tok/s
MTP: active
completion tokens: exactly 400
finish_reason: length
visible content: empty
The video pipeline was working.
The model had spent the entire 400-token response budget thinking internally before producing the visible answer.
For these multimodal tasks, we changed the request policy to:
enable_thinking = false
and/or provided sufficient response budget.
Then the video tests passed.
Important lesson
If Qwen3.8-Flash-Next returns an empty answer on video:
do not immediately assume video is broken.
Check whether thinking consumed the whole completion budget.
Performance
Short-context decode
We measured approximately:
71–72 tok/s
for clean short-context generation.
Depending on token-accounting methodology, the number can appear closer to the low-50 tok/s range.
Long-context decode
Our measured output rates were approximately:
100K context:
68.9–70.3 tok/s
150K:
67.5–69.1 tok/s
200K:
60.3–70.6 tok/s
250K:
64.7–65.6 tok/s
The important result for us was:
Even near 250K context, decode remained above approximately 60 tok/s.
Prefill
Cold prefill was approximately:
2,000–2,477 tok/s
across our long-context tests.
195K needle test
We tested retrieval around:
~195K tokens
The model returned the correct information.
Prefix reuse
At approximately 200K context:
Cold TTFT:
~74 seconds
Reused prefix TTFT:
~0.35 seconds
Approximately:
>200× improvement
for the reused prefix case.
This was one of the most impressive results in our testing.
Concurrency
Native C=4 aggregate throughput:
~85.1 tok/s
We also tested two concurrent ~200K-context workloads.
Both completed successfully.
There was substantial prefill contention, which is expected:
TTFT:
~90 s
~159 s
Shared decode aggregate was approximately:
~55 tok/s
Production-style soak test
Finally, we stopped treating it like a benchmark and treated it like a worker.
We ran the configuration continuously for about three hours.
Results:
Duration:
10,832 seconds
≈ 3.01 hours
Cycles:
320
Requests:
8,287
Errors:
0
Image/video successes:
322 / 322
Prefix cache:
~90%
Final MTP acceptance:
~0.604
Swap:
~1.16–1.20 GiB
Preemptions:
0
Container restarts:
0
Our verdict for this exact configuration:
CANARY_READY
Not “perfect.”
Not “guaranteed production-ready everywhere.”
But good enough that we are continuing serious testing with this recipe.
What surprised us most
Before starting, my expectation was:
Conservative recipe
= safest
Hybrid recipe
= best compromise
Aggressive TensorFold recipe
= fastest but probably fragile
What we observed was almost the opposite.
Spark 3 became stable partly by disabling optimizations.
Spark 4 became a long engineering/debugging exercise.
Spark 5 still had problems, but most of them were understandable:
-
insufficient free memory
-
wrong checkpoint assumption
-
Hugging Face authentication/environment issue
-
benchmark code bug
-
process-control script bug
-
inference-policy problem with video/thinking
Once those were corrected, the core TensorFold recipe behaved extremely well.
This changed one assumption for me:
“Aggressive” does not automatically mean “fragile.”
Sometimes a tightly integrated optimization stack can be easier to operate than a conservative-looking stack made from components that do not fit together cleanly.
Why I am sharing the failures
I think the failures are almost more useful than the final tok/s numbers.
If someone tells you:
“I got 72 tok/s.”
you still don’t know what to do when your installation fails.
I would rather tell you:
Here is the recipe.
Here is where ours failed.
Here is what the error looked like.
Here is what we changed.
Here is what turned out NOT to be a model problem.
Here is the configuration that finally survived real testing.
Hopefully that saves the next DGX Spark owner some time.
Final Spark 5 configuration summary
For anyone who mainly wants the final answer:
Hardware:
NVIDIA DGX Spark / GB10 / 128 GB
Model:
TensorFold/Qwen3.8-Flash-Next-MLX-4bit-MTP
Runtime:
TensorFold v0.6.1
Recipe:
MiaAI-Lab/Qwen3.8-Flash-Next-Single-DGX-Spark-TensorFold
Parallel:
5
Context:
262144
KV:
int8
MTP drafts:
6
MTP confidence:
0.60
MTP copy:
enabled
PLE on SSD:
enabled
Vision:
enabled
Vision max images:
50
Execution:
eager
CUDA graph:
disabled
Our measured result:
Correctness:
39 / 40
Short decode:
~71–72 tok/s
250K decode:
~65 tok/s class
Cold prefill:
~2,000–2,477 tok/s
195K needle:
PASS
3-hour soak:
8,287 requests
0 errors
Multimodal:
PASS after correcting thinking policy
Verdict:
CANARY_READY
If anyone else reproduces this on a DGX Spark, I would be very interested to see whether your numbers match ours.
And again: if you hit a technical problem, don’t ask me to debug the CUDA code. 😄
Ask your ChatGPT, give it the logs, and make it work for you. That’s exactly what I did.