Just like I did with my Faster Qwen3.5 122B, I’m basically doing the same with this great awesome new model from Qwen. Single-line command: git clone https://github.com/azampatti/Qwen3.8-Flash-Next-Int4-FAST.git && cd Qwen3.8-Flash-Next-Int4-FAST && bash setup.sh Repos: https://github.com/azampatti/Qwen3.8-Flash-Next-Int4-FAST/blob/main/README.md azampatti/Qwen3.8-Flash-Next-125B-A5B-INT4-AutoRound · Hugging Face Speed: Of course ‘it depends’ but in my tests, between 60-70 tok/s in coding. [image] Context: From vLLM’s launcher: GPU KV cache size: 644,732 tokens, Maximum concurrency for 262,144 tokens per request: 2.46x But HOW? I halved how many experts the model uses per token, from 10 to 5, then taught its shared experts to make up the difference by imitating the uncut model. Nothing was removed from the model. All 512 experts are still there, I just changed how many of them each token is routed to, which cuts active parameters from 6B to 4.8B and makes it faster, then healed the quality loss by training one small always-on component, 37.75M parameters with everything else frozen, to match the original’s output distribution and quality as close as possible. MTP Draft head stays in BF16 and Topk10 to predict at the best possible quality with a tiny bit of speed penalty Try it, enjoy and please provide me with feedback! I’m still improving this model every day, so I will be updating this thread when an improvement is posted. What’s Next? I’m testing new containers as more talented people are working on them. That’s why I wanted to push this model through HuggignFace as well Just make sure you also pay attention to the launch parameters I use to ensure the best performance. Quality should be within 2% of the original 4bits models (both NVFP4 and INT4 I tried scored closely from each other), but I’m testing avenues to raise the quality even further like I was able to do with 122B. It’s MUCH harder with this model though , but I’m on it! Bookmark this thread, hopefully I can do at least weekly updates with improvements for the next couple of weeks. The idea would be to re-run the one-liner and that would check and update everything. -Aldo EDIT: IMPORTANT - The default chat template for this model thinks in EXTRA HIGH. I attached two slightly modified chat templates that are xhigh and medium but with some tweaks to attack some scenarios for stronger coding in OpenCode. I did not included them by default to avoid “cheating” on comparisons, but consider using the “medium” chat template, it’s faster and stronger, but benchmark it on default. EDIT2: Up to 70tok/s Qwen3.8-Flash-Next-Int4-AutoRound - #13 by ds2atc @ds2atc benchmarked with tool-eval-bench and I’ve just personally validated in my DGX that this results are accurate.

Neat! BTW, Medium should be the default, so triple check your settings. :) (Perhaps you’re running low as default somewhere).

This recipe is stable enough for 13 tasks across a 10-hour coding and review session. (The DGX clock speed is limited to 2,000 MHz to prevent excessive heat.) [image]

@m0l0 I ran a full GPQA Diamond eval with 4 epochs x 198 samples on Qwen3.8 Flash Next NVFP4 (Single Spark) and GLM 5.3 Flash NVFP4 (Dual Sparks) and the accuracy scores were basically identical, slightly in favour of Qwen3.8 Flash Next (91.2% vs 90.5%), and very similar speeds: Qwen3.8-Flash-Next-NVFP4 ╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │gpqa_diamond (198 x 4 samples): openai/qwen3.8-flash-next │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ max_retries: 2, max_connections: 8, max_tokens: 131072, epochs: 4, fail_on_error: False, token_limit: 262144, time_limit: 10800, dataset: gpqa_diamond_74187e36ccadd6a06b1d98d13e064fed total time: 2 days, 3:32:22 openai/qwen3.8-flash-next 12,130,140 tokens [I: 241,346, O: 11,888,794, R: 11,640,590] choice accuracy 0.912 stderr 0.018 GLM-5.3-Flash-NVFP4 ╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │gpqa_diamond (198 x 4 samples): openai/glm-5-3-flash-nvfp4 │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ max_retries: 2, max_connections: 4, max_tokens: 131072, epochs: 4, fail_on_error: False, token_limit: 262144, time_limit: 10800, dataset: gpqa_diamond_74187e36ccadd6a06b1d98d13e064fed total time: 2 days, 4:57:44 openai/glm-5-3-flash-nvfp4 11,131,182 tokens [I: 213,432, O: 10,917,750, R: 10,774,610] choice accuracy 0.905 stderr 0.019 Experimental approach was very similar to what I already described here: https://blog.jonathanpage.com Qwen3.8 Flash Next is an extraordinary model that hits above its weight(s). Again I hope to blog about it when I have time but I made this game with it: https://games.jonathanpage.com/ I very much hope Qwen4 Flash will be open weights and run on ideally one Spark. Meanwhile the DeepSeek V4.1 Flash recipes requiring 3 or 4 Sparks are looking very tempting…

stop limiting clock and start editing fan curve hahaha

It’s weird I’ve an Asus and never got issues with stock GPU clock and stock FAN curve, my ambiante température is ~20°C in my room and my gpu is rarely above 80°C, we’re 3 to use the vLLM server and never got température issues.

Mine rarely gets above 65 :) [image] Cooler GB10 Temps - (almost) no performance lost DGX Spark / GB10 So a few times Iowered the clocks of my GB10 during long tasks to reduce the temps and I honestly never felt a lot (if any) performance lost, so today, I decided to actually measure: [image] I ran the same benchmark twice (codeneedle) in this case with Qwen3.6-35b-a3b-NVFP4. First run, normal clocks (hovered between 2400 and 2470mhz) and the 2nd run by limiting the clocks to 2000Mhz. Run 1 - Stock speed: GPU Power = 43w Speed (avg) = ~113tk/s you can see from the graph the orange line osc…

Today I tested the new Tensorfold engine and models–Vontra–Qwen3.8-Flash-Next-MLX-4bit-MTP. The engine itself is Very streamlined and minimalist compared to vLLM and does load fully in ~2 minutes. (Qwen3.8 Flash Next on one DGX Spark with TensorFold · Mia's AI Lab) GitHub - MiaAI-Lab/Qwen3.8-Flash-Next-Single-DGX-Spark-TensorFold: Qwen3.8 Flash Next on one DGX Spark (TensorFold) · GitHub It seems the tool eval bench is not fully compatible with Benchy, but the endpoint v1/ was an accurate test. Real world usage on Openclaw and Herms wfeels a tick slower on just all factors> Here are the best side by side results I could get against the INT4. Shape TensorFold (MLX-4bit) vLLM b12x (INT4-AutoRound) real turns (12×512-tok) mean 34.6 s/turn 8.6 s/turn real turns p50 / max 10.3 s / 118 s 9.4 s / 12 s decode stalls 5/12 (4–10 t/s) 0/12 clean-decode p50 ~55 t/s ~61 t/s acceptance (counter delta) 67.9% 77.3% micro c5 burst p50 e2e 3.65 s / 5.2 s 3.30 s / 3.50 s TF isn’t uniformly slower; it’s bimodal — near-parade (~55 t/s) when a decode graph path is hot, but 5/12 turns collapse into 4–10 t/s stalls in a memory-quiet state. b12x never stalls. That stall signature, not the eval’s +45% median, is what disqualifies it for production serving. (Still-unverified hypothesis: TF’s startup logged “0 decode graphs captured” — eager decode. If a future experiment ever reopens TF, that knob is the first thing to probe.)

Thanks for this recipe. We moved our two-Spark setup from the local-inference-lab Qwen3.8-Flash-Next NVFP4 checkpoint (LIL’s vLLM fork, TP2) to your INT4 AutoRound cluster recipe today. Sharing numbers plus three problems we hit, two with fixes and one still open, in case it helps others. Setup 2x DGX Spark (GB10), kernel 7.0.0-1019-nvidia, driver 580.178.04, current firmware, both 200G links up Eugr spark-vllm-docker @ 9296ced, pinned image eugr/spark-vllm-b12x@sha256:8e7e0621... qwen3.8-flash-next-int4-b12x.yaml (repo @ 8203768) unchanged except served names: TP2 + EP, MTP 5 greedy, block verify, 20g KV, PLE table on disk, RoCE all-reduce on Speed (streaming, usage token counts, thinking off) Test NVFP4 (LIL fork, TP2) INT4 cluster recipe Astro/Tailwind component, 1 stream 75 tok/s 116-128 tok/s Same, 4 streams total 189 293-316 llm-inference-bench decode C1/C4/C8 (0 ctx) 54 / 129 / 198 59 / 170 / 243 Same at 64K ctx 53 / 128 / 192 62 / 170 / 252 Prefill 8K / 128K 2,950 / 2,416 3,216 / 2,560 MTP accept length ~2.3 2.5-3.1 Engine steps/s are about the same on both, so the gain is all from drafting acceptance, which is why code sees the biggest jump. Quality: hotel-lights reasoning test 7/8 EXACT (NVFP4 was 8/8, one run answered 47 instead of 48) in 14.4 min vs 30.6 min, about 20% fewer thinking tokens per answer with the medium template. Tool calls 5/5 (qwen3_coder parser). Image input works. 90+ minutes of steady 3-stream load plus a 64K-token request every 5 min: 500+ requests, 0 failures, no throughput drift. Problem 1 (fixed): medium template rejects late system messages. medium_chat_template.jinja raises System message must be at the beginning. (HTTP 400) when a system message comes after the first turn. OpenCode and some other agent harnesses send those mid-conversation, so sessions die. Fix: copy the template and replace the raise_exception(...) line inside the if not loop.first branch with {{- ' system\n' + content + ' \n' }} Normal conversations render byte-identical to the stock template. We mount the patched copy with -v and point --chat-template at it. Problem 2 (fixed): launcher image-sync check false positive. launch-cluster.sh compares docker image inspect .Id between nodes. One of our nodes uses Docker’s containerd image store, which reports a different .Id for the identical pinned image, so the launch aborts with “image mismatch” even though both nodes show the same RepoDigests (8e7e0621...). We patched the check to also accept a shared registry digest. Re-pulling or docker save | load doesn’t help, because the IDs differ by storage driver. Problem 3 (open): hard, silent node freezes. Over one day we had 6 freezes across both nodes: no kernel messages, no Xid, no OOM, box pingable but SSH/HTTP dead, power cycle needed. The surviving node’s API shut down about 5 minutes later, and its RoCE local_ack_timeout_err / req_transport_retries_exceeded counters went up (we read that as a consequence of the peer dying, not the cause). The circumstances varied: 2 during model load, about 5 min in (warmup/graph capture), on the same node 2 during steady serving, one on each node, with 22-24 GB MemAvailable and flat memory for the preceding 15 min 2 while running ComfyUI (Krea image / LTX video) on the worker next to the model. Before one of these, the kernel logged NVRM: ... Out of memory [NV_ERR_NO_MEMORY] while MemAvailable still showed ~17 GB. Most of “available” was page cache (mmapped PLE table + weights), and MemFree was only single-digit GB. Things that helped or that we learned: vm.min_free_kbytes = 4194304 on both nodes, to keep real free pages for the GPU driver. Don’t co-run image/video models on these nodes with this recipe unless you trim KV; free memory is ~6-7 GB per node while serving. After a couple of hard power-offs, one node came back with the GPU pinned at 507 MHz / ~10 W under load (other node 2,490 MHz / 33 W) and an RTC read error at boot. That cut cluster speed ~40%. A full power drain (unplug brick 60 s) fixed both. Worth checking nvidia-smi --query-gpu=clocks.sm,power.draw on both nodes under load if your cluster numbers look low. For comparison, the NVFP4/LIL TP2 setup on the same boxes ran most of a week with two unexplained freezes, both on the same node (the same node that later froze during INT4 loads). So we can’t rule out a hardware or power issue on that node making things worse. We now have netconsole to the peer node, softlockup_panic, the hardware watchdog, a 5 s journald sync, and a 2 s logger for memory, GPU and RoCE counters running, to catch the next one. If anyone has seen hard freezes with this recipe, especially with VLLM_ENABLE_ROCE_ALLREDUCE=1, during the load/warmup phase or under steady serving, I’d like to compare notes. Our next test is the same recipe with the RoCE all-reduce off (NCCL only), to see if the freezes stop.

Interesting about the freezes. Never ever seen those in cluster (only OOMs while adjusting the recipe or if I had something else loaded beforehand) You might want to lower your clocks (see this thread) and test it again to see if that helps with stability or not. I’m always running my GB10’s at ~2000mhz (or 2150 when testing). Definitely it’s something that it worth investigating as the higher throughput/prefill that my model has compared to NVFP4 might cause it to run hotter and consume more.

If you want to use MLX format (from Metal) the best quant that it is there is Jundot (creator of oMLX) - maximum quality. Jundot/Qwen3.8-Flash-Next-oQ4e-mtp I quickly looked at vanilla mlx-style model structure (Vontra) and it appears that PLE table is contained inside the tensor set, so Jundot’s version should work. Otherwise for a separate .bin file loading format I have my own quant for mlx-serve - mix of Jundot’s and Ddalcu quants with a separate .bin table for PLE.

Up to 70tok/s Qwen3.8-Flash-Next-Int4-AutoRound

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10

azampatti October 2, 2026, 2:19pm 254

Mine rarely gets above 65 :)

Topic		Replies	Views
Qwen3.6-27B is out! DGX Spark / GB10 agentic-ai	300	36022	June 30, 2026
Single DGX-Spark - Qwen 3.8-Flash-Next at ~43tok/sec in Coding DGX Spark / GB10	79	17897	October 5, 2026
Bfloat16 Quality = Speed? DGX Spark / GB10	106	8688	May 26, 2026
Qwen3.8-Flash-Next DGX Spark / GB10	327	43728	October 23, 2026
Multi-Agent Workload: Qwen3.8-Flash-Next NVFP4 on 2× DGX Spark: 174 t/s aggregate, 74 % aider polyglot, 262K context DGX Spark / GB10 clustering , hp , spark , deepseek	11	1470	October 27, 2026
Fastest Qwen 3.5 122B Int4 recipe on DGX Spark tested and published on Spark-Arena DGX Spark / GB10 llama	59	5798	June 3, 2026
Qwen3.5 27B optimisation thread starting at 30+ t/s TP=1 DGX Spark / GB10 llama , agentic-ai	23	3616	May 11, 2026
Qwen3.8-Flash-Next recipe for 1x GB10, At mtp=3 c=1, code is ~47.5t/s, json is ~60t/s. Prefix cache OK. Intel AutoRound int4 quant DGX Spark / GB10 llama	12	2931	September 9, 2026
Qwen/Qwen3.6-35B-A3B (and FP8) has landed DGX Spark / GB10 agentic-ai	309	37042	June 22, 2026
Qwen3.8-Flash-FP8 dual sparks DGX Spark / GB10	65	5544	September 7, 2026

Up to 70tok/s Qwen3.8-Flash-Next-Int4-AutoRound

Related topics