Qwen3.8 Flash Next NVFP4 single spark

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10 Projects
1

Hi all, I’ve been doing some testing with Qwen3.8 Flash Next NVFP4 and offloading the n-gram table to nvme. I’ve finally got it working with a tweaked SGLang. Currently at about 14-15tps decode single request without mtp and it’s producing coherent output. I’ll do more work/testing today and share the modifications done once mtp is working. It’s my first post here, wanted to wait until I had something to contribute, so apologies if I’ve put this in the wrong location.

Update 1

Mtp is working. tps is roughly 18-19tps single stream decode in prose and 48-51 tps single stream decode in code.

Update 2

Cuda graphs are currently not working. I’m going to keep at it.

Update 3

Cuda graphs are now working! I’m going to organize this mess I’ve made into something deployable for those that want to try it out

2

I’ve had success with Qwen3.8-Flash-Next-UD-Q4_K_XL via llama.cpp though two things:

  • tool-eval-bench tests 66-68 fail hard with what looks to be chat template issues
  • i have had the model behave very well in some agentic coding until about 100k context then completely loses the plot, and even states something like “I don’t know where I am or what I was doing”

But this is running comparably to 3.8-27 for me so it could well be an improved daily driver.

3

Are you quantizing the KV cache? I ask because every time I quantize the cache the model loses the plot at about 100k context.

4

Thanks for the tip, but no, it’s fp16 kv cache. It’s early days though so it may resolve. Certainly the model looks quite good otherwise.

5

So, which do you prefer?
Qwen3.8-Flash-Next-GGUF/UD-IQ4_XS vs Qwen3.8-27b-FP8, they all can run on 96g VRAM.

6

I have been testing the vLLM-based single-Spark implementation from:

I am using the RadixArk NVFP4 checkpoint with the PLE table memory-mapped from NVMe. The model runs and generally produces coherent output, but I found a significant reproducibility issue with greedy decoding.

I ran the same 50 prompts three times, sequentially, with only one active request and:

  • temperature=0

  • top_p=1

  • identical prompts and server configuration

Results:

  • 13/50 prompts produced different outputs between runs

  • 5/50 changed the extracted value or evaluation score

  • two runs consumed the full 16,384-token reasoning budget and returned empty content

  • divergence often started at the very first reasoning token, sometimes including a Hungarian/English language switch

For comparison, the same test was completely stable with:

  • llama.cpp IQ4_XS: 0/50 differences

  • 35B FP8 on vLLM with MTP: 0/50

  • 122B NVFP4 on vLLM: 0/50

Disabling MTP did not solve the main issue: 11 of the 13 problematic prompts still varied, including four score-changing cases. However, the 16k-token empty-response failure disappeared, so MTP may be involved in that separate failure mode.

I am currently isolating CUDA graphs and the mmap workers, and checking first-token logprobs. Another hypothesis under investigation is the QSA persistent_topk path used on SM121, but I do not have a confirmed root cause yet.

Has anyone running the SGLang version on the same RadixArk checkpoint tested repeated identical greedy requests? It would be very useful to know whether the output and first-token logprobs remain stable with MTP enabled and disabled.

I can share the detailed test matrix and logs if useful.

7

Title: Non-deterministic prefill at temperature 0 on GB10: Qwen3.8-Flash-Next + vLLM persistent_topk (root cause + fix)

Qwen3.8-Flash-Next (NVFP4) served with vLLM on a DGX Spark: byte-identical greedy requests (temperature 0, single request in flight) return different outputs run to run. On my 50-task Hungarian extraction eval: 13/50 unstable, 5 of them with a different extracted date or amount. The same checkpoint via llama.cpp: 0/50.

Root cause, isolated one switch at a time: the QSA sparse-attention indexer’s top-k. On GB10 (compute capability family 12x) vLLM routes it to persistent_topk, and its atomicAdd slot assignment changes the selected top-2048 set between launches, so prefill logits move by up to 3 nats. Related open correctness bug: vllm#51782.

Quick test for your own stack: send the same prompt 10x with temperature=0, max_tokens=1, top_logprobs=20 and compare the top-20 lists byte-for-byte. Stock kernel: 0/6 probed prompts were stable, including ones that looked “stable” at suite level.

Fix that worked: replace the selection with torch.topk(sorted=False) + canonical tie ordering (one-file overlay on qsa.py, env-switched). Bit-identical outputs, prefill 2360 → 1760 tok/s (1.35x instead of the 2.9x a full sort costs), decode and MTP unchanged; full eval re-run 0/50 unstable.

Full write-up with the isolation tables and probe: Non-deterministic Qwen3.8-Flash-Next: the faulty vLLM kernel and its fix | DocAI Blog
Happy to test a branch on GB10.

8

Tool calls are looking good here. Cleaning things up and making sure this is deployable and not a “ThIs WoRkS oN mY mAcHiNe” issue

9

Im having issues with both prefill and concurrency on my build. Im going to stop working on this for the meantime. I know I said I would upload but I dont want to push something I feel is broken. Thanks for your support! Hopefully I’ll actually have something to contribute to you all in the future.