Single DGX-Spark - Qwen 3.8-Flash-Next at ~43tok/sec in Coding

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Mind you, the context is tight, I have another variation that I’ll update tonight that lowers the quality JUST A BIT but allows for more context. You’ll be up and running in ~ 30 mins (there’s some work the script does by compressing the ‘table’ this model has).

One Liner:

curl -fsSL https://raw.githubusercontent.com/azampatti/GB10-3.8-Flash-Next/main/install-3.8flash.sh | bash && ~/launch-3.8flash.sh

This is just myself and Claude playing with FP8 heads (instead of BF16) and fitting a 200k context into this beautiful machine. I’m running more tests now but performance is QUITE nice :) I want to still push it a bit more towards the 50tok/sec if possible but I couldn’t find a kernel that gives me more than 20tok/s before MTP (it’s a huge part of the performance gain for this model).

Speed Example:

image
image798×172 10.6 KB

Repo:

(just run the installer)

You should have THIS NVFP4 model pre-downloaded.

Go test it out, give me feedback, enjoy!

2

BTW, launcher now gives you two options:

$ ./launch-3.8flash.sh 
  Which configuration?
    1) R=4 n-gram table  --  200k context   (default, best quality)
    2) PLE OFF           --  256k context, 4+ concurrent long sessions
  Choose [1]:

Option 2 is with the n-gram table completely off. Speed is the exact same but it supports 1M+ context (it varies but it showed me 1.5M the last time, so let’s just call it more than 1M).

Still measuring the quality drop, its noticeable more inconsistent but honestly it’s not drastic in my preliminary testing.

3

Hi @azampatti
What is your personal opinion about the quality and speed of this model compared to DeepSeek?

I used Qwen 3.5 for a long time(albond and prismaquant), and now I use DeepSeek (0xSnow - Mia AI Lab)

4

Out of curiousity, OP I believe you have 2 sparks at this point? But I see a lot of people with 2 sparks still optimizing single spark models. Is it just to get more agents running vs one better agent? Just pure curiousity on my part. I’m just still weighing whether I want to keep this second spark vs simply sticking with one machine. Spending 4k just to run bigger models when the smaller ones just keep getting better… it hurts.

Maybe another dumb question from a layman (me), but what is the advantage of this over this variant? GitHub - blazux/qwen3.8-Flash-DGX: recipe for running Qwen3.8-Flash-Next on a single DGX Spark · GitHub

I think one advantage is that this is multimodal as well. Supposedly. Whether this is of any use really depends on your use case. My experience so far with the blazux hybrid variant though is it takes way longer reasoning than DSV4F on “high” reasoning effort. On my 100k context categorization/data retrieval benchmark task it has run out of 50k reasoning tokens several times in a row now. I wonder if this one will perform any better, otherwise 200k context just will not be enough for it to even handle tasks. DSV4F in high reasoning has generally been finishing with 38-40k, and Qwen 3.5 122B was doing a slightly worse job in less than half that much.

5

Good for start. We can say, lunch Qwen3.8 Flash Next possible on single DGX Spark.

6

For me its too early to tell. I do see deepseek-flash paying the price of a very low quant in a single DGX with not-that-great consistency, but also codes really well.

I don’t have that many sessions in 3.8-flash to have a good opinion but 3.8-Flash it’s slightly behind DeepSeekV4-Flash in some tests I ran. But again, too early to tell for me.

7

I just have the one GB10 :) I’m still weighting into getting a 2nd one but I can’t justify one more so. It might never happen :)

The main difference between mine and Blazux variant, is the handling of n-gram and the quantization at the time of loading the model.

8

I don’t like DSv4 FLash and Pro for real life. In reality, we have only 40% success requests to this model. So I try to focus on much stable: Qwen and Gemma. Qwen better for performance optimization on single Spark.
I can get about 40+ t/s on single spark with qwen3.8-flash-next-nvfp4 or use my current qwen3.8-27b (50+ t/s for single and up to 200 t/s for parallel).
Both are the best for local cases and single DGX Spark enough.
As for me next better model in 500b around. So 2x DGX Spark just waste money, because you have limitation on 2x connection cases.

9

I’ve been working to optimize the 27B model on my single gx10 and though to give this recipe a try. It works very well – Thank you for sharing!

I realize a lot of optimization went into this recipe and wonder: is it safe to reduce the context back to 128k, reduce concurrent requests to 4 and reduce the mem fraction to make space for another smaller multimodal model? Are any settings that must strictly remain unchanged or in equilibrium with one another?

10

hello. I think you need to add --sleep-on-idle flag, otherwise SGLang keeps spinning on CPU when no inference is running

11

Thanks! Yeah, I never care to add it but I will :)

12

I have 2 Sparks and I find myself utilizing the single spark approach off and on depending on the model I am testing.

Like you said, there are some impressive models comfortably in the single spark space. With dual sparks I find myself trying out combinations of models of specific speciality and wiring them up with agent harnesses. The “brain” model is what is often discussed and benchmarked on these forums. It is rather experimental as I try and find the best combination of models to do what I am looking for. When the “brain” fits on a single spark, It communicates and works with its support models on the second spark over the CX-7 network connection.

The odd issue is the gap in model sizing. It is often tiny, or massive. There is very little in the in between. We see fewer 200Bish parameter models that we really should with systems like Spark on the market.

13

@azampatti I only want to say, big thanks. Trying your version for few days already and I have to say, it is really good. The best Qwen 3.8-Flash-Next I tried so far. (others were either too slow, or having bad quality) The speed is very good and quality-wise I see no problems, it give great results to me on both coding and general inference. Before I was using Qwen 3.8 27B and this model, especially your version beats it on my use cases

If anyone is still looking for a very good Qwen 3.8 Flash Next, I can recommend your work to everyone

14

+1 running it for 2 days now, 0 issues.

15

I also running this for 2 days with option 1, and no issues so far. Prefill/decoding speed is impressive for a single spark. Compare to vllm methods, this sglang method does not heat up my spark. it is only around 60 degrees celcius during long running agent work. This one + openhands = magic.

16

Thanks all for your feedback!

I can also recommend doing this Cooler GB10 Temps - (almost) no performance lost for better temperatures. All the numbers I published are my “underclock numbers”, so you can run 2000-2150Mhz and be cooler on your GB10’s

Also, stay put.. I’m working in a new version that it’s MUCH faster. I’m already working at 60tok/s in single stream, but I’m in the process of retraining the shared experts to improve the quality (similar process as I did here → A Faster (maybe better?) Qwen 3.5 122B ).

It’s still a few days away as reaching speed is easy, improving quality is hard :)

17

Cool and thank you! Looking forward to it.

18

Out of curiosity, I ran your sgbench.sh against my existing Qwen setup and thought I would share the results. I use this model with Hermes Agent on an ASUS GX10 with NVIDIA GB10 and 128 GB of unified memory.

My setup is Qwen3.8-Flash-Next, RadixArk NVFP4 with FP8 hybrid side layers, with the Sharp chat template loaded separately. I reused my existing locally converted RadixArk FP8-hybrid weights and took the chat template from travelinlance’s Hybrid + Sharp release.

Model links:

I run it through Docker Compose using the patched vLLM image from lancelind’s repository, which builds on blazux’s work:

Exact container image:

ghcr.io/lancelind/qwen38-flash-dgx@sha256:62a77b7c2806385cd23aeb7f6c4979b29f1b0db941bdc0f4791da90aa15f5bb9

My settings during the benchmark:

vLLM:                       0.1.dev20073+g8e685d198
NVIDIA driver:              580.159.03
API model name:             qwen3.8-flash-next
Configured max context:     500000, YaRN factor 4
Max concurrent sequences:   2
GPU memory utilization:     0.75
KV cache:                   auto / BF16
MTP / speculative decoding: disabled
Prefix caching:             enabled
Chunked prefill:             enabled
Max batched tokens:         8192
Chat template:              Sharp / froggeric v22.1

The following flags were also enabled:

VLLM_FP8_HYBRID=1
VLLM_PLE_MMAP=1
VLLM_PLE_GPU_GATHER=1
VLLM_PLE_DECODE_WARM=1
VLLM_QSA_EXACT_TOPK=1
VLLM_USE_FLASHINFER_SAMPLER=1
VLLM_USE_DEEP_GEMM=0

For the comparison, I used your benchmark:

I ran ~/sgbench.sh 1 directly on the GX10. The only script change was replacing port 30000 with my server’s port, 8010. The prompts, temperature, token limits, timing method, and two-run loop were unchanged. The model was already running; I did not restart or reconfigure it for this test.

Results from September 2, 2026:

Active model: qwen3.8-flash-next
Parallel streams: 1

── Run 1/2 ──────────────────────────────────────
  [Q&A] 256 tokens in 10.77s = 23.7 tok/s (prompt: 178)
  [Code] 511 tokens in 21.36s = 23.9 tok/s (prompt: 185)
  [JSON] 584 tokens in 24.14s = 24.1 tok/s (prompt: 203)
  [Math] 64 tokens in 2.82s = 22.6 tok/s (prompt: 184)
  [LongCode] 2048 tokens in 85.31s = 24.0 tok/s (prompt: 192)

── Run 2/2 ──────────────────────────────────────
  [Q&A] 256 tokens in 10.85s = 23.5 tok/s (prompt: 178)
  [Code] 511 tokens in 21.15s = 24.1 tok/s (prompt: 185)
  [JSON] 584 tokens in 23.75s = 24.5 tok/s (prompt: 203)
  [Math] 64 tokens in 2.80s = 22.8 tok/s (prompt: 184)
  [LongCode] 2048 tokens in 82.45s = 24.8 tok/s (prompt: 192)

=== Done ===

Run 2 produced 3,463 completion tokens in 141 seconds overall, or 24.56 tok/s. This uses total request time, including prefill.

For LongCode, I got 24.8 tok/s versus 42.7 tok/s in your screenshot, making your result approximately 1.72× faster on that workload.

The configurations differ: mine uses vLLM, a configured 500k context limit, the Sharp template, and no MTP; your setup uses SGLang, 200k context, HashK R=4, and NEXTN depth 3. I am treating this as a comparison of two specific serving configurations. The benchmark itself uses short prompts, so this run does not evaluate answer quality or performance with an actual 500k-token input.

19

I think it’s very important to clearly state that vision is disabled in this setup and that the model is launched with --language-only.

This is a major limitation, and for a lot of users it’s actually one of the key reasons to consider moving away from DS4F in the first place: DS4F has no vision support, even though as a text model it works extremely well and is sometimes even better than Qwen.

So it’s pretty disappointing to find out only after digging into the setup that vision is disabled here as well.

I would strongly recommend putting this at the very top of the GitHub README, ideally in the first line or first warning block, and also mentioning it explicitly in the post/title.

Right now this is very easy to miss. I didn’t see any obvious notice in the post, the title, or at first glance on GitHub saying that vision is disabled. This is not a minor implementation detail — for many people it completely changes whether this setup is useful to them.

Something as simple as this at the top would make it clear immediately:

⚠️ Vision is currently disabled. This setup runs Qwen3.8-Flash-Next with --language-only, so image input is not supported.

20

I get your result without MTP, you might want to double check on that.

But yeah, I’ve played with quantization in some of the BF16 heads to gain performance with minimal to no-loss.