Has anyone tried Qwen 3.8 27B with DFlash2 on a DGX Spark yet?
And this one too, I have been using Mia-Labโs regular MTP branch, works well.
You will need to change the reasoning effort though. It defaults to xhigh, which is EXTREMELY HIGH thinking
yes, Iโm getting 40-42 tok/s
Finally something who gives a real difference, now itโs a little bit more comfortable with Dflash2 :
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:--------|---------------:|-----------------:|--------------:|-----------------:|-----------------:|-----------------:|
| Acantic | pp2048 | 2989.79 ยฑ 77.34 | | 846.90 ยฑ 30.65 | 637.01 ยฑ 30.65 | 846.90 ยฑ 30.65 |
| Acantic | tg32 | 45.25 ยฑ 7.50 | 46.71 ยฑ 7.75 | | | |
| Acantic | pp2048 | 2980.80 ยฑ 73.20 | | 844.18 ยฑ 28.66 | 634.29 ยฑ 28.66 | 844.18 ยฑ 28.66 |
| Acantic | tg128 | 27.33 ยฑ 1.19 | 27.67 ยฑ 0.94 | | | |
| Acantic | pp2048 @ d4096 | 2067.78 ยฑ 242.06 | | 2872.51 ยฑ 329.41 | 2662.62 ยฑ 329.41 | 2872.51 ยฑ 329.41 |
| Acantic | tg32 @ d4096 | 31.15 ยฑ 11.05 | 32.04 ยฑ 11.49 | | | |
| Acantic | pp2048 @ d4096 | 2043.14 ยฑ 209.36 | | 2990.48 ยฑ 344.56 | 2780.59 ยฑ 344.56 | 2990.48 ยฑ 344.56 |
| Acantic | tg128 @ d4096 | 28.75 ยฑ 1.00 | 29.33 ยฑ 0.94 | | | |
Workload matrix v1 โ endpoint http://127.0.0.1:8002 (model id: Acantic)
greedy, fresh single-turn prompts, decode net of prefill (two-call delta)
math (EN, eval-style) : 47.2 tok/s
code (EN) : 40.0 tok/s
code (DE) : 39.1 tok/s
technical explain (FR) : 24.0 tok/s
reasoning (FR) : 46.6 tok/s
free prose (EN) : 21.8 tok/s
free prose (FR) : 20.9 tok/s
free prose (DE) : 18.4 tok/s
Before with best MTP/Dspark config :
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:--------|---------------:|-----------------:|-------------:|----------------:|----------------:|----------------:|
| Acantic | pp2048 | 3004.54 ยฑ 146.02 | | 857.00 ยฑ 36.05 | 644.76 ยฑ 36.05 | 857.00 ยฑ 36.05 |
| Acantic | tg32 | 25.85 ยฑ 1.20 | 26.67 ยฑ 0.94 | | | |
| Acantic | pp2048 | 3090.33 ยฑ 83.71 | | 816.45 ยฑ 31.32 | 604.20 ยฑ 31.32 | 816.45 ยฑ 31.32 |
| Acantic | tg128 | 22.54 ยฑ 1.52 | 23.00 ยฑ 1.63 | | | |
| Acantic | pp2048 @ d4096 | 2676.93 ยฑ 9.34 | | 2294.11 ยฑ 55.94 | 2081.86 ยฑ 55.94 | 2294.11 ยฑ 55.94 |
| Acantic | tg32 @ d4096 | 22.14 ยฑ 3.63 | 22.67 ยฑ 3.77 | | | |
| Acantic | pp2048 @ d4096 | 2662.70 ยฑ 6.62 | | 2300.43 ยฑ 29.10 | 2088.18 ยฑ 29.10 | 2300.43 ยฑ 29.10 |
| Acantic | tg128 @ d4096 | 23.99 ยฑ 3.07 | 24.33 ยฑ 3.09 | | | |
math (EN, eval-style) : 38.8 tok/s
code (EN) : 31.0 tok/s
code (DE) : 27.0 tok/s
technical explain (FR) : 23.0 tok/s
reasoning (FR) : 31.4 tok/s
free prose (EN) : 16.1 tok/s
free prose (FR) : 15.8 tok/s
free prose (DE) : 13.4 tok/s
I tried applying the path using eugr vllm but itโs complaining about โhf_overrides must be a dict for get_quant_config to get the quantization config from it.โ Iโll try rebuilding the container again from scratch and applying the pr again, maybe I missed something
How does DFlash2 performance scale compared to MTP? At high context and/or high concurrency, are you guys seeing comparable performance for both PP and TG?
Currently waiting for mainline SGLang release to have DFlash2 support incorporated, with qwen38 support as well. There are some workarounds where you will have to build your own docker image. I would rather wait for main of SGLang to catch up and use the official docker image.
Currently using MTP, seems fine. But surely the default xhigh thinking effort is truly insane
After running it through my benchmarking tasks (simulated after what I do at work) I get a jump from dspark ~25 to dflash2 ~38tps (average weighted by spans of sglang actually giving me tokens, stuff like waiting for tool calls excluded). Itโs finally usable. MTP was slightly worse than dspark.
I decided to evaluate if 27B FP8 can be used as a main model on 2 sparks - I used it successfully on Metal but given the context and parameter limitation - not sure for 2 sparks. Tested Miaโs SGLang config in all possible variation.
Final result is real:
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ ๐ Benchmark Complete โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ โ
โ Model: qwen3.8-27b-fp8-tp2 โ
โ Score: 94 / 100 โ
โ Rating: โ
โ
โ
โ
โ
Excellent โ
โ Benchmark: tool-eval-bench v2.6.1.dev1+gedb37ba11 โ
โ Engine: SGLang โ
โ Quantization: FP8 โ
โ Max context: 262,144 tokens โ
โ โ
โ โ
78 passed โ ๏ธ 9 partial โ 1 failed โ
โ Points: 165/176 โ
โ โ
โ Quality: 94/100 โ
โ Responsiveness: 24/100 (median turn: 6.5s) โ
โ Deployability: 73/100 (ฮฑ=0.7) โ
โ Weakest: I Context & State (80%) โ
โ โ
โ Completed in 345.7s โ
โ โ
โ ๐ Token Usage: โ
โ Total: 668,157 tokens โ Efficiency: 0.2 pts/1K tokens โ
โ โ
โ โโ How this score is calculated โโ โ
โ โข Each scenario: pass=2pt, partial=1pt, fail=0pt โ
โ โข Category %: earned / max per category โ
โ โข Final score: (total points / max points) ร 100 โ
โ โข Deployability: 0.7รquality + 0.3รresponsiveness โ
โ โข Responsiveness: logistic curve (100 at <1s, ~50 at 3s, 0 at >10s) โ
โ โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Speed at 8 seqs = 200 t/s, faster than DS4F at 16.
Completion of the hardest test is also much faster - under 400s, vs 500-ish for 16-sec on DS4F.
Biggest caveat:
- use new version of the sglang image for dspark2
- DISABLE CUDA GRAPHS
With CUDA GRAPH you get up to 65 t/s (without - 50-55 t/s) but best result is 86/100, more errors, warnings. Dlash generate mismatches and quality degrades hard.
And another one for good measure - now with YaRN enabled and max length set to 500k - quality expected to take a hit but still
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ ๐ Benchmark Complete โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ โ
โ Model: qwen3.8-27b-fp8-tp2 โ
โ Score: 90 / 100 โ
โ Rating: โ
โ
โ
โ
โ
Excellent โ
โ Benchmark: tool-eval-bench v2.6.1.dev1+gedb37ba11 โ
โ Engine: SGLang โ
โ Quantization: FP8 โ
โ Max context: 500,000 tokens โ
โ โ
โ โ
73 passed โ ๏ธ 13 partial โ 2 failed โ
โ Points: 159/176 โ
โ โ
โ Quality: 90/100 โ
โ Responsiveness: 25/100 (median turn: 6.2s) โ
โ Deployability: 70/100 (ฮฑ=0.7) โ
โ Weakest: M Autonomous Planning (50%) โ
โ โ
โ Completed in 335.1s โ
โ โ
โ ๐ Token Usage: โ
โ Total: 626,194 tokens โ Efficiency: 0.2 pts/1K tokens โ
โ โ
โ โโ How this score is calculated โโ โ
โ โข Each scenario: pass=2pt, partial=1pt, fail=0pt โ
โ โข Category %: earned / max per category โ
โ โข Final score: (total points / max points) ร 100 โ
โ โข Deployability: 0.7รquality + 0.3รresponsiveness โ
โ โข Responsiveness: logistic curve (100 at <1s, ~50 at 3s, 0 at >10s) โ
โ โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Thatโs with --parallel 8
Since llama-benchy is not working with SGLang (now at least), home-grown long context bench delivered:
# SGLang Long-Context Benchmark
- Endpoint: `http://192.168.1.8:8889`
- Model: `qwen3.8-27b-fp8-tp2`
- Context targets: `[131072, 262144, 393216, 500000]` tokens
- Output tokens: `256`
- Repetitions: `1`
- Vocabulary size used for synthetic IDs: `248320`
- Request mode: serial native SGLang `/generate`, explicit `input_ids`, streamed
- Sampling: temperature 0, ignore EOS, fixed max_new_tokens
The prompt length is target minus reserved output tokens. This keeps the total request within the configured context limit.
SGLang server metrics are calculated as before/after Prometheus deltas; a row is contaminated if the delta cannot be isolated to one request.
## Per-case results
| Target | Prompt | Total | Status | Server prefill tok/s | Server decode tok/s | Server TTFT | Client TTFT | Client latency | Spec accept | Cache hit |
|---:|---:|---:|:---:|---:|---:|---:|---:|---:|---:|---:|
| 131072 | 130816 | 131072 | ok | 1102.811511 | 62.486947 | 0.231443 | 0.235976 | 4.318942 | 0.589286 | 130752.0 |
| 262144 | 261888 | 262144 | ok | 838.037716 | 55.166095 | 312.528957 | 312.547611 | 317.172349 | 0.807143 | 0.0 |
| 393216 | 392960 | 393216 | ok | 681.686403 | 34.413752 | 576.481964 | 576.499055 | 583.91214 | 0.392857 | 0.0 |
| 500000 | 499744 | 500000 | ok | 590.020716 | 21.977818 | 847.018759 | 847.039955 | 858.555737 | 0.253571 | 0.0 |
## Median by context target
| Target | Successful samples | Prefill tok/s | Decode tok/s | Server TTFT | Client latency |
|---:|---:|---:|---:|---:|---:|
| 131072 | 1/1 | 1102.81151 | 62.486947 | 0.231443 | 4.318942 |
| 262144 | 1/1 | 838.037716 | 55.166095 | 312.528957 | 317.172349 |
| 393216 | 1/1 | 681.686403 | 34.413752 | 576.481964 | 583.91214 |
| 500000 | 1/1 | 590.020716 | 21.977818 | 847.018759 | 858.555737 |
## Metric definitions
- **Server prefill tok/s** = exact uncached prompt tokens from the per-request histogram divided by `prefill_forward` stage seconds; the interval counter is used only if >
- **Server decode tok/s** = `inter_token_latency_seconds` observation count divided by its sum; this is steady decode cadence and excludes the first-token interval.
- **Decode metric tokens** includes SGLang scheduler decode work, including accepted speculative tokens; it is not necessarily identical to visible completion tokens.
- **Spec accept** is the post-request SGLang scheduler gauge and may represent the last scheduler logging window rather than a lifetime aggregate.
- **Cache hit** is the SGLang `prefill_cache` delta. A non-zero value means the case was not cold.
- Raw Prometheus before/after snapshots are in `metrics/`.
Here is my 2cents for my NVFP4 + Dflash2 run on llama-benchy for (2x DGX Spark cluster):
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|---|---|---|---|---|---|---|
| qwen38-27b-unsloth-nvfp4-dflash2 | pp2048 | 2932.26 ยฑ 103.58 | 652.03 ยฑ 13.02 | 649.20 ยฑ 13.02 | 652.03 ยฑ 13.02 | |
| qwen38-27b-unsloth-nvfp4-dflash2 | tg32 | 54.53 ยฑ 4.80 | 56.29 ยฑ 4.96 | |||
| qwen38-27b-unsloth-nvfp4-dflash2 | pp2048 @ d4096 | 3039.30 ยฑ 61.00 | 1856.55 ยฑ 74.86 | 1853.72 ยฑ 74.86 | 1856.55 ยฑ 74.86 | |
| qwen38-27b-unsloth-nvfp4-dflash2 | tg32 @ d4096 | 40.10 ยฑ 7.07 | 41.25 ยฑ 7.50 | |||
| qwen38-27b-unsloth-nvfp4-dflash2 | pp2048 @ d8192 | 2491.88 ยฑ 3.40 | 3746.95 ยฑ 35.85 | 3744.12 ยฑ 35.85 | 3746.95 ยฑ 35.85 | |
| qwen38-27b-unsloth-nvfp4-dflash2 | tg32 @ d8192 | 38.41 ยฑ 4.65 | 39.65 ยฑ 4.80 | |||
| qwen38-27b-unsloth-nvfp4-dflash2 | pp2048 @ d16384 | 2189.22 ยฑ 8.71 | 7630.57 ยฑ 62.97 | 7627.74 ยฑ 62.97 | 7632.29 ยฑ 62.88 | |
| qwen38-27b-unsloth-nvfp4-dflash2 | tg32 @ d16384 | 43.58 ยฑ 11.84 | 44.99 ยฑ 12.22 |
Tested with various depths. Let me know if anyone wants other llama-benchy tests with different configs.
Getting solid 2000-3000 tok/s prefill range with ~45 tok/s gen.