Qwen 3.8 27B + DFlash2

Has anyone tried Qwen 3.8 27B with DFlash2 on a DGX Spark yet?

just trying this one: GitHub - Weschera/Qwen3.8-27B-NVFP4-DFlash2-DGX-Spark: Reproducible Qwen3.8-27B NVFP4 + DFlash2 recipe and qualification evidence for one NVIDIA DGX Spark ยท GitHub

And this one too, I have been using Mia-Labโ€™s regular MTP branch, works well.

You will need to change the reasoning effort though. It defaults to xhigh, which is EXTREMELY HIGH thinking

yes, Iโ€™m getting 40-42 tok/s

Finally something who gives a real difference, now itโ€™s a little bit more comfortable with Dflash2 :

| model   |           test |              t/s |      peak t/s |        ttfr (ms) |     est_ppt (ms) |    e2e_ttft (ms) |
|:--------|---------------:|-----------------:|--------------:|-----------------:|-----------------:|-----------------:|
| Acantic |         pp2048 |  2989.79 ยฑ 77.34 |               |   846.90 ยฑ 30.65 |   637.01 ยฑ 30.65 |   846.90 ยฑ 30.65 |
| Acantic |           tg32 |     45.25 ยฑ 7.50 |  46.71 ยฑ 7.75 |                  |                  |                  |
| Acantic |         pp2048 |  2980.80 ยฑ 73.20 |               |   844.18 ยฑ 28.66 |   634.29 ยฑ 28.66 |   844.18 ยฑ 28.66 |
| Acantic |          tg128 |     27.33 ยฑ 1.19 |  27.67 ยฑ 0.94 |                  |                  |                  |
| Acantic | pp2048 @ d4096 | 2067.78 ยฑ 242.06 |               | 2872.51 ยฑ 329.41 | 2662.62 ยฑ 329.41 | 2872.51 ยฑ 329.41 |
| Acantic |   tg32 @ d4096 |    31.15 ยฑ 11.05 | 32.04 ยฑ 11.49 |                  |                  |                  |
| Acantic | pp2048 @ d4096 | 2043.14 ยฑ 209.36 |               | 2990.48 ยฑ 344.56 | 2780.59 ยฑ 344.56 | 2990.48 ยฑ 344.56 |
| Acantic |  tg128 @ d4096 |     28.75 ยฑ 1.00 |  29.33 ยฑ 0.94 |                  |                  |                  |
Workload matrix v1 โ€” endpoint http://127.0.0.1:8002 (model id: Acantic)
greedy, fresh single-turn prompts, decode net of prefill (two-call delta)

  math (EN, eval-style)   :  47.2 tok/s
  code (EN)               :  40.0 tok/s
  code (DE)               :  39.1 tok/s
  technical explain (FR)  :  24.0 tok/s
  reasoning (FR)          :  46.6 tok/s
  free prose (EN)         :  21.8 tok/s
  free prose (FR)         :  20.9 tok/s
  free prose (DE)         :  18.4 tok/s

Before with best MTP/Dspark config :

| model   |           test |              t/s |     peak t/s |       ttfr (ms) |    est_ppt (ms) |   e2e_ttft (ms) |
|:--------|---------------:|-----------------:|-------------:|----------------:|----------------:|----------------:|
| Acantic |         pp2048 | 3004.54 ยฑ 146.02 |              |  857.00 ยฑ 36.05 |  644.76 ยฑ 36.05 |  857.00 ยฑ 36.05 |
| Acantic |           tg32 |     25.85 ยฑ 1.20 | 26.67 ยฑ 0.94 |                 |                 |                 |
| Acantic |         pp2048 |  3090.33 ยฑ 83.71 |              |  816.45 ยฑ 31.32 |  604.20 ยฑ 31.32 |  816.45 ยฑ 31.32 |
| Acantic |          tg128 |     22.54 ยฑ 1.52 | 23.00 ยฑ 1.63 |                 |                 |                 |
| Acantic | pp2048 @ d4096 |   2676.93 ยฑ 9.34 |              | 2294.11 ยฑ 55.94 | 2081.86 ยฑ 55.94 | 2294.11 ยฑ 55.94 |
| Acantic |   tg32 @ d4096 |     22.14 ยฑ 3.63 | 22.67 ยฑ 3.77 |                 |                 |                 |
| Acantic | pp2048 @ d4096 |   2662.70 ยฑ 6.62 |              | 2300.43 ยฑ 29.10 | 2088.18 ยฑ 29.10 | 2300.43 ยฑ 29.10 |
| Acantic |  tg128 @ d4096 |     23.99 ยฑ 3.07 | 24.33 ยฑ 3.09 |                 |                 |                 |

  math (EN, eval-style)   :  38.8 tok/s
  code (EN)               :  31.0 tok/s
  code (DE)               :  27.0 tok/s
  technical explain (FR)  :  23.0 tok/s
  reasoning (FR)          :  31.4 tok/s
  free prose (EN)         :  16.1 tok/s
  free prose (FR)         :  15.8 tok/s
  free prose (DE)         :  13.4 tok/s

I tried applying the path using eugr vllm but itโ€™s complaining about โ€œhf_overrides must be a dict for get_quant_config to get the quantization config from it.โ€ Iโ€™ll try rebuilding the container again from scratch and applying the pr again, maybe I missed something

How does DFlash2 performance scale compared to MTP? At high context and/or high concurrency, are you guys seeing comparable performance for both PP and TG?

Currently waiting for mainline SGLang release to have DFlash2 support incorporated, with qwen38 support as well. There are some workarounds where you will have to build your own docker image. I would rather wait for main of SGLang to catch up and use the official docker image.

Currently using MTP, seems fine. But surely the default xhigh thinking effort is truly insane

After running it through my benchmarking tasks (simulated after what I do at work) I get a jump from dspark ~25 to dflash2 ~38tps (average weighted by spans of sglang actually giving me tokens, stuff like waiting for tool calls excluded). Itโ€™s finally usable. MTP was slightly worse than dspark.

I decided to evaluate if 27B FP8 can be used as a main model on 2 sparks - I used it successfully on Metal but given the context and parameter limitation - not sure for 2 sparks. Tested Miaโ€™s SGLang config in all possible variation.

Final result is real:

โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ ๐Ÿ† Benchmark Complete โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”‚                                                                                                                                                                         โ”‚
โ”‚    Model:  qwen3.8-27b-fp8-tp2                                                                                                                                          โ”‚
โ”‚    Score:  94 / 100                                                                                                                                                     โ”‚
โ”‚    Rating: โ˜…โ˜…โ˜…โ˜…โ˜… Excellent                                                                                                                                              โ”‚
โ”‚    Benchmark: tool-eval-bench v2.6.1.dev1+gedb37ba11                                                                                                                    โ”‚
โ”‚    Engine:       SGLang                                                                                                                                                 โ”‚
โ”‚    Quantization: FP8                                                                                                                                                    โ”‚
โ”‚    Max context:  262,144 tokens                                                                                                                                         โ”‚
โ”‚                                                                                                                                                                         โ”‚
โ”‚    โœ… 78 passed   โš ๏ธ  9 partial   โŒ 1 failed                                                                                                                           โ”‚
โ”‚    Points: 165/176                                                                                                                                                      โ”‚
โ”‚                                                                                                                                                                         โ”‚
โ”‚    Quality:        94/100                                                                                                                                               โ”‚
โ”‚    Responsiveness: 24/100  (median turn: 6.5s)                                                                                                                          โ”‚
โ”‚    Deployability:  73/100  (ฮฑ=0.7)                                                                                                                                      โ”‚
โ”‚    Weakest: I Context & State (80%)                                                                                                                                     โ”‚
โ”‚                                                                                                                                                                         โ”‚
โ”‚    Completed in 345.7s                                                                                                                                                  โ”‚
โ”‚                                                                                                                                                                         โ”‚
โ”‚    ๐Ÿ“Š Token Usage:                                                                                                                                                      โ”‚
โ”‚    Total: 668,157 tokens  โ”‚  Efficiency: 0.2 pts/1K tokens                                                                                                              โ”‚
โ”‚                                                                                                                                                                         โ”‚
โ”‚    โ”€โ”€ How this score is calculated โ”€โ”€                                                                                                                                   โ”‚
โ”‚    โ€ข Each scenario: pass=2pt, partial=1pt, fail=0pt                                                                                                                     โ”‚
โ”‚    โ€ข Category %: earned / max per category                                                                                                                              โ”‚
โ”‚    โ€ข Final score: (total points / max points) ร— 100                                                                                                                     โ”‚
โ”‚    โ€ข Deployability: 0.7ร—quality + 0.3ร—responsiveness                                                                                                                    โ”‚
โ”‚    โ€ข Responsiveness: logistic curve (100 at <1s, ~50 at 3s, 0 at >10s)                                                                                                  โ”‚
โ”‚                                                                                                                                                                         โ”‚
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ

Speed at 8 seqs = 200 t/s, faster than DS4F at 16.
Completion of the hardest test is also much faster - under 400s, vs 500-ish for 16-sec on DS4F.

Biggest caveat:

  • use new version of the sglang image for dspark2
  • DISABLE CUDA GRAPHS

With CUDA GRAPH you get up to 65 t/s (without - 50-55 t/s) but best result is 86/100, more errors, warnings. Dlash generate mismatches and quality degrades hard.

And another one for good measure - now with YaRN enabled and max length set to 500k - quality expected to take a hit but still

โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ ๐Ÿ† Benchmark Complete โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”‚                                                                                                                                                                         โ”‚
โ”‚    Model:  qwen3.8-27b-fp8-tp2                                                                                                                                          โ”‚
โ”‚    Score:  90 / 100                                                                                                                                                     โ”‚
โ”‚    Rating: โ˜…โ˜…โ˜…โ˜…โ˜… Excellent                                                                                                                                              โ”‚
โ”‚    Benchmark: tool-eval-bench v2.6.1.dev1+gedb37ba11                                                                                                                    โ”‚
โ”‚    Engine:       SGLang                                                                                                                                                 โ”‚
โ”‚    Quantization: FP8                                                                                                                                                    โ”‚
โ”‚    Max context:  500,000 tokens                                                                                                                                         โ”‚
โ”‚                                                                                                                                                                         โ”‚
โ”‚    โœ… 73 passed   โš ๏ธ  13 partial   โŒ 2 failed                                                                                                                          โ”‚
โ”‚    Points: 159/176                                                                                                                                                      โ”‚
โ”‚                                                                                                                                                                         โ”‚
โ”‚    Quality:        90/100                                                                                                                                               โ”‚
โ”‚    Responsiveness: 25/100  (median turn: 6.2s)                                                                                                                          โ”‚
โ”‚    Deployability:  70/100  (ฮฑ=0.7)                                                                                                                                      โ”‚
โ”‚    Weakest: M Autonomous Planning (50%)                                                                                                                                 โ”‚
โ”‚                                                                                                                                                                         โ”‚
โ”‚    Completed in 335.1s                                                                                                                                                  โ”‚
โ”‚                                                                                                                                                                         โ”‚
โ”‚    ๐Ÿ“Š Token Usage:                                                                                                                                                      โ”‚
โ”‚    Total: 626,194 tokens  โ”‚  Efficiency: 0.2 pts/1K tokens                                                                                                              โ”‚
โ”‚                                                                                                                                                                         โ”‚
โ”‚    โ”€โ”€ How this score is calculated โ”€โ”€                                                                                                                                   โ”‚
โ”‚    โ€ข Each scenario: pass=2pt, partial=1pt, fail=0pt                                                                                                                     โ”‚
โ”‚    โ€ข Category %: earned / max per category                                                                                                                              โ”‚
โ”‚    โ€ข Final score: (total points / max points) ร— 100                                                                                                                     โ”‚
โ”‚    โ€ข Deployability: 0.7ร—quality + 0.3ร—responsiveness                                                                                                                    โ”‚
โ”‚    โ€ข Responsiveness: logistic curve (100 at <1s, ~50 at 3s, 0 at >10s)                                                                                                  โ”‚
โ”‚                                                                                                                                                                         โ”‚
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ

Thatโ€™s with --parallel 8

Since llama-benchy is not working with SGLang (now at least), home-grown long context bench delivered:

# SGLang Long-Context Benchmark

- Endpoint: `http://192.168.1.8:8889`
- Model: `qwen3.8-27b-fp8-tp2`
- Context targets: `[131072, 262144, 393216, 500000]` tokens
- Output tokens: `256`
- Repetitions: `1`
- Vocabulary size used for synthetic IDs: `248320`
- Request mode: serial native SGLang `/generate`, explicit `input_ids`, streamed
- Sampling: temperature 0, ignore EOS, fixed max_new_tokens

The prompt length is target minus reserved output tokens. This keeps the total request within the configured context limit.
SGLang server metrics are calculated as before/after Prometheus deltas; a row is contaminated if the delta cannot be isolated to one request.

## Per-case results

| Target | Prompt | Total | Status | Server prefill tok/s | Server decode tok/s | Server TTFT | Client TTFT | Client latency | Spec accept | Cache hit |
|---:|---:|---:|:---:|---:|---:|---:|---:|---:|---:|---:|
| 131072 | 130816 | 131072 | ok | 1102.811511 | 62.486947 | 0.231443 | 0.235976 | 4.318942 | 0.589286 | 130752.0 |
| 262144 | 261888 | 262144 | ok | 838.037716 | 55.166095 | 312.528957 | 312.547611 | 317.172349 | 0.807143 | 0.0 |
| 393216 | 392960 | 393216 | ok | 681.686403 | 34.413752 | 576.481964 | 576.499055 | 583.91214 | 0.392857 | 0.0 |
| 500000 | 499744 | 500000 | ok | 590.020716 | 21.977818 | 847.018759 | 847.039955 | 858.555737 | 0.253571 | 0.0 |

## Median by context target

| Target | Successful samples | Prefill tok/s | Decode tok/s | Server TTFT | Client latency |
|---:|---:|---:|---:|---:|---:|
| 131072 | 1/1 | 1102.81151 | 62.486947 | 0.231443 | 4.318942 |
| 262144 | 1/1 | 838.037716 | 55.166095 | 312.528957 | 317.172349 |
| 393216 | 1/1 | 681.686403 | 34.413752 | 576.481964 | 583.91214 |
| 500000 | 1/1 | 590.020716 | 21.977818 | 847.018759 | 858.555737 |

## Metric definitions

- **Server prefill tok/s** = exact uncached prompt tokens from the per-request histogram divided by `prefill_forward` stage seconds; the interval counter is used only if >
- **Server decode tok/s** = `inter_token_latency_seconds` observation count divided by its sum; this is steady decode cadence and excludes the first-token interval.
- **Decode metric tokens** includes SGLang scheduler decode work, including accepted speculative tokens; it is not necessarily identical to visible completion tokens.
- **Spec accept** is the post-request SGLang scheduler gauge and may represent the last scheduler logging window rather than a lifetime aggregate.
- **Cache hit** is the SGLang `prefill_cache` delta. A non-zero value means the case was not cold.
- Raw Prometheus before/after snapshots are in `metrics/`.

Here is my 2cents for my NVFP4 + Dflash2 run on llama-benchy for (2x DGX Spark cluster):

model test t/s peak t/s ttfr (ms) est_ppt (ms) e2e_ttft (ms)
qwen38-27b-unsloth-nvfp4-dflash2 pp2048 2932.26 ยฑ 103.58 652.03 ยฑ 13.02 649.20 ยฑ 13.02 652.03 ยฑ 13.02
qwen38-27b-unsloth-nvfp4-dflash2 tg32 54.53 ยฑ 4.80 56.29 ยฑ 4.96
qwen38-27b-unsloth-nvfp4-dflash2 pp2048 @ d4096 3039.30 ยฑ 61.00 1856.55 ยฑ 74.86 1853.72 ยฑ 74.86 1856.55 ยฑ 74.86
qwen38-27b-unsloth-nvfp4-dflash2 tg32 @ d4096 40.10 ยฑ 7.07 41.25 ยฑ 7.50
qwen38-27b-unsloth-nvfp4-dflash2 pp2048 @ d8192 2491.88 ยฑ 3.40 3746.95 ยฑ 35.85 3744.12 ยฑ 35.85 3746.95 ยฑ 35.85
qwen38-27b-unsloth-nvfp4-dflash2 tg32 @ d8192 38.41 ยฑ 4.65 39.65 ยฑ 4.80
qwen38-27b-unsloth-nvfp4-dflash2 pp2048 @ d16384 2189.22 ยฑ 8.71 7630.57 ยฑ 62.97 7627.74 ยฑ 62.97 7632.29 ยฑ 62.88
qwen38-27b-unsloth-nvfp4-dflash2 tg32 @ d16384 43.58 ยฑ 11.84 44.99 ยฑ 12.22


Tested with various depths. Let me know if anyone wants other llama-benchy tests with different configs.

Getting solid 2000-3000 tok/s prefill range with ~45 tok/s gen.