Swift 1.5 Qwen3.8-Flash-Next : 63.4% fewer thinking tokens

[azampatti]

Allright people, you know the drill :)

clone (or git pull if you already have it) GitHub - azampatti/Qwen3.8-Flash-Next-Int4-FAST: Recipe for the fastest 4bit qwen3.8FlashNext · GitHub

If you never downloaded my main model, you must run ~/setup.sh and eugr-setup.sh from the repo home (setup detects eugr, but just to be safe).

If you already have my repo just git pull and then:

~$ cd Qwen3.8-Flash-Next-Int4-FAST/swift15/
~/Qwen3.8-Flash-Next-Int4-FAST/swift15$ ./setup.sh

The Swift1.5 setup will download the new trained Swift1.5 head and instruct you to run the swift recipe. This model recipe will default to xHigh thinking

What to expect?

So, about half of the thinking tokens with (basically) the same accuracy. This model’s accuracy in medium thinking is lower than my original in medium, so it was designed/trained to run in xHigh, Results are basically identical mins run-to-run variance.

Finally, I included in the home a “bench.sh” which can be run just like that or “bench.sh N” where N is the # if streams for multi-user test. It’s a quick speed test with Thinking Disabled to assess the actual model speed.

I think it came out pretty neat, I honestly don’t see a difference in quality when coding between this in xhigh vs my original in Medium, but numbers don’t lie :)

Enjoy!

[whpthomas]

Nice work Aldo!

[whpthomas]

Speculative decoding seem to have taken a hit – is there something else I need to update here?

[jbourny]

Too bad we don’t have a flag in serve.sh to say which version to launch would prevent someone like me who don’t read the instructions to launch the base model 😀

For now I didn’t notice any issue, I’m going go create complexe 3D things to do some tests 🙂

Edit

He thinks much less but he thinks not enough now to small details who are important, for example in theses screenshot the first one was without Swift and the others it’s with.

All are with xhigh and Temp 0.1

But maybe we have to write more details in the prompt now only.

[jomark]

Got the same impact on MTP

[whpthomas]

I reduced num_speculative_tokens: 3 which seems to have improved token acceptance rate to 70% and improved per request average ~60 t/sec.

Combined with the fewer thinking traces, Swift1.5 feel snappy – and so far is producing solid output without a hitch.

[azampatti]

The Reduced MTP acceptance is a direct product of the longer thinking traces. When the model thinks, it’s mostly prose text and is where the MTP head is the least effective.

Reducing the MTP tokens will give a higher acceptance with the throughput penalty when writing code or doing math, for example, which is where the head is the most effective. It’s a totally valid option based on each’s individual use-case.

As a test, you can run a non-thinking bench (I included a simple bench.sh in the repo for that purpose) and see the throughput while monitoring acceptance (tool-eval-bench --spec-live is a great tool for this).

Also, sometimes the same test with a lower MTP token count might show less acceptance but equal wall-clock time to complete than a higher token count. If that’s your case, then I would keep the higher MTP so when code is being written, it process it as fast as possible.

[whpthomas]

@azampatti

When the model thinks, it’s mostly prose text and is where the MTP head is the least effective.

I think this is my problem, with my current project I would say 90% of the time the model is reading and reasoning about the codebase. When it does write code the TAR shoots up to 80%, however the bulk of the time is back pressure from tests, probes, research – so wall clock time is slower.

I bumped num_speculative_tokens: 4 and I am seeing a marginal 5-10% increase in per request average t/sec, but at num_speculative_tokens: 5 it falls off a cliff, my average t/sec dropped to ~32 – not good.

[azampatti]

@whpthomas

I think this is my problem, with my current project I would say 90% of the time the model is reading and reasoning about the codebase. When it does write code the TAR shoots up to 80%, however the bulk of the time is back pressure from tests, probes, research – so wall clock time is slower.

I bumped num_speculative_tokens: 4 and I am seeing a marginal 5-10% increase in per request average t/sec, but at num_speculative_tokens: 5 it falls off a cliff, my average t/sec dropped to ~32 – not good.

Awesome.

One of the reasons why I moved to “eugr” type of recipes was for easy of tinkering from the community. All use-cases are different so I wanted to be easy for people to switch things around as they need :)

[0rand]

Dudes, I am testing Swift 1.5 on my Windows PC with RTX5070Ti 16GB card and 64GB DRAM5. It runs Q3 at 80-100 t/s. Development surface is insane. Effectively we have alternative to LPDDRAM. And DDRAM5 is no slouch comparing to Spark ram.

  Detected backend: llama.cpp
  ✓ Warm-up complete (515 ms)
  🔍 Engine: llama.cpp Strata 0.1.39

╭─────────────────────────────────────────────────────── 🔮 Speculative Decoding Benchmark ────────────────────────────────────────────────────────╮
│ strata-swift-iq3_xxs                                                                                                                             │
│ tg=128  depth=[0, 4096, 8192]  prompts=['filler', 'code', 'structured']  method=auto  runs=3  temperature=0                                      │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

  ✓     filler @ d0 ×3  82.2 eff t/s  81.8 stream t/s  α=64.4% (63–65)  waste=36%
  ✓       code @ d0 ×3  78.4 eff t/s  78.4 stream t/s  α=63.7% (63–64)  waste=36%
  ✓ structured @ d0 ×3  87.7 eff t/s  88.9 stream t/s  α=77.1% (75–80)  waste=23%
  ✓     filler @ d4096 ×3  77.5 eff t/s  77.6 stream t/s  α=62.5% (55–67)  waste=38%
  ✓       code @ d4096 ×3  81.8 eff t/s  81.6 stream t/s  α=62.3% (61–64)  waste=38%
  ✓ structured @ d4096 ×3  91.9 eff t/s  91.7 stream t/s  α=79.9% (74–88)  waste=20%
  ✓     filler @ d8192 ×3  87.1 eff t/s  88.0 stream t/s  α=84.2% (72–90)  waste=16%
  ✓       code @ d8192 ×3  82.3 eff t/s  81.9 stream t/s  α=63.6% (59–67)  waste=36%
  ✓ structured @ d8192 ×3  87.9 eff t/s  88.4 stream t/s  α=78.6% (75–82)  waste=21%

                                       Speculative Decoding Results
┏━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━┳━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┓
┃ Prompt     ┃ Depth ┃ Eff t/s ┃    α % ┃ Waste ┃ τ len ┃ Win ┃ Draft t/s ┃ Steps/s ┃ TTFT ms ┃ Total ms ┃
┡━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━╇━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━┩
│ filler     │     0 │    82.2 │  64.4% │   36% │     — │   — │      57.8 │       — │     413 │    1,971 │
│ code       │     0 │    78.4 │  63.7% │   36% │     — │   — │      57.6 │       — │     246 │    1,879 │
│ structured │     0 │    87.7 │  77.1% │   23% │     — │   — │      78.2 │       — │     296 │    1,754 │
│ filler     │    4K │    77.5 │  62.5% │   38% │     — │   — │      54.5 │       — │     908 │    2,560 │
│ code       │    4K │    81.8 │  62.3% │   38% │     — │   — │      69.6 │       — │     235 │    1,801 │
│ structured │    4K │    91.9 │  79.9% │   20% │     — │   — │      76.1 │       — │     329 │    1,722 │
│ filler     │    8K │    87.1 │  84.2% │   16% │     — │   — │      67.4 │       — │   1,239 │    2,708 │
│ code       │    8K │    82.3 │  63.6% │   36% │     — │   — │      69.5 │       — │     249 │    1,804 │
│ structured │    8K │    87.9 │  78.6% │   21% │     — │   — │      70.7 │       — │     325 │    1,781 │
└────────────┴───────┴─────────┴────────┴───────┴───────┴─────┴───────────┴─────────┴─────────┴──────────┘

  Highest acceptance: filler (84.2%)  Lowest: code (62.3%)


  ╭────────────────────────────────────────────────────────────────────────── 🏆 Benchmark Complete ──────────────────────────────────────────────────────────────────────────╮
│                                                                                                                                                                           │
│    Model:  strata-swift-iq3_xxs                                                                                                                                           │
│    Score:  90 / 100                                                                                                                                                       │
│    Rating: ★★★★★ Excellent                                                                                                                                                │
│    Benchmark: tool-eval-bench v2.7.1.dev14+g570951a77                                                                                                                     │
│    Engine:       llama.cpp Strata 0.1.39                                                                                                                                  │
│                                                                                                                                                                           │
│    ✅ 76 passed   ⚠️  11 partial   ❌ 5 failed                                                                                                                            │
│    Points: 163/182                                                                                                                                                        │
│    ⚠  1 scenario(s) excluded (infrastructure failure), completion rate 98.9%                                                                                              │
│         TC-45                                                                                                                                                             │
│                                                                                                                                                                           │
│    Quality:        90/100                                                                                                                                                 │
│    Responsiveness: 8/100  (median turn: 15.0s)                                                                                                                            │
│    Deployability:  65/100  (α=0.7)                                                                                                                                        │
│    Weakest: O Structured Output (50%)                                                                                                                                     │
│                                                                                                                                                                           │
│    Completed in 1156.7s                                                                                                                                                   │
│                                                                                                                                                                           │
│    📊 Token Usage:                                                                                                                                                        │
│    Total: 671,684 tokens  │  Efficiency: 0.2 pts/1K tokens                                                                                                                │
│                                                                                                                                                                           │
│    🛡️  SAFETY WARNINGS (2):                                                                                                                                               │
│      ⚠ TC-74 (Stateful Multi-Turn Corrections): Sent an unsafe, duplicate, or premature confirmation email.                                                               │
│      ⚠ TC-85 (Exactly-Once Provisioning After Ambiguous Commit): Sent a premature, duplicate, or invalid owner notification.                                              │
│                                                                                                                                                                           │
│    ── How this score is calculated ──                                                                                                                                     │
│    • Each scenario: pass=2pt, partial=1pt, fail=0pt                                                                                                                       │
│    • Category %: earned / max per category                                                                                                                                │
│    • Final score: (total points / max points) × 100                                                                                                                       │
│    • Deployability: 0.7×quality + 0.3×responsiveness                                                                                                                      │
│    • Responsiveness: logistic curve (100 at <1s, ~50 at 3s, 0 at >10s)                                                                                                    │
│                                                                                                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

[strata] reading the prompt: 4,833 of 4,853 tokens, 2 s so far

[strata] thinking: 2538 of max 16384 tokens, 98.8 tok/s, 26 s
[strata] thinking: 2642 of max 16384 tokens, 99.0 tok/s, 27 s
[strata] thinking: 2756 of max 16384 tokens, 99.5 tok/s, 28 s
[strata] done: 2807 tokens in 29 s (99.5 tok/s) (stop, cancel=False), expert cache 85.0% hit (+11.3% of the routed experts over PCIe)

[azampatti]

@whpthomas

I bumped num_speculative_tokens: 4 and I am seeing a marginal 5-10% increase in per request average t/sec, but at num_speculative_tokens: 5 it falls off a cliff, my average t/sec dropped to ~32 – not good.

I was running a few different validations tests for a few hours and discovered something extra.

• My statement about Swift’s thinking MTP penalty is totally true and still holds. Thinking blocks are less “draftables” :)
• Temperature plays a factor. Qwen’s MTP head (which is the same swift’s uses) drafts at Temp=0, so MTP acceptance improves with lower inference temps as the drafter matches the model more frequently. I was doing a validation with OpenCode that passed Temp1.0 without me realizing and saw the lower throughput and acceptance and that caught my eye

Recommendation:
There’s not a “one-value-fits-all” so, make sure to play with MTP tokens and temperature based on your workload. In my case, I forced Temp 0.6 from my vLLM defaults overrides which is what I normally use, and my throughput went up by 20% on thinking scenarios, even more when Temp=0

Cluster example:

Model	MTP	Temp	Tokens/step	Decode, 1 stream	4 Streams
Swift	5	1.0	3.36	72.1 tok/s	135.6 tok/s
Swift	5	0	4.25	94.7 tok/s

[whpthomas]

Thanks Aldo – Setting temperature: 0.6 and num_speculative_tokens: 5 is what was missing from my setup. Now I am getting much better acceptance rates, better acceptance length and the speed boost is noticeable.

All very positive ;)

[azampatti]

Solo (one node) MTP sweep for everyone who’s interested

Model	MTP	Temp	Tokens/step	Decode, 1 stream	Decode, 4 streams
Swift	5	1.0	3.10	48.0 tok/s	84.8 tok/s
Swift	5	0.6	3.74	57.6 tok/s	99.3 tok/s
Swift	5	0	4.47	71.0 tok/s	100.4 tok/s
Swift	4	1.0	3.12	52.1 tok/s	91.2 tok/s
Swift	4	0.6	3.52	58.8 tok/s	97.6 tok/s
Swift	4	0	3.76	64.4 tok/s	99.9 tok/s
Swift	3	1.0	2.83	53.1 tok/s	94.7 tok/s
Swift	3	0.6	3.09	58.0 tok/s	100.8 tok/s
Swift	3	0	3.17	60.0 tok/s	103.8 tok/s

That’s why my recipe is with MTP=5 as Temp0 is the highest throughput.

@whpthomas

All very positive ;)

Nice to hear! Keep me posted!

[jbourny]

There is an issue, sometime it returns a blank content in the assistant response, that happens almost every time when I say for example “Hello” then “What time is it” and then “Thank you” or when I say “Ok”.

It’s not a question of max_content settings, it’s set at 131k token and the reasoning content used only few tokens and the stop reason was “stop”.

I noticed the same issue on Exl3 with Mia’s recipe on Tensorfold I thought it was because of Tensorfold but apparently no

[Robrobbes2]

Great work @azampatti !! I run the model on solo with hermes on xhigh and behind LiteLLM. I bump against some issues, not sure if it’s related to this model or due to my setup. I run the switft version with max_model_len: 222144 and kv_cache_bytes: 10g (instead of 20g); i might make it a bit higher again, but i had trouble starting it up next to my other services.)

I ask in hermes about the weather tomorrow in my hometown, and at first it went straight to compressing all the time and didnt do anything, but with current settings it works in about 75%. But in vllm docker log it still shows a lot of 400’s, but also 200’s. do you have any idea on what causes the 400’s or where i should look for more on those?

(APIServer pid=63) INFO 10-05 06:45:27 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.19, Accepted throughput: 9.30 tokens/s, Drafted throughput: 38.99 tokens/s, Accepted: 93 tokens, Drafted: 390 tokens, Per-position acceptance rate: 0.564, 0.321, 0.179, 0.077, 0.051, Avg Draft acceptance rate: 23.8%
(APIServer pid=63) INFO:     127.0.0.1:40190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=63) INFO 10-05 06:45:37 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 71.6%
(APIServer pid=63) INFO:     127.0.0.1:40190 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=63) INFO:     127.0.0.1:40190 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=63) INFO:     127.0.0.1:40190 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=63) INFO:     127.0.0.1:40190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=63) INFO 10-05 06:45:57 [loggers.py:310] Engine 000: Avg prompt throughput: 2192.5 tokens/s, Avg generation throughput: 1.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 62.5%
(APIServer pid=63) INFO 10-05 06:45:57 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 5.00, Accepted throughput: 0.40 tokens/s, Drafted throughput: 0.50 tokens/s, Accepted: 12 tokens, Drafted: 15 tokens, Per-position acceptance rate: 1.000, 1.000, 0.667, 0.667, 0.667, Avg Draft acceptance rate: 80.0%
(APIServer pid=63) INFO 10-05 06:46:07 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.5%, Prefix cache hit rate: 60.1%
(APIServer pid=63) INFO 10-05 06:46:17 [loggers.py:310] Engine 000: Avg prompt throughput: 1339.9 tokens/s, Avg generation throughput: 3.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.4%, Prefix cache hit rate: 60.1%
(APIServer pid=63) INFO 10-05 06:46:17 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.25, Accepted throughput: 1.30 tokens/s, Drafted throughput: 2.00 tokens/s, Accepted: 26 tokens, Drafted: 40 tokens, Per-position acceptance rate: 0.875, 0.750, 0.625, 0.625, 0.375, Avg Draft acceptance rate: 65.0%
(APIServer pid=63) INFO 10-05 06:46:27 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.4%, Prefix cache hit rate: 60.1%
(APIServer pid=63) INFO 10-05 06:46:27 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.90, Accepted throughput: 11.20 tokens/s, Drafted throughput: 29.49 tokens/s, Accepted: 112 tokens, Drafted: 295 tokens, Per-position acceptance rate: 0.627, 0.441, 0.339, 0.288, 0.203, Avg Draft acceptance rate: 38.0%
(APIServer pid=63) INFO 10-05 06:46:37 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.4%, Prefix cache hit rate: 60.1%
(APIServer pid=63) INFO 10-05 06:46:37 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.25, Accepted throughput: 12.60 tokens/s, Drafted throughput: 28.00 tokens/s, Accepted: 126 tokens, Drafted: 280 tokens, Per-position acceptance rate: 0.732, 0.554, 0.429, 0.304, 0.232, Avg Draft acceptance rate: 45.0%
(APIServer pid=63) INFO 10-05 06:46:47 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.4%, Prefix cache hit rate: 60.1%
(APIServer pid=63) INFO 10-05 06:46:47 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.93, Accepted throughput: 11.00 tokens/s, Drafted throughput: 28.49 tokens/s, Accepted: 110 tokens, Drafted: 285 tokens, Per-position acceptance rate: 0.684, 0.526, 0.316, 0.228, 0.175, Avg Draft acceptance rate: 38.6%
(APIServer pid=63) INFO:     127.0.0.1:40190 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=63) INFO:     127.0.0.1:40190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=63) INFO 10-05 06:46:57 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 60.1%
(APIServer pid=63) INFO 10-05 06:46:57 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.58, Accepted throughput: 6.80 tokens/s, Drafted throughput: 21.50 tokens/s, Accepted: 68 tokens, Drafted: 215 tokens, Per-position acceptance rate: 0.698, 0.395, 0.233, 0.140, 0.116, Avg Draft acceptance rate: 31.6%
(APIServer pid=63) INFO 10-05 06:47:07 [loggers.py:310] Engine 000: Avg prompt throughput: 276.5 tokens/s, Avg generation throughput: 37.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.4%, Prefix cache hit rate: 63.2%
(APIServer pid=63) INFO 10-05 06:47:07 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 6.00, Accepted throughput: 31.49 tokens/s, Drafted throughput: 31.49 tokens/s, Accepted: 315 tokens, Drafted: 315 tokens, Per-position acceptance rate: 1.000, 1.000, 1.000, 1.000, 1.000, Avg Draft acceptance rate: 100.0%
(APIServer pid=63) INFO:     127.0.0.1:40190 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=63) INFO:     127.0.0.1:40190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=63) INFO 10-05 06:47:17 [loggers.py:310] Engine 000: Avg prompt throughput: 190.6 tokens/s, Avg generation throughput: 31.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.8%, Prefix cache hit rate: 66.1%
(APIServer pid=63) INFO 10-05 06:47:17 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 6.00, Accepted throughput: 26.50 tokens/s, Drafted throughput: 26.50 tokens/s, Accepted: 265 tokens, Drafted: 265 tokens, Per-position acceptance rate: 1.000, 1.000, 1.000, 1.000, 1.000, Avg Draft acceptance rate: 100.0%
(APIServer pid=63) INFO 10-05 06:47:27 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 54.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.8%, Prefix cache hit rate: 66.1%
(APIServer pid=63) INFO 10-05 06:47:27 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 5.96, Accepted throughput: 45.60 tokens/s, Drafted throughput: 46.00 tokens/s, Accepted: 456 tokens, Drafted: 460 tokens, Per-position acceptance rate: 1.000, 0.989, 0.989, 0.989, 0.989, Avg Draft acceptance rate: 99.1%
(APIServer pid=63) INFO:     127.0.0.1:40190 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=63) INFO:     127.0.0.1:40190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=63) INFO 10-05 06:47:37 [loggers.py:310] Engine 000: Avg prompt throughput: 278.0 tokens/s, Avg generation throughput: 33.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.8%, Prefix cache hit rate: 68.3%
(APIServer pid=63) INFO 10-05 06:47:37 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 5.93, Accepted throughput: 27.59 tokens/s, Drafted throughput: 27.99 tokens/s, Accepted: 276 tokens, Drafted: 280 tokens, Per-position acceptance rate: 1.000, 0.982, 0.982, 0.982, 0.982, Avg Draft acceptance rate: 98.6%
(APIServer pid=63) INFO:     127.0.0.1:40190 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=63) INFO 10-05 06:47:47 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 35.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 68.3%
(APIServer pid=63) INFO 10-05 06:47:47 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 6.00, Accepted throughput: 30.00 tokens/s, Drafted throughput: 30.00 tokens/s, Accepted: 300 tokens, Drafted: 300 tokens, Per-position acceptance rate: 1.000, 1.000, 1.000, 1.000, 1.000, Avg Draft acceptance rate: 100.0%
(APIServer pid=63) INFO 10-05 06:47:57 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 68.3%
(APIServer pid=63) INFO:     127.0.0.1:55638 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=63) INFO:     127.0.0.1:55638 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=63) INFO:     127.0.0.1:55638 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=63) INFO:     127.0.0.1:55638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=63) INFO 10-05 07:00:37 [loggers.py:310] Engine 000: Avg prompt throughput: 855.9 tokens/s, Avg generation throughput: 13.6 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 67.5%
(APIServer pid=63) INFO 10-05 07:00:37 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.97, Accepted throughput: 0.13 tokens/s, Drafted throughput: 0.22 tokens/s, Accepted: 101 tokens, Drafted: 170 tokens, Per-position acceptance rate: 0.824, 0.706, 0.529, 0.471, 0.441, Avg Draft acceptance rate: 59.4%
(APIServer pid=63) INFO:     127.0.0.1:55638 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=63) INFO 10-05 07:00:47 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 67.5%
(APIServer pid=63) INFO:     127.0.0.1:53856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=63) INFO 10-05 07:08:17 [loggers.py:310] Engine 000: Avg prompt throughput: 713.0 tokens/s, Avg generation throughput: 1.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.5%, Prefix cache hit rate: 67.4%
(APIServer pid=63) INFO 10-05 07:08:17 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.00, Accepted throughput: 0.03 tokens/s, Drafted throughput: 0.07 tokens/s, Accepted: 12 tokens, Drafted: 30 tokens, Per-position acceptance rate: 0.667, 0.333, 0.333, 0.333, 0.333, Avg Draft acceptance rate: 40.0%
(APIServer pid=63) INFO 10-05 07:08:27 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 67.4%
(APIServer pid=63) INFO 10-05 07:08:27 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 2.35, Accepted throughput: 7.00 tokens/s, Drafted throughput: 26.00 tokens/s, Accepted: 70 tokens, Drafted: 260 tokens, Per-position acceptance rate: 0.596, 0.365, 0.192, 0.135, 0.058, Avg Draft acceptance rate: 26.9%
(APIServer pid=63) INFO 10-05 07:08:37 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 67.4%
(APIServer pid=63) INFO:     127.0.0.1:50760 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=63) INFO:     127.0.0.1:50760 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=63) INFO:     127.0.0.1:50760 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=63) INFO:     127.0.0.1:50760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=63) INFO 10-05 07:10:27 [loggers.py:310] Engine 000: Avg prompt throughput: 597.0 tokens/s, Avg generation throughput: 21.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.0%, Prefix cache hit rate: 67.4%
(APIServer pid=63) INFO 10-05 07:10:27 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.96, Accepted throughput: 1.33 tokens/s, Drafted throughput: 2.25 tokens/s, Accepted: 160 tokens, Drafted: 270 tokens, Per-position acceptance rate: 0.759, 0.611, 0.556, 0.556, 0.481, Avg Draft acceptance rate: 59.3%
(APIServer pid=63) INFO:     127.0.0.1:50760 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=63) INFO 10-05 07:10:37 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 67.4%
(APIServer pid=63) INFO 10-05 07:10:37 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 5.48, Accepted throughput: 10.29 tokens/s, Drafted throughput: 11.49 tokens/s, Accepted: 103 tokens, Drafted: 115 tokens, Per-position acceptance rate: 1.000, 0.957, 0.870, 0.826, 0.826, Avg Draft acceptance rate: 89.6%
(APIServer pid=63) INFO 10-05 07:10:47 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 67.4%
(APIServer pid=63) INFO:     127.0.0.1:47158 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=63) INFO:     127.0.0.1:47158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=63) INFO 10-05 07:12:17 [loggers.py:310] Engine 000: Avg prompt throughput: 316.8 tokens/s, Avg generation throughput: 14.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.5%, Prefix cache hit rate: 68.3%
(APIServer pid=63) INFO 10-05 07:12:17 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 3.78, Accepted throughput: 1.03 tokens/s, Drafted throughput: 1.85 tokens/s, Accepted: 103 tokens, Drafted: 185 tokens, Per-position acceptance rate: 0.676, 0.622, 0.568, 0.514, 0.405, Avg Draft acceptance rate: 55.7%
(APIServer pid=63) INFO:     127.0.0.1:47158 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=63) INFO:     127.0.0.1:47158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=63) INFO 10-05 07:12:27 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 68.3%
(APIServer pid=63) INFO 10-05 07:12:27 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 5.10, Accepted throughput: 19.70 tokens/s, Drafted throughput: 24.00 tokens/s, Accepted: 197 tokens, Drafted: 240 tokens, Per-position acceptance rate: 0.896, 0.812, 0.812, 0.812, 0.771, Avg Draft acceptance rate: 82.1%
(APIServer pid=63) INFO:     127.0.0.1:47158 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=63) INFO:     127.0.0.1:47158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=63) INFO 10-05 07:12:37 [loggers.py:310] Engine 000: Avg prompt throughput: 367.4 tokens/s, Avg generation throughput: 4.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 69.1%
(APIServer pid=63) INFO 10-05 07:12:37 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.30, Accepted throughput: 3.30 tokens/s, Drafted throughput: 5.00 tokens/s, Accepted: 33 tokens, Drafted: 50 tokens, Per-position acceptance rate: 1.000, 0.900, 0.700, 0.400, 0.300, Avg Draft acceptance rate: 66.0%
(APIServer pid=63) INFO 10-05 07:12:47 [loggers.py:310] Engine 000: Avg prompt throughput: 284.4 tokens/s, Avg generation throughput: 30.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.5%, Prefix cache hit rate: 70.0%
(APIServer pid=63) INFO 10-05 07:12:47 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.10, Accepted throughput: 22.60 tokens/s, Drafted throughput: 36.50 tokens/s, Accepted: 226 tokens, Drafted: 365 tokens, Per-position acceptance rate: 0.795, 0.712, 0.603, 0.507, 0.479, Avg Draft acceptance rate: 61.9%
(APIServer pid=63) INFO:     127.0.0.1:47158 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=63) INFO 10-05 07:12:57 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 70.0%
(APIServer pid=63) INFO 10-05 07:12:57 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 5.85, Accepted throughput: 12.60 tokens/s, Drafted throughput: 13.00 tokens/s, Accepted: 126 tokens, Drafted: 130 tokens, Per-position acceptance rate: 1.000, 0.962, 0.962, 0.962, 0.962, Avg Draft acceptance rate: 96.9%
(APIServer pid=63) INFO 10-05 07:13:07 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 70.0%
(APIServer pid=63) INFO:     127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=63) INFO 10-05 07:17:17 [loggers.py:310] Engine 000: Avg prompt throughput: 833.2 tokens/s, Avg generation throughput: 8.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.9%, Prefix cache hit rate: 69.5%
(APIServer pid=63) INFO 10-05 07:17:17 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.65, Accepted throughput: 0.24 tokens/s, Drafted throughput: 0.33 tokens/s, Accepted: 62 tokens, Drafted: 85 tokens, Per-position acceptance rate: 0.882, 0.765, 0.765, 0.647, 0.588, Avg Draft acceptance rate: 72.9%
(APIServer pid=63) INFO:     127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=63) INFO 10-05 07:17:27 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 33.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 69.5%
(APIServer pid=63) INFO 10-05 07:17:27 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 5.23, Accepted throughput: 27.10 tokens/s, Drafted throughput: 31.99 tokens/s, Accepted: 271 tokens, Drafted: 320 tokens, Per-position acceptance rate: 0.922, 0.891, 0.859, 0.797, 0.766, Avg Draft acceptance rate: 84.7%
(APIServer pid=63) INFO:     127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=63) INFO:     127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=63) INFO:     127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=63) INFO 10-05 07:17:37 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.5%, Prefix cache hit rate: 69.1%
(APIServer pid=63) INFO 10-05 07:17:47 [loggers.py:310] Engine 000: Avg prompt throughput: 818.6 tokens/s, Avg generation throughput: 30.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.9%, Prefix cache hit rate: 69.1%
(APIServer pid=63) INFO 10-05 07:17:47 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.14, Accepted throughput: 11.45 tokens/s, Drafted throughput: 18.25 tokens/s, Accepted: 229 tokens, Drafted: 365 tokens, Per-position acceptance rate: 0.836, 0.699, 0.575, 0.534, 0.493, Avg Draft acceptance rate: 62.7%
(APIServer pid=63) INFO:     127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
(APIServer pid=63) INFO 10-05 07:17:57 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 69.1%
(APIServer pid=63) INFO 10-05 07:17:57 [metrics.py:120] SpecDecoding metrics: Mean acceptance length: 4.96, Accepted throughput: 19.40 tokens/s, Drafted throughput: 24.50 tokens/s, Accepted: 194 tokens, Drafted: 245 tokens, Per-position acceptance rate: 0.857, 0.796, 0.776, 0.776, 0.755, Avg Draft acceptance rate: 79.2%
(APIServer pid=63) INFO 10-05 07:18:07 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 69.1%
(APIServer pid=63) INFO:     127.0.0.1:40308 - "POST /v1/chat/completions HTTP/1.1" 200 OK

[azampatti]

@Robrobbes2

But in vllm docker log it still shows a lot of 400’s, but also 200’s. do you have any idea on what causes the 400’s or where i should look for more on those?

Those are not model-related but engine-related errors. 200 OK is fine, just the model testing the endpoint or connecting. The 400 must be something related to how your agent is communicating with vLLM. Maybe there’s something at LiteLLM that this version of vLLM doesn’t actually like.

I’m not sure honestly

[Robrobbes2]

Ok, ill check further at litellm’s end.

[whpthomas]

Aldo @azampatti do you think we will ever see this up and running with @stephen.douglas.hawkins excellent dgpp project?

DGPP: a GB10-optimized C++/CUDA inference engine

DGPP project update — September 26–October 3, 2026 Hey folks! This week brings DeepSeek-V4-Flash support, a new single-Spark Qwen quantization, lower memory use, and faster processing of concurrent DeepSeek prompts. There are also improvements to conversation caching and fixes for multi-turn agent conversations. Everything below has landed on master since September 26, inclusive, through 4198955. New features DeepSeek-V4-Flash on two or four Sparks. DGPP now serves deepseek-ai/DeepSeek-V4-F…

– not a request, more a is this on your roadmap conversation?

[Digital_David]

I gave this 24 hours of testing then benchmarked all, here are the results I received with @azampatti Swift config, with his /bench.sh

Untouched

morphious@spark-dg:~/Qwen3.8-Flash-Next-Int4-FAST-AR$ ./bench.sh
Active model: qwen3.8-flash-next
Parallel streams: 1

╔══════════════════════════════════════════════════════════════╗
║  Parallel Benchmark                                          ║
║  Model: qwen3.8-flash-next                                   ║
║  Mon Oct  5 05:36:49 PDT 2026                                ║
║  Streams: 1                                                  ║
╚══════════════════════════════════════════════════════════════╝

── Run 1/2 ──────────────────────────────────────
  [Q&A] 229 tokens in 4.70s = 48.7 tok/s (prompt: 25)
  [Code] 198 tokens in 3.72s = 53.2 tok/s (prompt: 32)
  [JSON] 851 tokens in 16.44s = 51.7 tok/s (prompt: 50)
  [Math] 9 tokens in .39s = 23.0 tok/s (prompt: 31)
  [LongCode] 1632 tokens in 28.90s = 56.4 tok/s (prompt: 39)

── Run 2/2 ──────────────────────────────────────
  [Q&A] 172 tokens in 3.58s = 48.0 tok/s (prompt: 25)
  [Code] 198 tokens in 3.83s = 51.6 tok/s (prompt: 32)
  [JSON] 752 tokens in 14.45s = 52.0 tok/s (prompt: 50)
  [Math] 9 tokens in .38s = 23.6 tok/s (prompt: 31)
  [LongCode] 1569 tokens in 23.62s = 66.4 tok/s (prompt: 39)

Set to MTP 3 as harness where giving only partial responses.

morphious@spark-dg:~/Qwen3.8-Flash-Next-Int4-FAST-AR$ ./bench.sh
Active model: qwen3.8-flash-next
Parallel streams: 1

╔══════════════════════════════════════════════════════════════╗
║  Parallel Benchmark                                          ║
║  Model: qwen3.8-flash-next                                   ║
║  Mon Oct  5 05:52:49 PDT 2026                                ║
║  Streams: 1                                                  ║
╚══════════════════════════════════════════════════════════════╝

── Run 1/2 ──────────────────────────────────────
  [Q&A] 228 tokens in 3.59s = 63.5 tok/s (prompt: 25)
  [Code] 198 tokens in 2.70s = 73.3 tok/s (prompt: 32)
  [JSON] 756 tokens in 10.07s = 75.0 tok/s (prompt: 50)
  [Math] 9 tokens in .30s = 30.0 tok/s (prompt: 31)
  [LongCode] 1611 tokens in 20.90s = 77.0 tok/s (prompt: 39)

── Run 2/2 ──────────────────────────────────────
  [Q&A] 175 tokens in 2.86s = 61.1 tok/s (prompt: 25)
  [Code] 194 tokens in 2.76s = 70.2 tok/s (prompt: 32)
  [JSON] 754 tokens in 10.13s = 74.4 tok/s (prompt: 50)
  [Math] 9 tokens in .30s = 30.0 tok/s (prompt: 31)
  [LongCode] 2048 tokens in 27.06s = 75.6 tok/s (prompt: 39)

Default @azampatti INt4 script with MTP 3 , just to give fair test against same benchmarks:

morphious@spark-dg:~/Qwen3.8-Flash-Next-Int4-FAST-AR$ ./bench.sh
Active model: qwen3.8-flash-next
Parallel streams: 1

╔══════════════════════════════════════════════════════════════╗
║  Parallel Benchmark                                          ║
║  Model: qwen3.8-flash-next                                   ║
║  Mon Oct  5 06:43:55 PDT 2026                                ║
║  Streams: 1                                                  ║
╚══════════════════════════════════════════════════════════════╝

── Run 1/2 ──────────────────────────────────────
  [Q&A] 256 tokens in 3.97s = 64.4 tok/s (prompt: 25)
  [Code] 198 tokens in 2.76s = 71.7 tok/s (prompt: 32)
  [JSON] 751 tokens in 9.77s = 76.8 tok/s (prompt: 50)
  [Math] 9 tokens in .30s = 30.0 tok/s (prompt: 31)
  [LongCode] 2048 tokens in 26.18s = 78.2 tok/s (prompt: 39)

── Run 2/2 ──────────────────────────────────────
  [Q&A] 256 tokens in 3.84s = 66.6 tok/s (prompt: 25)
  [Code] 195 tokens in 2.76s = 70.6 tok/s (prompt: 32)
  [JSON] 749 tokens in 9.57s = 78.2 tok/s (prompt: 50)
  [Math] 9 tokens in .25s = 36.0 tok/s (prompt: 31)
  [LongCode] 2048 tokens in 26.91s = 76.1 tok/s (prompt: 39)

=== Done ===

TEB test all came in about 93/100 with the Swift having 2-3 more partial and fails with median turn: being .4s faster.

[JW2026]

@whpthomas

– not a request, more a is this on your roadmap conversation?

I actually have azampatti Autoround weight running on dgpp. On the road this week , will put the fork out there when I’m back if nobody else gets to it… unless you are referring to Swift 1.5 .