Qwen3.5 27B optimisation thread starting at 30+ t/s TP=1

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
,
1

I’ve been following the @Albond thread on the 122B model optimisations but for me the Intel autoround quant is not usable for agentic tasks like using OpenCode because of a bug with that quant where it just stops generating in the middle of tool calls. There is also a lot of talk recently about how the dense 27B model can perform better than the 122B MoE so I was keen to try it out.

Initially I found the performance is unacceptably slow for 27B. I went through osoleve/Qwen3.5-27B-Text-NVFP4-MTP and got around 20 t/s with MTP but that one is text only and I regularly use the great vision capabilities of Qwen3.5. I tried the Intel/Qwen3.5-27B-int4-AutoRound but that one does not have MTP enabled and it is a lot slower than osoleve at ~13 t/s.

Now I see the recent DFlash speculative decoding https://huggingface.co/z-lab/Qwen3.5-27B-DFlash and decided to give it a try with spark-vllm-docker and found it is working quite well with Intel/Qwen3.5-27B-int4-AutoRound.

hf download z-lab/Qwen3.5-27B-DFlash

Note their suggested vllm parameters, particularly the “model” property refers to their speculative decoding model:

vllm serve Intel/Qwen3.5-27B-int4-AutoRound \
  --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.5-27B-DFlash", "num_speculative_tokens": 15}' \
  --attention-backend flash_attn \
  --max-num-batched-tokens 32768

I’m getting about 30 T/s with general queries but the degree of speculative decoding at 15 means it is quite variable. I did some benchmarks, firstly I used the bench_qwen35.sh from the albond github https://github.com/albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4/blob/master/bench_qwen35.sh

╔══════════════════════════════════════════════════════╗
║  Qwen3.5-27B-Intel-int4-DFlash Benchmark: test
║  Wed  8 Apr 11:24:33 UTC 2026
╚══════════════════════════════════════════════════════╝

── Run 1/2 ──────────────────────────────────────
  [Q&A] 256 tokens in 5.69s = 44.9 tok/s (prompt: 23)
  [Code] 507 tokens in 7.80s = 65.0 tok/s (prompt: 30)
  [JSON] 1024 tokens in 22.25s = 46.0 tok/s (prompt: 48)
  [Math] 64 tokens in 1.76s = 36.3 tok/s (prompt: 29)
  [LongCode] 2048 tokens in 31.45s = 65.1 tok/s (prompt: 37)

── Run 2/2 ──────────────────────────────────────
  [Q&A] 256 tokens in 5.69s = 44.9 tok/s (prompt: 23)
  [Code] 507 tokens in 7.81s = 64.9 tok/s (prompt: 30)
  [JSON] 1024 tokens in 22.18s = 46.1 tok/s (prompt: 48)
  [Math] 64 tokens in 1.76s = 36.3 tok/s (prompt: 29)
  [LongCode] 2048 tokens in 31.46s = 65.0 tok/s (prompt: 37)

You can see that fairly predictable text like coding perform very well, 60+ t/s.

I also ran llama-benchy to see the hit that the speculative decoding makes on the underlying t/s:

| model                            |            test |            t/s |     peak t/s |         ttfr (ms) |      est_ppt (ms) |     e2e_ttft (ms) |
|:---------------------------------|----------------:|---------------:|-------------:|------------------:|------------------:|------------------:|
| Intel/Qwen3.5-27B-int4-AutoRound |          pp2048 | 1004.03 ± 3.08 |              |    2191.17 ± 6.74 |    2041.12 ± 6.74 |    2191.28 ± 6.73 |
| Intel/Qwen3.5-27B-int4-AutoRound |            tg32 |    8.71 ± 0.58 | 10.18 ± 0.69 |                   |                   |                   |
| Intel/Qwen3.5-27B-int4-AutoRound |  pp2048 @ d4096 | 1032.69 ± 0.74 |              |    6100.56 ± 4.27 |    5950.51 ± 4.27 |    6100.63 ± 4.27 |
| Intel/Qwen3.5-27B-int4-AutoRound |    tg32 @ d4096 |    9.15 ± 0.83 | 11.05 ± 1.20 |                   |                   |                   |
| Intel/Qwen3.5-27B-int4-AutoRound |  pp2048 @ d8192 | 1030.67 ± 2.35 |              |  10086.03 ± 23.07 |   9935.98 ± 23.07 |  10086.17 ± 23.07 |
| Intel/Qwen3.5-27B-int4-AutoRound |    tg32 @ d8192 |    8.40 ± 0.70 |  9.84 ± 1.02 |                   |                   |                   |
| Intel/Qwen3.5-27B-int4-AutoRound | pp2048 @ d16384 | 1014.41 ± 1.18 |              |  18320.31 ± 21.74 |  18170.26 ± 21.74 |  18320.41 ± 21.71 |
| Intel/Qwen3.5-27B-int4-AutoRound |   tg32 @ d16384 |    8.14 ± 0.43 |  9.36 ± 0.41 |                   |                   |                   |
| Intel/Qwen3.5-27B-int4-AutoRound | pp2048 @ d32768 |  921.86 ± 6.12 |              | 37919.98 ± 252.11 | 37769.93 ± 252.11 | 37920.08 ± 252.12 |
| Intel/Qwen3.5-27B-int4-AutoRound |   tg32 @ d32768 |    7.45 ± 0.01 |  8.00 ± 0.00 |                   |                   |                   |

Some vllm output of the speculative decoding acceptance rates for 16 tokens:

(APIServer pid=55) INFO:     172.19.0.2:42192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=55) INFO 04-08 10:47:03 [loggers.py:259] Engine 000: Avg prompt throughput: 511.5 tokens/s, Avg generation throughput: 28.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=55) INFO 04-08 10:47:03 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 5.74, Accepted throughput: 23.70 tokens/s, Drafted throughput: 79.99 tokens/s, Accepted: 237 tokens, Drafted: 800 tokens, Per-position acceptance rate: 0.940, 0.840, 0.780, 0.560, 0.380, 0.260, 0.260, 0.240, 0.200, 0.080, 0.040, 0.040, 0.040, 0.040, 0.020, 0.020, Avg Draft acceptance rate: 29.6%
(APIServer pid=55) INFO 04-08 10:47:13 [loggers.py:259] Engine 000: Avg prompt throughput: 797.9 tokens/s, Avg generation throughput: 16.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.1%, Prefix cache hit rate: 0.0%
(APIServer pid=55) INFO 04-08 10:47:13 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.68, Accepted throughput: 11.80 tokens/s, Drafted throughput: 70.40 tokens/s, Accepted: 118 tokens, Drafted: 704 tokens, Per-position acceptance rate: 0.773, 0.591, 0.455, 0.227, 0.182, 0.136, 0.114, 0.114, 0.045, 0.023, 0.023, 0.000, 0.000, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 16.8%
(APIServer pid=55) INFO 04-08 10:47:23 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 33.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.1%, Prefix cache hit rate: 0.0%
(APIServer pid=55) INFO 04-08 10:47:23 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 4.24, Accepted throughput: 25.60 tokens/s, Drafted throughput: 126.39 tokens/s, Accepted: 256 tokens, Drafted: 1264 tokens, Per-position acceptance rate: 0.810, 0.570, 0.443, 0.380, 0.304, 0.203, 0.139, 0.127, 0.114, 0.063, 0.038, 0.038, 0.013, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 20.3%
(APIServer pid=55) INFO 04-08 10:47:33 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 32.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.3%, Prefix cache hit rate: 0.0%
(APIServer pid=55) INFO 04-08 10:47:33 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 4.15, Accepted throughput: 24.90 tokens/s, Drafted throughput: 126.39 tokens/s, Accepted: 249 tokens, Drafted: 1264 tokens, Per-position acceptance rate: 0.823, 0.608, 0.418, 0.342, 0.278, 0.253, 0.228, 0.089, 0.063, 0.013, 0.013, 0.013, 0.013, 0.000, 0.000, 0.000, Avg Draft acceptance rate: 19.7%
(APIServer pid=55) INFO 04-08 10:47:43 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 33.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.3%, Prefix cache hit rate: 0.0%

Recipe:

description: vLLM serving Intel/Qwen3.5-27B-int4-AutoRound with DFlash
model: Intel/Qwen3.5-27B-int4-AutoRound
container: vllm-node-tf5
build_args:
  - '--tf5'
mods:
  - mods/fix-qwen3.5-chat-template
defaults:
  port: 8000
  host: 0.0.0.0
  tensor_parallel: 1
  gpu_memory_utilization: 0.75
  max_model_len: 262144
  max_num_seqs: 10
env:
command: |
  vllm serve Intel/Qwen3.5-27B-int4-AutoRound \
      --tensor-parallel-size {tensor_parallel} \
      --max-num-seqs {max_num_seqs} \
      --enable-prefix-caching \
      --gpu-memory-utilization {gpu_memory_utilization} \
      --max-model-len {max_model_len} \
      --enable-auto-tool-choice \
      --dtype auto \
      --tool-call-parser qwen3_coder \
      --reasoning-parser qwen3 \
      --chat-template unsloth.jinja \
      --load-format fastsafetensors \
      --attention-backend flash_attn \
      --max-num-batched-tokens 32768 \
      --trust-remote-code \
      --host {host} \
      --port {port}
recipe_version: '1'
name: Intel/Qwen3.5-27B-int4-AutoRound-DFlash
cluster_only: false
solo_only: false

Run command:

./run-recipe.sh qwen3.5-27b-int4-autoround --solo --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.5-27B-DFlash", "num_speculative_tokens": 16}'

I’m not an expert on python, I come from a dotnet background so I’m hoping some of the guru’s here will be able to tweak this even further. I don’t think people on spark give 27B much of a look because of the speed. Hopefully this DFlash speculative decode will ignite some renewed interest in the 27B dense here. For me it is currently the optimum on a single spark with DFlash.

Any ideas on how to improve on this Intel autoround + DFlash?

2

Huh, I need to go back to try the Autoround then - I got this running with the FP8 and had trouble with acceptance rates (10-20%). Thought it might be inherent to the Flash_Attn backend which is required, but maybe there’s a FP8 specific bug somewhere.

I think DFlash is going to be outright transformative for the Spark platform.

Biggest downside right now is the flash_attn backend. It’s hard required, and cuts down context efficiency. Seems to increase some overhead in the setting of concurrent queries as well.

3

You can see my acceptance rates in the log were 15-30%

Don’t know much about this but isn’t it a trade off between a reduction in the underlying t/s being outweighed by the accepted draft token rate? I’m not trying to optimise the % acceptance rate but the number accepted minus the reduction in the underlying t/s.

4

Wow, I tried and indeed the speed up in TG is great! What’s the catch? Why don’t we hear more about DFlash? Seems to me it’s a bigger deal than TurboQuant for us.

5

So, using the exact same call and the official Intel Autoround quant, I am seeing a lot of spiraling output, extremely low acceptance rates, and very poor responses from this combination.

The official FP8 quant is stable/resistant to this. The int4 Autoround is NOT. The drafter is trained on the full model; FP8 is preserved but I don’t think the int4-autoround is close enough. I’ve A/B tested with and without this MTP and some of this behavior is actually just the int4 quant being suboptimal. Even when it doesn’t spiral off into uselessness, what I think is happening is that the int4 is not as confident in what it should be saying when verifying the drafts, and the MTP suggestions lead it down wrong paths.

Use this combination with caution. I would not take it into production.

Instead, go download the official FP8 quant and use it with DFlash instead. You’ll end up with about 240k instead of 262k context, with token rates still way above int4-AutoRound even at the low acceptance we’re seeing right now. With FP8 model and this drafter, I get 15-18 tok/s.

6

Both will be “big deals” for us.

DFlash is new, these models are being trained by one academic lab. How new? 27B-DFlash drafter was made publicly available barely 24 hours ago (it hit HF 8 days back, but was gated and they didn’t seem to be approving requests but rather just removed the gate yesterday).

7

Well, I didn’t think anyone would think a spark-vllm-docker recipe with it’s 0.19.1rc1.xyz untested wheels is supposed to be a production setup. This is about finding out what is possible with optimisation.

What were your low acceptance rates for int4 that were worse than fp8? I think people need to reassess what counts as low when using this drafter as you can afford for low rates with so many tokens predicted.

I’ve not seen any bad quality output so far in my testing, but I’m very interested in objectively quantifying any quality loss. I’m new to these forums so what is a sensible objective test that can be done at home with 1 spark? Tokens/s vs quality is an important tradeoff. I don’t think I need to go to the slow FP8 so far but yes I’m sure I’m losing some quality for the increased t/s but how much?

8

recipe in dual spark:

description: vLLM serving Intel/Qwen3.5-27B-int4-AutoRound with DFlash
model: Intel/Qwen3.5-27B-int4-AutoRound
container: vllm-node-tf5
build_args:
  - '--tf5'
mods:
  - mods/fix-qwen3.5-chat-template
defaults:
  port: 8000
  host: 0.0.0.0
  tensor_parallel: 2
  gpu_memory_utilization: 0.75
  max_model_len: 262144
  max_num_seqs: 10
env:
command: |
  vllm serve Intel/Qwen3.5-27B-int4-AutoRound \
      --tensor-parallel-size {tensor_parallel} --distributed-executor-backend ray\
      --max-num-seqs {max_num_seqs} \
      --enable-prefix-caching \
      --gpu-memory-utilization {gpu_memory_utilization} \
      --max-model-len {max_model_len} \
      --enable-auto-tool-choice \
      --dtype auto \
      --tool-call-parser qwen3_coder \
      --reasoning-parser qwen3 \
      --chat-template unsloth.jinja \
      --load-format fastsafetensors \
      --attention-backend flash_attn \
      --max-num-batched-tokens 32768 \
      --trust-remote-code \
      --host {host} \
      --port {port}
recipe_version: '1'
name: Intel/Qwen3.5-27B-int4-AutoRound-DFlash
cluster_only: false
solo_only: false

start with:

./run-recipe.sh qwen3.5-27b-int4-autoround --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.5-27B-DFlash", "num_speculative_tokens": 16}'

benchmark:

── Run 1/2 ──────────────────────────────────────
  [Q&A] 256 tokens in 4.06s = 63.0 tok/s (prompt: 23)
  [Code] 507 tokens in 5.37s = 94.4 tok/s (prompt: 30)
  [JSON] 1024 tokens in 17.09s = 59.9 tok/s (prompt: 48)
  [Math] 64 tokens in 1.23s = 52.0 tok/s (prompt: 29)
  [LongCode] 2048 tokens in 21.99s = 93.1 tok/s (prompt: 37)

── Run 2/2 ──────────────────────────────────────
  [Q&A] 256 tokens in 3.96s = 64.6 tok/s (prompt: 23)
  [Code] 507 tokens in 5.52s = 91.8 tok/s (prompt: 30)
  [JSON] 1024 tokens in 16.25s = 63.0 tok/s (prompt: 48)
  [Math] 64 tokens in 1.22s = 52.4 tok/s (prompt: 29)
  [LongCode] 2048 tokens in 22.54s = 90.8 tok/s (prompt: 37)
9

I did some tests to evaluate any change to the output quality of applying DFlash to the int4 autoround using DeepEval and 3 subjects selected at random from MMLU. With hindsight MMLU maybe isn’t the best test because modern LLM’s find it fairly easy and so it is harder to differentiate between a lot of high scores, what is noise etc. These tests take several hours (630 tests in total) so I’ll just present what I’ve got for now.

Intel/Qwen3.5-27B-int4-AutoRound with DFlash:

Overall Score: 0.9809523809523809
Task Scores:
                           Task     Score
0  high_school_computer_science  0.990000
1                     astronomy  0.967105
2        elementary_mathematics  0.984127

Intel/Qwen3.5-27B-int4-AutoRound (no DFlash):

Overall Score: 0.9761904761904762
Task Scores:
                           Task     Score
0  high_school_computer_science  0.980000
1                     astronomy  0.960526
2        elementary_mathematics  0.981481

The first thing to note is that the scores are similar, so there is apparently no output quality degradation measured by this small set of tests. Temperature/noise can account for the score difference and so I don’t think there is any significance in which is higher. I’m a bit suspicious that all tests scored higher in the DFlash but would need to see more tests. The other thing is that the non-DFlash result includes one instance of the model spending hours producing an answer on test set 0 and this didn’t happen in the DFlash tests. Something to keep an eye on.

Summary: I’ll keep using DFlash!

Here’s the test method. Change your base_url and model_name. Change the batch size from 10 if your vllm has max_num_seqs set lower than that. It sets the number of parallel tests it will run, so your vllm has to be able to handle it.

mkdir deepeval

cd deepeval/

python3 -m venv venv

source venv/bin/activate

pip install deepeval langchain-openai pandas datasets

cat << 'EOF' > mmlu_eval.py
from typing import List
from langchain_openai import ChatOpenAI
from deepeval.models import DeepEvalBaseLLM
from deepeval.benchmarks import MMLU
from deepeval.benchmarks.mmlu.task import MMLUTask

# Step 1: Create a custom LLM class for your remote API
class RemoteQuantizedLLM(DeepEvalBaseLLM):
    def __init__(self, base_url: str, model_name: str, api_key: str):
        self.model = ChatOpenAI(
            model=model_name,
            base_url=base_url,
            openai_api_key=api_key
        )
    
    def load_model(self):
        return self.model
    
    def generate(self, prompt: str) -> str:
        """Generate a single prediction"""
        chat_model = self.load_model()
        response = chat_model.invoke(prompt)
        return response.content
    
    async def a_generate(self, prompt: str) -> str:
        """Async version for faster evaluation"""
        chat_model = self.load_model()
        response = await chat_model.ainvoke(prompt)
        return response.content
    
    def batch_generate(self, prompts: List[str]) -> List[str]:
        """Optional: batch generation for speed"""
        chat_model = self.load_model()
        responses = chat_model.batch(prompts)
        return [r.content for r in responses]
    
    def get_model_name(self) -> str:
        return "Remote Quantized LLM"

# Step 2: Instantiate your model
remote_llm = RemoteQuantizedLLM(
    base_url="http://spark:8000/v1",
    model_name="Intel/Qwen3.5-27B-int4-AutoRound",
    api_key=""
)

# Step 3: Define MMLU benchmark with specific tasks
benchmark = MMLU(
    tasks=[
        MMLUTask.HIGH_SCHOOL_COMPUTER_SCIENCE,
        MMLUTask.ASTRONOMY,
        MMLUTask.ELEMENTARY_MATHEMATICS
    ],
    n_shots=5  # Few-shot learning (default is 5, max is 5)
)

# Step 4: Evaluate
results = benchmark.evaluate(
    model=remote_llm,
    batch_size=10  # Speeds up evaluation if batch_generate is implemented
)

# Step 5: View results
print(f"Overall Score: {benchmark.overall_score}")
print(f"Task Scores:\n{benchmark.task_scores}")
print(f"Prediction Details:\n{benchmark.predictions}")
EOF

python mmlu_eval.py
10

I was able to get the FP8 model to 18-20 tok/s using DFlash drafting with coherent output using the --no-enable-prefix-caching flag. Still getting fairly frequent spiraling/incorrect output on int4.

A comment on HF suggests that acceptance rates go way up with --no-enable-prefix-caching flag. Nobody knows why right now, but my new theory is that there is a SM12.x bug in Flash Attention. A bad one (causing incorrect output). Because virtually everyone on SM12.x has been using Flashinfer over Flash Attention, this could certainly be true.

I seem to have stable int4 output with a derivative of this build pinned to vllm=0.19.0 (for the Qwen3.5-122B-A10B optimized build in the other thread), but DFlash was not yet merged into vLLM as of 0.19.0. So something seems to have gone wrong between 0.19.0 and current head.

Until we have a root cause, the difference in our experience may be simply that my container has the bug and yours does not. I have verified I’m otherwise using the same startup config and the underlying models are correct and up to date.

Anyway, try --no-enable-prefix-caching - obviously not ideal for general and especially agentic use, but just to benchmark. I’m trying to rebuild my vLLM container with current HEAD presently.

11

You might want to have a look at this as well, apparently the weights of the smaller qwen models are incorrect. https://old.reddit.com/r/LocalLLaMA/comments/1sfwauj/qwen3535ba3buncensoredfernfloweraigguf

12

My setup is only slightly behind latest code:

commit 7bc4e4ce5e75020a26957a9a29e0edcb2fcccfd0 (HEAD -> main, origin/main, origin/HEAD)
Author: Eugene Rakhmatulin <eugr@eugr.com>
Date:   Sat Apr 4 10:46:06 2026 -0700

    Fixes #158 by adding  build args to gemma4 recipe

Using the prebuilt wheels, one behind latest:

wheels/flashinfer_cubin-0.6.7-py3-none-any.whl
wheels/flashinfer_jit_cache-0.6.7-cp39-abi3-manylinux_2_28_aarch64.whl
wheels/flashinfer_python-0.6.7-py3-none-any.whl
wheels/vllm-0.19.1rc1.dev46+gc5e3454e5.d20260406.cu132-cp312-cp312-linux_aarch64.whl

I tried that prefix caching change already and didn’t notice anything besides the expected increase in pp time. I just ran the albond quick bench again:

–enable-prefix-caching:

── Run 1/2 ──────────────────────────────────────
  [Q&A] 256 tokens in 5.23s = 48.9 tok/s (prompt: 23)
  [Code] 507 tokens in 7.86s = 64.5 tok/s (prompt: 30)
  [JSON] 1024 tokens in 21.85s = 46.8 tok/s (prompt: 48)
  [Math] 64 tokens in 1.56s = 41.0 tok/s (prompt: 29)
  [LongCode] 2048 tokens in 29.35s = 69.7 tok/s (prompt: 37)

── Run 2/2 ──────────────────────────────────────
  [Q&A] 256 tokens in 5.16s = 49.6 tok/s (prompt: 23)
  [Code] 507 tokens in 7.87s = 64.4 tok/s (prompt: 30)
  [JSON] 1024 tokens in 21.78s = 47.0 tok/s (prompt: 48)
  [Math] 64 tokens in 1.56s = 41.0 tok/s (prompt: 29)
  [LongCode] 2048 tokens in 29.36s = 69.7 tok/s (prompt: 37)

–no-enable-prefix-caching:

── Run 1/2 ──────────────────────────────────────
  [Q&A] 256 tokens in 5.57s = 45.9 tok/s (prompt: 23)
  [Code] 512 tokens in 7.72s = 66.3 tok/s (prompt: 30)
  [JSON] 1024 tokens in 21.77s = 47.0 tok/s (prompt: 48)
  [Math] 64 tokens in 1.52s = 42.1 tok/s (prompt: 29)
  [LongCode] 2048 tokens in 33.19s = 61.7 tok/s (prompt: 37)

── Run 2/2 ──────────────────────────────────────
  [Q&A] 256 tokens in 5.59s = 45.7 tok/s (prompt: 23)
  [Code] 512 tokens in 7.69s = 66.5 tok/s (prompt: 30)
  [JSON] 1024 tokens in 21.34s = 47.9 tok/s (prompt: 48)
  [Math] 64 tokens in 1.52s = 42.1 tok/s (prompt: 29)
  [LongCode] 2048 tokens in 33.21s = 61.6 tok/s (prompt: 37)

I’m using “num_speculative_tokens”: 12 now in the startup, seems slightly better performance.

–tool-call-parser qwen3_xml in the recipe

I also changed mods/fix-qwen3.5-chat-template/chat_template.jinja to fix a cache invalidation bug. Apparently the template changes historic output that didn’t contain reasoning and it adds empty think tags which invalidates the cache. See https://huggingface.co/Qwen/Qwen3.5-122B-A10B/discussions/22

13

One of the replies there says: I don’t have enough system resources to process 27B one in safetensors format.

So not confirmed to affect original weights of 27B. They only checked unsloth gguf by the looks of it.

14

@norman.2 ,@PlumeM I found this repository which seems like it may be trying to tackle the same issue as mentioned in that reddit thread regarding the botched Qwen3.5 weights on some of the smaller variants GitHub - decibuild/qwen-ssm-repair: Fix silent context collapse & looping in Qwen 3.5. Statistical repair for drifted SSM tensors caused by AdamW + sparse MoE routing. · GitHub

I can’t vouch for it working, but I found it by looking around for the particular layer names mentioned in that reddit thread, and it’s a very recently created repo.

15

Good find. Easy enough to check; grab the weights from the model in the Reddit post and do a dry run with this tool. Since the post states which specific output was correct, and I think does at least one more in a comment, those would validate the repo.

Then it just needs generalization to work on safetensors.

16

I finally got a vllm optimized container rebuilt compatible with this, and acceptance rates are back up. I can confirm the speed on this synthetic benchmark is incredible with Qwen3.5-27B-int4-AutoRound, similar to yours:

── Run 2/2 ──────────────────────────────────────
  [Q&A] 256 tokens in 5.24s = 48.8 tok/s (prompt: 23)
  [Code] 507 tokens in 7.19s = 70.5 tok/s (prompt: 30)
  [JSON] 1024 tokens in 21.11s = 48.5 tok/s (prompt: 48)
  [Math] 64 tokens in 1.45s = 44.1 tok/s (prompt: 29)
  [LongCode] 2048 tokens in 28.74s = 71.2 tok/s (prompt: 37)

And it seems far more stable to the spiraling output problems I saw previously. Thus, I do think there must have been a bug in my vLLM container before.

For real-world tasks in my professional domain I think I’ll turn it down to 8 instead of 15, because for complex specialized work I see 15-20 tok/s rather than 40+. Nonetheless, it’s a big improvement…

17

Just a Question, why do you run this versus Albonds Qwen 3.5 122B Int 4 (runs also at 50tok/s which is “competetive” i would say? Do you need more speed? In that case why not use the sparse qwen 35B model? If you need Quality wouldn’t the 122B be better? What was your experience with these and for which Use-Case? Thanks

18

I do run 122B-A10B quite a bit, with that specific harness.

I also run 35B-A3B quite a bit, especially for VLM work and with tasks dispatched in parallel.

However, there are tasks for which a dense model just seems to do better. More constantly active (not routed) parameters helps with complex document analysis, coherent understanding of an entire codebase, and similar.

On benchmarks, 27B looks is nearly equivalent to 122B-A10B. But in practice, when running them both on real-world data I cannot share publicly, I find the 122B reliable but the 27B will usually come up with a few subtle points the 122B does not. This category is also highly relevant for Gemma4 where the 31B definitely outperforms the 26B-A4B, and Google apparently decided not to release the ~120B MoE at the last minute so we don’t have a 31B MoE competitor.

Thus, I find it valuable to optimize for performance in this space on GB10.

19

Thanks for the detailed answer.

20

Seems like this pr is the key to fix: [Spec Decode] Add Sliding Window Attention support to DFlash drafter by jianc99 · Pull Request #40898 · vllm-project/vllm