Qwen3.5-122B-A10B on single Spark: up to 51 tok/s (v2.1 — patches + quick-start + benchmark)

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
, , , ,
62

could somebody help, please.
I have started the model with vllm.
for testing issue I have started “open webUI”
asking the model simple question: “tell me a joke”
it starts in thinking mode prepare the answer. good thinking and finaly when he finished thinkink it writes:

Let’s try this one:
“Why don’t scientists trust atoms?”
“Because they make up everything!”

so it has 2 almost the same answers and after that it writes (it’s quote below). and it start to generate flush content without any sence.
help please, how to configure vllm to work proper with the model?

That’s short, clean, and widely understood as funny by many people (dad jokes). Alternatively, I could offer something slightly longer narrative-wise for more engagement potentiality but shortness is often better for quick requests like this in chat interfaces… Actually no… wait let me think of another option that might be slightly more engaging than just one-liner but not too long either… Wait actually shorter is fine here but maybe provide two options so they have choice! No keep it simple - pick ONE good one first - then add an optional second if needed - But instructions say ‘tell me’ singular so best give ONE solid punchline first! Okay let’s stick with classic structure setup-punchline format clearly separated visually! Hmm perhaps adding emojis would make it friendlier too yes good idea! Okay final decision will be science-themed because universal appeal plus easy to understand even non-scientists get meaning behind phrase ‘make up everything’ works both literally figuratively clever wordplay perfect choice right now time to formulate response carefully ensuring tone remains light fun conversational style suitable general audience without any risk offending anyone based on cultural gender religious political sensitivities etc all clear check complete proceeding generation stage now ready outputting final answer below according guidelines provided earlier regarding safety policies which are fully respected here since topic entirely harmless innocent humor request only nothing violating rules whatsoever confirmed again before sending off message user waiting patiently expect prompt reply soon thank you very much indeed appreciate opportunity assist today happy joking around together wonderful day ahead surely going great stay safe out there always remember laugh daily keeps soul healthy wise saying goes somewhere true enough anyway back main task delivering requested content efficiently effectively satisfying query completely done thinking process concluding now generating response accordingly following established patterns successful interactions previously observed maintained consistency throughout conversation history preserving quality standards expected by platform users everywhere globally accessing service via various devices networks languages dialects none matter when core intent simply seeking amusement joy laughter shared human experience connecting through common

63

What is model and vLLM version? For quick start I suggest try GitHub - eugr/spark-vllm-docker: Docker configuration for running VLLM on dual DGX Sparks · GitHub

64

thank you for your previous recommendation 2 days ago. I am using spark-vllm-docker. - its great! thank you very much.
I am in progress with your hybrid model. But I wanted to make some tests for you and everybody in this thread with Sehyo\Qwen3.5-122B-A10B-NVFP4 model. (becouse this model is with MTP too)
and I have got the trouble I wrote in previous message.
My start script for spark-vllm-docker is

./launch-cluster.sh --solo exec \
  vllm serve \
    Sehyo/Qwen3.5-122B-A10B-NVFP4 \
    --port 8000 --host 0.0.0.0 \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.90 \
    --reasoning-parser qwen3 \
    --attention-backend FLASHINFER \
    --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \
    --kv-cache-dtype fp8
65
  1. NVFP4 on SM121 needs Marlin env vars
    -e VLLM_USE_FLASHINFER_MOE_FP4=0
    -e VLLM_NVFP4_GEMM_BACKEND=marlin
    -e VLLM_TEST_FORCE_FP8_MARLIN=1
  2. start minimal and add features one by one:
    ./launch-cluster.sh --solo exec
    vllm serve
    Sehyo/Qwen3.5-122B-A10B-NVFP4
    –port 8000 --host 0.0.0.0
    –tensor-parallel-size 1
    –gpu-memory-utilization 0.90
    –reasoning-parser qwen3
    –attention-backend FLASHINFER

If that works correctly, then try adding --speculative-config for MTP.

Note: In my testing, NVFP4 on SM121 only gets ~16.6 tok/s vs 28.3 tok/s for Intel AutoRound INT4 (Intel/Qwen3.5-122B-A10B-int4-AutoRound). The INT4 model is 70% faster because Marlin INT4 kernels are more efficient than the NVFP4 fallback path. MTP works perfectly with the INT4 model — I get 36.5 tok/s with MTP-1.

I think, main cause of garbage text, “–kv-cache-dtype fp8”, Remove --kv-cache-dtype fp8.

66

Wow thanks for all of this, this is so cool. I just bought my DGX Spark today, delivery in 3 days. Can’t wait to try this out.

Just a few questions @Albond and sorry for them already in advanced I am pretty new to local llms.

  • You said you are working on a 50 tok/s version with 256k context, how is this even possible given the memory bandwidth limits and when do you expect it to come. What is the physical ceiling for tok/s with this model size on this hardware?
  • Does the Multi-Token-Prediction make the Model quality worse? like how much percentage points worse output can we roughly expect from MTP and this speedup from high 20s to 38tok/s. (Because you said Q4KM is 4-5% worse than this quant, which is of course a big deal for agentic use-cases where any % boost in reliability is well sought after.
  • Can all of this also be done for Nemotron Super? Or does the current Architecture of the Nemotron model not support the features here?
  • Lastly if Nvidia makes NVFP4 support stable and good for the DGX Spark, what do you think we would then get in terms of decode and prefill for 120B models?

Sorry for bothering you, and I understand if you can’t answer all/any of them. If you do though, thank you :)

67

OK probably a stupid question. I use Intel AutoRound for the qwen3.5 122B model. I just want to do the MTP part instead of the hybrid part. If I am understanding correctly, I just need to use patch-02 on the safe tensor file? But the add-mtp-weights.py seems to point to a /path/to/hybrid-checkpoint so I have to have a hybrid step first?

I did try directly parsing the command of MTP to eugr’s receipe. and I got this error
vllm serve: error: argument --speculative-config: Value ‘{“method”:“qwen3_next_mtp”,“num_speculative_tokens”: cannot be converted to <function loads at 0xfac6c3134fe0>.

68

You’ve got curly quotes in that flag. Try again replacing them all with standard quotes.

69

I see. yeah applied this
–speculative-config ‘{“method”:“mtp”,“num_speculative_tokens”:1}’

And now it works just fine without any need to patch it seems

Decoding speed definitely feels snappier compared to before

70

  1. The theoretical limit is around 75 tok/s for Qwen3 235B A22B. However, this assumes no other activity — such as streaming tokens to the user. In practice, I’d put the realistic ceiling at ~50 tok/s for 32k context, and I’ll share a detailed breakdown soon. For 256k context, expect around ~40 tok/s and less when context grow … unfortunately. So just run with 256k context ~50 tok/s, but more longe work with Qwen3 235B A22B requred overhead for each next tocken.

  2. No — MTP generally improves or maintains model quality, not the opposite. Quality drops are only a risk with poor implementation. MTP is a training-time change: the model learns to predict multiple tokens ahead, which provides a richer gradient signal — quality doesn’t degrade. Q4KM is a different story — it’s weight quantization, where you’re literally discarding information (32-bit → 4-bit). That loss is irreversible, and the ~4-5% benchmark degradation is a real and tangible cost.

  3. Nemotron Super didn’t work well for my use cases, so after reviewing the current LLM landscape I’m focusing on Qwen3 235B A22B.

  4. Stable NVFP4 support would add roughly +2–3 tok/s. The key bottleneck here is memory bandwidth, not compute — FP4 helps with model size and compute efficiency, but the LPDDR5X bandwidth wall remains the hard ceiling.

71

Qwen3.5-122B-A10B on single Spark: 38.4 → 51 tok/s (v2 update + TurboQuant)

Quick update on the original post. The optimizations kept going after v1, and we broke through the “memory bandwidth ceiling” I complained about last time. Turns out it wasn’t the ceiling — it was the LM Head.

What changed: 38.4 → 51 tok/s (+33%)

Step tok/s Gain What
v1 (Hybrid INT4+FP8 + MTP-1) 38.4 — Previous post
+ INT8 LM Head v2 44 +15% Shared-weight Triton GEMV
+ MTP-2 51 +33% num_speculative_tokens:2

Two new optimizations, both surprisingly simple in hindsight.

INT8 LM Head v2 — The LM Head (248K × 3072 = 729 MB in BF16) was eating 24% of every decode step. The trick: quantize it to INT8 at runtime (per-channel, no calibration needed) and use a custom Triton GEMV kernel that reads the weight matrix once per batch instead of once per token. Goes from 24% to 84% bandwidth utilization on that single operation. No quality impact — INT8 per-channel preserves top-k rankings for the output layer.

MTP-2 — In the original post I said “Qwen3.5 only ships with 1 MTP layer” and assumed MTP-2 wouldn’t work. I was wrong. num_speculative_tokens:2 works fine — position 2 gets ~80% acceptance rate on 122B. Free +10% just by changing a number in the launch config. (On the smaller 35B model MTP-2 gives 0% — it’s already bandwidth-saturated — but 122B benefits from the larger batch.)

256K context works (without tricks)

A pleasant surprise: our optimizations freed enough memory that 256K context fits without any KV cache compression. The model gets 355K tokens of KV cache at --gpu-memory-utilization 0.90, which comfortably holds one 256K request.

I was wrong about the memory ceiling too, apparently.

A note on context length vs speed: The 51 tok/s number is for short prompts (~30-50 tokens). As context fills up, attention becomes more expensive — each new token attends to all previous tokens. At 32K filled context, expect ~48 tok/s. At 128K+, noticeably slower. This isn’t a bug, it’s how attention works. The 256K capacity means the model can handle it, not that it’ll be fast doing it.

Optional: TurboQuant KV Cache (4x throughput, -22% speed)

For those who need to serve multiple users at long context, we also added TurboQuant (Google, ICLR 2026) KV cache compression. It compresses the KV cache from bf16 to ~3.5 bits per element — 4.27x compression.

Config tok/s KV Cache Concurrent users @ 256K
v2 (standard) 51 355K tokens 1
v2 + TurboQuant 39 (-22%) 1.4M tokens 5

The -22% speed penalty is real and unavoidable. TQ replaces FlashInfer with a Triton attention backend (FlashInfer doesn’t support custom KV formats), and uses PIECEWISE CUDA graphs instead of FULL. We spent a week writing a custom CUDA fused kernel that computes attention directly on packed TQ data — went through 8 iterations, achieved 238x speedup from the naive version — and it’s still 1.4x slower than FlashInfer. The fundamental issue: bit unpacking + codebook lookup costs ~1.5x more ALU ops per KV position than a simple bf16 dot product. With 4.3x less data but 1.5x more compute per byte, there’s no free lunch.

TQ is useful if you need throughput (5 concurrent 256K users vs 1). For single-user use, standard v2 at 51 tok/s is better.

Important: Qwen3.5 was tested by its developers on contexts up to 256K only. The 1.4M token cache could theoretically handle ~1M context, but model quality beyond 256K is unverified. Also, the Intel AutoRound INT4 quantization was calibrated for standard context lengths — deeper contexts would need a fresh calibration run.

What else we tried and killed

After v1, we ran 20 experiments. The INT8 LM Head and MTP-2 were the only survivors. Here’s what else died:

  • EAGLE-3 speculative: +10%, but loses to MTP-2 (needs separate draft weights, MTP head is built-in and faster)
  • INT8 Shared Expert: GIBBERISH output. FP8→INT8 re-quantization destroys the calibrated scales. Don’t do this.
  • SERE expert re-routing: Expert similarity ~0.02 with 256 experts. They’re all unique. SERE needs coarser MoE.
  • AdaptiveSoftmax LM Head: Correct math, terrible memory access pattern (2.5% BW utilization). Full INT8 GEMV wins.
  • 2:4 Structured Sparsity: FFN=512 is too small. No intra-expert redundancy.
  • KnapSpec self-speculative: Draft reads 75% of weights vs MTP head 4.4%. Wrong optimization target for BW-bound.

Updated repo

Everything is in the repo, updated to v2:

GitHub - albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4: Qwen3.5-122B-A10B on DGX Spark: 28.3 → 38.4 tok/s (+36%) · GitHub

New in v2:

  • Dockerfile.v2 — all-in-one from base vLLM (hybrid + INT8 LM Head)
  • Dockerfile.v2-tq — optional TurboQuant variant
  • Step-by-step README with explicit vLLM build instructions
  • Version pinning (vLLM 0.19.1, exact commit 2a69949bd)
  • TurboQuant adapted for SM121 with 8 SM121-specific fixes

The hybrid steps (step 1) are optional — if you just want MTP-2 + INT8 LM Head without building a hybrid checkpoint, skip step 1 and use the Intel AutoRound checkpoint directly. The Docker image works with both.

Tested environment

Component Version
vLLM 0.19.1.dev0+g2a69949bd
PyTorch 2.12.0.dev20260403+cu130
FlashInfer 0.6.7
CUDA 13.2

Patches are version-specific. vLLM changes its internals frequently — if you update vLLM, the patches will likely need adjusting (as eugr already discovered with the community Docker).

Huge thanks to everyone in this thread and the broader Spark community — the feedback on v1 directly led to several of these improvements. Special thanks to eugr for adapting the hybrid patch for spark-vllm-docker, to the TurboQuant contributors (mitkox, bjk110, 0xSero), and to everyone who reported issues and tested the patches. Norman’s confirmation of similar numbers and AoE’s 56 tok/s on 2x Spark via Ray were particularly motivating.

51 tok/s from a box that sits under my desk. Not bad for “consumer Blackwell.”

72

looking forward to test it in the morning

73

Who’s TQ implementation did you use? @flash3 has done some fairly substantial testing with various implementations.

74

Great work! Looking forward to applying your changes and getting a free speedup as Qwen3.5 122B int4 has been my daily drjver for a while now.

I see in the repo you say the same patches also help Qwen3.5 35B achieve higher tps. Maybe a stupid question, but would Qwen 3.5 27B also be sped up somewhat by some of the patches even though it is a dense model?

75

Own TQ. Google describe all needed details and I optimize it on GPU DGX Spark. Reference to mitkox and bjk110 for some save time. I’ve described details in “Optional: TurboQuant KV Cache Compression” → “How it works” in README in project. I will review flash3 and retest and if find some new for TQ, will update project.

76

Qwen3.5-27B is a Dense model, which presents the same performance limitations as Gemma-4-31B with unified memory bw on the DGX Spark. For Dense models in general, the only way to achieve truly high throughput is to build custom TPU infrastructure — similar to what Google does. Even with optimizations, you’re looking at around 10-15 tokens/second maximum, which is a hard ceiling.
In that context, a 122B model actually offers a better intelligence-to-performance ratio at that speed. So practically speaking, MoE (Mixture of Experts) models are the only ones that make real sense to run on DGX Spark — unless 10–15 tok/s is an acceptable speed for your use case.

77

Many thanks for the reply. I am aware that the dense models are quite a bit slower, was hoping/wondering whether some of the optimizations to unlock faster inference in 122B MoE will also be applicable for the dense 27B, especially the MTP.

As I said I have been happily using Qwen3.5 122B as my daily driver because of its good balance of speed (even before your changes) and intelligence for my needs. The attractive part of 27B is the lower memory footprint, potentially freeing up memory for other uses on the Spark, at the cost of slower inference. I have never used this dense model before but from what I read its performance is very close to 122B, so if it was faster than today (but still materially slower than optimised 122B) it would have less drawbacks which in certain circumstances would make it an attractive model to use.

78

I want to try MoEification next time for Dense as example Gemma-4-31B-it → ~120B 10A. Not sure about quality and all, but maybe this will be only one way to run it on GPU.

79

Following along at home…

Step 1 when I run

python patches/01-hybrid-int4-fp8/build-hybrid-checkpoint.py

I get

DGX_Spark_Qwen3.5-122B-A10B-AR-INT4/patches/01-hybrid-int4-fp8/build-hybrid-checkpoint.py", line 140, in validate_gptq_input
    raise FileNotFoundError(f"GPTQ directory does not exist: {gptq_dir}")

I have .cache/huggingface/hub/models--Intel--Qwen3.5-122B-A10B-int4-AutoRound downloaded and running already. Where do I get the GPTQ directory from?

Thanks for all this – very keen to give it a try.

Solved - I have two snapshots in that folder

.cache/huggingface/hub/models--Intel--Qwen3.5-122B
-A10B-int4-AutoRound/snapshots
total 16
drwxr-xr-x 4 root root 4096 Mar 16 02:53 .
drwxr-xr-x 6 root root 4096 Mar 16 03:14 ..
drwxr-xr-x 2 root root 4096 Mar  3 09:16 bfac534d4d8742dd15e46f7efdf73336b0213970
drwxr-xr-x 2 root root 4096 Mar 16 03:14 de5e04f218487cb2fb762c6e9aabec784fb40d00

INTEL_DIR=$(find ~/.cache/huggingface/hub/models–Intel–Qwen3.5-122B-A10B-int4-AutoRound/snapshots -maxdepth 1 -mindepth 1 -type d)

was returning an array, deleted the older one, now Step 1 is working…

Thanks!

80

The --gptq-dir flag is just historically named — it also accepts the Intel AutoRound INT4 directory (that’s exactly what the README uses). You need to point it at the snapshot directory inside the HF cache, not the top-level models–Intel–… folder (that one only has blobs/, refs/, snapshots/).

From the README (Step 1):

huggingface-cli download Intel/Qwen3.5-122B-A10B-int4-AutoRound
INTEL_DIR=$(find ~/.cache/huggingface/hub/models–Intel–Qwen3.5-122B-A10B-int4-AutoRound/snapshots -maxdepth 1 -mindepth 1 -type d)

python patches/01-hybrid-int4-fp8/build-hybrid-checkpoint.py
–gptq-dir “$INTEL_DIR”
–fp8-repo Qwen/Qwen3.5-122B-A10B-FP8
–output ~/models/qwen35-122b-hybrid-int4fp8
–force

You can also just echo $INTEL_DIR first to confirm it resolves to something like …/snapshots// containing config.json and model--of-.safetensors. That’s the path the script wants.

The 408 “unmatched FP8 tensor” warnings during the build are expected — that’s why --force is in the example.

81

One last question @Albond what Temperature and other settings would you recommend for Agentic(Coding) workflows for example with Opencode or others. I am asking becuase the “Launch”, doesnt include it:

docker run -d --name vllm-qwen35
–gpus all --net=host --ipc=host
-v ~/models:/models
vllm-qwen35-v2
serve /models/qwen35-122b-hybrid-int4fp8
–served-model-name qwen
–port 8000
–max-model-len 262144
–gpu-memory-utilization 0.90
–reasoning-parser qwen3
–attention-backend FLASHINFER
–speculative-config ‘{“method”:“mtp”,“num_speculative_tokens”:2}’

Also is there anyhting else your recommend to use for highest quality outputs for that usecase?

Thank you so much, your work here is amazing.