Custom built vLLM + Qwen3.5-35B on NVIDIA DGX Spark (GB10) — sustained 50 tok/s, 1M context

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

vLLM + Qwen3.5-35B on NVIDIA DGX Spark (GB10) — 50 tok/s, 1M context

Qwen3.5-35B-A3B-FP8 running on the DGX Spark’s GB10 (SM121 Blackwell) with vLLM. Sharing the setup and a pre-built Docker image specifically built and tuned for this model on the GB10. This container with this model has over 60 hours of testing, usage and runs almost 24/7 in my lab, so you can pull and go with confidence that it works reliably. There is a community docker version by @eug’s and others as well :)

Results:

  • 50 tok/s decode throughput (2400 MHz GPU clock)

  • 1M token context via runtime YaRN override (--hf-overrides)

  • 34.6 GiB model footprint — leaves ~74 GiB for FP8 KV cache

  • Model loads in ~34s with fastsafetensors

Stack:

  • vLLM 0.16.0-dev with SM121 patches (rebased on upstream main)

  • PyTorch 2.12.0 nightly (cu130, ARM64)

  • Triton 3.6.0 (built from source for SM121)

  • FlashInfer attention + TRITON FP8 MoE with GB10-tuned kernel configs (E=256, N=512, block 128x128)

  • FP8 KV cache + chunked prefill + prefix caching

1M context note: Qwen3.5 ships with max_position_embeddings: 262144. For 1M you need to apply YaRN at runtime since only 25% of layers use softmax attention (the rest is DeltaNet linear). Set VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 and pass:


--hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}'

Docker image (ARM64/GB10 only):


docker pull hellohal2064/vllm-qwen3.5-gb10:latest

Includes GB10-tuned MoE configs, reasoning/tool-call parsers, and FlashInfer JIT cache support. Model not included — download Qwen3.5-35B-A3B-FP8 separately.

Full README with docker run command and API examples on Docker Hub.

Source: github.com/seli-equinix/vllm (branch: feature/sm121-gb10-support)

3

Thanks. It is really slow with a big context, but better than nothing.

model test t/s peak t/s ttfr (ms) est_ppt (ms) e2e_ttft (ms)
Qwen3.5-35B-A3B-FP8 pp2048 5019.02 ± 31.22 373.22 ± 6.43 371.49 ± 6.43 373.29 ± 6.43
Qwen3.5-35B-A3B-FP8 tg128 49.03 ± 0.05 50.00 ± 0.00
Qwen3.5-35B-A3B-FP8 pp2048 @ d100000 3451.73 ± 9.35 26790.89 ± 36.05 26789.15 ± 36.05 26790.95 ± 36.04
Qwen3.5-35B-A3B-FP8 tg128 @ d100000 41.38 ± 0.17 42.33 ± 0.47
Qwen3.5-35B-A3B-FP8 pp2048 @ d200000 2306.70 ± 5.22 79454.33 ± 141.73 79452.60 ± 141.73 79454.39 ± 141.71
Qwen3.5-35B-A3B-FP8 tg128 @ d200000 35.55 ± 0.21 37.00 ± 0.00
Qwen3.5-35B-A3B-FP8 pp2048 @ d500000 1157.95 ± 0.45 392972.17 ± 138.08 392970.43 ± 138.08 392972.24 ± 138.08
Qwen3.5-35B-A3B-FP8 tg128 @ d500000 24.98 ± 0.07 26.67 ± 0.47
4

@pontostroy Thanks for the feedback :) 49.03 ± 0.05 t/s on a Spark with a large context window is really solid — that’s pushing close to the theoretical limit for this hardware. The GB10 has 273 GB/s memory bandwidth, and since Qwen3.5-35B-A3B only activates ~3B parameters per token (MoE), the theoretical ceiling is around 91 tok/s. Hitting ~49 t/s means we’re at roughly 54% memory bandwidth utilization, which is excellent once you factor in YaRN overhead, KV cache management, and attention computation at extended context depths.

Worth noting the variance too — ±0.05 t/s at baseline and ±0.07 at 500K context means the performance is extremely consistent run-to-run. That kind of stability matters for production workloads where you need predictable latency, not just high throughput. I have done a lot of work to make it very stable.

25 tok/s at 500K context is also really encouraging — for perspective, that’s still faster than reading speed (~4 words/sec), so it remains fully interactive even at half a million tokens of context. The whole point of this build is to show what’s achievable on a 140W desktop device. Obviously nothing like data center hardware (H100 has ~12x the memory bandwidth at 3.35 TB/s), but for local inference on a $3K desktop it’s a different ballgame entirely.

Hoping this helps with whatever AI project you’re working on! :)

5

FYI. My results on FE Spark

| model                    |             test |            t/s |     peak t/s |          ttfr (ms) |       est_ppt (ms) |      e2e_ttft (ms) |
|:-------------------------|-----------------:|---------------:|-------------:|-------------------:|-------------------:|-------------------:|
| Qwen/Qwen3.5-35B-A3B-FP8 |            tg128 |   51.37 ± 0.34 | 52.00 ± 0.00 |                    |                    |                    |
| Qwen/Qwen3.5-35B-A3B-FP8 | pp2048 @ d100000 | 3466.09 ± 6.96 |              |   31683.64 ± 59.51 |   29442.30 ± 59.51 |   31683.72 ± 59.50 |
| Qwen/Qwen3.5-35B-A3B-FP8 |  tg128 @ d100000 |   41.09 ± 0.15 | 42.67 ± 0.47 |                    |                    |                    |
| Qwen/Qwen3.5-35B-A3B-FP8 | pp2048 @ d200000 | 2190.60 ± 1.16 |              |   94475.85 ± 48.91 |   92234.50 ± 48.91 |   94475.94 ± 48.90 |
| Qwen/Qwen3.5-35B-A3B-FP8 |  tg128 @ d200000 |   35.25 ± 0.20 | 36.67 ± 0.47 |                    |                    |                    |
| Qwen/Qwen3.5-35B-A3B-FP8 | pp2048 @ d500000 | 1067.53 ± 0.43 |              | 472531.29 ± 189.19 | 470289.95 ± 189.19 | 472531.39 ± 189.20 |
| Qwen/Qwen3.5-35B-A3B-FP8 |  tg128 @ d500000 |   23.93 ± 0.06 | 25.67 ± 0.47 |                    |                    |                    |
6

@kosta That is about what I am getting on my ASUS spark. I really hope the container works well for you. I have been using it for about a week and it is really stable. The biggest thing is the ± they are very low and very consistent. if you look at some of the other image it can be much higher. Trying to get this to a production level :)

7

This is impressive but it would be appreciated to contextualize (heh) the incremental gains against eugr’s community vllm-node with the same flags and overrides.

I like the idea, but not necessarily enough to justify keeping a totally separate dedicated image and ~25+ GB of space for it unless it really is a substantial jump in performance above ~128k. And even in that case, it would be ideal to get the fixes/improvements folded into the community build.

8

@joshua.dale.warner Thanks :) for the feedback. I post my images not to take away from what @eugr is doing — his setup is focused on supporting all the models that can run on the Spark, which is a different goal and really valuable work. My builds are what I’m using to build a production system for the company I work for. In this case, this container is specifically tuned and heavily tested for this one model to make it production ready.

Some of what I find along the way I try to push upstream to the main vLLM repo. But the reason I shared the container is simpler than that — I’ve already put in the hours validating that it works and is very stable running this model, so why not save someone else the time? Some folks would rather spend their time building on top of a working setup than getting the setup itself working — and I think that’s a great thing.

This container has over 60 hours of testing and runs almost 24/7 in my lab. I’ll probably add that to the card or this thread to help explain why I shared it :)

10

Using this version and it’s lightning fast! Thanks for this! Please post if you do other versions like this. Like the new Qwen versions coming out or even ablitherated ones.

11

thanks for the feedback :)

12

This is my main chat model I’m using (Eugr’s vLLM) for Agent zero. I’m very interested in your settings will try them out!!

13

Let me revise. I’m currently hitting between 30-50 tokens/sec and 70% or so cache hits. Should I bother messing with what I have set up?

14

Sigh…I downloaded the docker image. I have the model already. But I can’t find directions on the docker switches to fire it up correctly. I told ya I’m a newb bigtime. Any chance you would be kind enough to give me a full command to fire up the docker container you built with all the flags and tweaks to run this model the way you are for max performance, I really want to try it out for work. But I’m dumb :( I’m on github trying to figure it out but I’m failing :(.

Most appreciated and humbled if you do!

Cheers.

15

I don’t know what happened, the quick start on Docker Hub is no longer there, just some long code. I am looking at it now. will let you know as that should have all the instruction to use.

16

i have updated everything - including the docker image. docker hub should have all the correct info now.

don’t know what broke docker hub :(

17

let me know if you have any other challanges.

18

Thank you,hellohal2064, for sharing Docker. This container runs very quickly and is very stable. It’s really surprising. I’m curious about how you managed to do this. Could you share some of your experiences with me?For example:fastsafetensors and flashinfer and mtp and triton and torch.compile;

19

Hal, could you pleasE do a uncensored version of this?

20

Thanks for the questions and feedback — and apologies for the slow response!

You can find my main repository for all my vLLM work here: vLLM Fork. It includes most of what I’ve built and tested. I published this container as it’s what I personally use for my work, and I wanted to share a fast, reliable, and easy-to-set-up container. It’s optimized specifically for this model.

@eugr and the upstream maintainers focus on other models and experimental features — they do great work exploring new directions. My goal with this version was to provide something that anyone can use right away without needing to spend a lot of time learning or tuning their Spark setup. With this, you can just get started, run a few commands, and have your Spark running smoothly in no time.

I’m always happy to answer any questions!

21

How good is this a tool use?