I recently got my first Spark and love being able to run a bunch of bigger models like Qwen 3.5 122b - dense models sadly disappointed so far, though.
To reach higher quality/faster inference speed, I consider a second Spark to either run dense models with more t/s or at least run higher quality MoE models.
For those who made the upgrade: was it worth it for inferencing? I do some training but primarily use my Spark as inference server for a single model right now (Qwen 3.5 35b in FP8) via spark-vllm-docker.
Appreciate your responses – I tried digging through this forum but a lot of responses are either rather old or not conclusive.
I have an economical take about this: when you buy a single spark you’re basically wasting $1500
We’ve running new models every week there. Feel free to contribute.
After I got the first one I was fairly convinced I needed just a bit more speed and capacity than I got from a single unit so picked up #2 . It’s great to have capacity to run models like Minimax M2.5 with full context for coding. I feel like I get a lot more out of the system now, I don’t regret it.
Thanks! I’m very tempted! I fear I didn’t see any dual spark benchmarks for Qwen 3.5 27b yet. Would you be able to give it a pass if you have a moment? I can contribute the single model benchmark, even if the result is not that impressive.
If it wasn’t for @eugr spark-vllm-docker, I would’ve thrown my spark in the trash. Instead I bought a second one and I’m glad I did.
It’s true. You really need to buy at least two if you want the best value. One is good enough to run the newest diffusion models (like FLUX.2, for example, for which I wrote a custom fastsafetensors loader node), but a better use for these systems seems to be running bigger LLMs. I decided to buy a second one just for that. I knew about the memory bandwidth limitation, but I don’t really care that much. Just being able to run models like Qwen3.5-397B (in 4 bit) with acceptable performance makes it worth it. You really want that extra RAM for large models and the KV cache to maintain large context windows.
You will see a material increase, but it will still be relatively slow. Sparse MoE models like 3.5 122b run much better on this hardware.
AoE
March 9, 2026, 7:12pm
8
Having 2 allows me to run Qwen3.5-122B + Open WebUI + custom RAG (VL) + ComfyUI running all the time with good speed and good enough quality for a couple of concurrent users. It was not convenient to always have to switch model depending on the task at hand when we had only one.
That’e exactly where I am at
relc
March 10, 2026, 12:46pm
10
Also two here. They replicate quickly. I would say it depends on the budget. For research having a second one was a good choice.
I caved in and got a 2nd spark and that shiny qsfp cable. After some back and forth I got the clustering to work as expected and see the first pretty strong results – will play with a dense model tomorrow. Running with @eugr ’s amazing spark-vllm-docker and my own launch script.
Happy to hear people’s thought if there is any room for optimization on below vllm config:
vllm serve Intel/Qwen3.5-122B-A10B-int4-AutoRound \
--host 0.0.0.0 \
--port 8080 \
--gpu-memory-utilization 0.75 \
--load-format fastsafetensors \
--max-model-len 262144 \
--max-num-batched-tokens 16384 \
--enable-prefix-caching \
--enable-chunked-prefill \
--dtype auto \
--kv-cache-dtype auto \
--max-num-seqs 4 \
--trust-remote-code \
--chat-template unsloth.jinja \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--reasoning-parser qwen3 \
--tensor-parallel-size 2 \
--distributed-executor-backend ray
llama-benchy --base-url ``http://0.0.0.0:8080/v1`` --model Intel/Qwen3.5-122B-A10B-int4-AutoRound --latency-mode api --pp 2048 --depth 0 4096 8192 16384 --tg 32 128 --concurrency 4
| model | test | t/s (total) | t/s (req) | peak t/s | peak t/s (req) | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:---------------------------------------|---------------------:|----------------:|-----------------:|--------------:|-----------------:|-------------------:|-------------------:|-------------------:|
| Intel/Qwen3.5-122B-A10B-int4-AutoRound | pp2048 (c4) | 3844.66 ± 12.06 | 1480.32 ± 723.55 | | | 1652.67 ± 580.82 | 1651.01 ± 580.82 | 1652.69 ± 580.81 |
| Intel/Qwen3.5-122B-A10B-int4-AutoRound | tg32 (c4) | 58.57 ± 5.77 | 25.89 ± 9.03 | 126.33 ± 0.47 | 33.08 ± 1.76 | | | |
| Intel/Qwen3.5-122B-A10B-int4-AutoRound | pp2048 (c4) | 3859.73 ± 7.42 | 1376.87 ± 411.81 | | | 1624.22 ± 456.37 | 1622.56 ± 456.37 | 1624.25 ± 456.37 |
| Intel/Qwen3.5-122B-A10B-int4-AutoRound | tg128 (c4) | 100.31 ± 3.90 | 27.53 ± 2.63 | 137.33 ± 1.89 | 34.33 ± 0.47 | | | |
| Intel/Qwen3.5-122B-A10B-int4-AutoRound | pp2048 @ d4096 (c4) | 3957.67 ± 1.41 | 1070.41 ± 139.76 | | | 5828.54 ± 661.00 | 5826.88 ± 661.00 | 5828.57 ± 660.99 |
| Intel/Qwen3.5-122B-A10B-int4-AutoRound | tg32 @ d4096 (c4) | 50.03 ± 0.11 | 27.59 ± 8.61 | 127.00 ± 0.00 | 32.96 ± 1.13 | | | |
| Intel/Qwen3.5-122B-A10B-int4-AutoRound | pp2048 @ d4096 (c4) | 3962.32 ± 3.63 | 1078.39 ± 128.43 | | | 5773.79 ± 622.38 | 5772.12 ± 622.38 | 5773.81 ± 622.36 |
| Intel/Qwen3.5-122B-A10B-int4-AutoRound | tg128 @ d4096 (c4) | 88.80 ± 4.52 | 26.91 ± 3.06 | 133.33 ± 1.89 | 33.33 ± 0.47 | | | |
| Intel/Qwen3.5-122B-A10B-int4-AutoRound | pp2048 @ d8192 (c4) | 3913.24 ± 3.31 | 1205.56 ± 332.98 | | | 9028.30 ± 1932.91 | 9026.64 ± 1932.91 | 9028.32 ± 1932.89 |
| Intel/Qwen3.5-122B-A10B-int4-AutoRound | tg32 @ d8192 (c4) | 21.71 ± 0.03 | 20.57 ± 10.77 | 123.00 ± 0.00 | 30.75 ± 0.43 | | | |
| Intel/Qwen3.5-122B-A10B-int4-AutoRound | pp2048 @ d8192 (c4) | 3908.68 ± 1.08 | 1204.45 ± 332.89 | | | 9037.00 ± 1936.03 | 9035.34 ± 1936.03 | 9037.02 ± 1936.02 |
| Intel/Qwen3.5-122B-A10B-int4-AutoRound | tg128 @ d8192 (c4) | 55.24 ± 0.25 | 23.39 ± 5.89 | 124.00 ± 0.00 | 31.00 ± 0.00 | | | |
| Intel/Qwen3.5-122B-A10B-int4-AutoRound | pp2048 @ d16384 (c4) | 3743.54 ± 4.70 | 1141.74 ± 239.43 | | | 16779.10 ± 3039.37 | 16777.43 ± 3039.37 | 16779.12 ± 3039.35 |
| Intel/Qwen3.5-122B-A10B-int4-AutoRound | tg32 @ d16384 (c4) | 14.14 ± 0.01 | 14.57 ± 9.98 | 116.00 ± 0.00 | 29.00 ± 0.00 | | | |
| Intel/Qwen3.5-122B-A10B-int4-AutoRound | pp2048 @ d16384 (c4) | 3732.94 ± 0.97 | 1138.42 ± 238.62 | | | 16827.29 ± 3046.99 | 16825.62 ± 3046.99 | 16827.31 ± 3046.97 |
| Intel/Qwen3.5-122B-A10B-int4-AutoRound | tg128 @ d16384 (c4) | 40.82 ± 0.02 | 19.21 ± 6.57 | 116.00 ± 0.00 | 29.00 ± 0.00 | | | |
I really need to try the autoround version. Here’s what i’m getting with the FP8:
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:----------------------|----------------:|----------------:|-------------:|----------------:|----------------:|----------------:|
| Qwen3.5-122B-A10B-FP8 | pp2048 | 3629.67 ± 6.27 | | 592.85 ± 14.20 | 512.61 ± 14.20 | 592.98 ± 14.22 |
| Qwen3.5-122B-A10B-FP8 | tg32 | 32.12 ± 0.08 | 33.16 ± 0.08 | | | |
| Qwen3.5-122B-A10B-FP8 | pp2048 @ d4096 | 3692.04 ± 18.91 | | 1596.89 ± 39.22 | 1516.65 ± 39.22 | 1597.02 ± 39.22 |
| Qwen3.5-122B-A10B-FP8 | tg32 @ d4096 | 32.44 ± 0.32 | 33.49 ± 0.33 | | | |
| Qwen3.5-122B-A10B-FP8 | pp2048 @ d8192 | 3558.71 ± 7.49 | | 2669.43 ± 24.52 | 2589.19 ± 24.52 | 2669.53 ± 24.53 |
| Qwen3.5-122B-A10B-FP8 | tg32 @ d8192 | 32.32 ± 0.23 | 33.37 ± 0.24 | | | |
| Qwen3.5-122B-A10B-FP8 | pp2048 @ d16384 | 3675.96 ± 24.64 | | 4647.34 ± 18.49 | 4567.10 ± 18.49 | 4647.44 ± 18.48 |
| Qwen3.5-122B-A10B-FP8 | tg32 @ d16384 | 31.41 ± 0.39 | 32.43 ± 0.40 | | | |
| Qwen3.5-122B-A10B-FP8 | pp2048 @ d32768 | 3494.11 ± 17.31 | | 9120.80 ± 82.05 | 9040.56 ± 82.05 | 9120.91 ± 82.06 |
| Qwen3.5-122B-A10B-FP8 | tg32 @ d32768 | 30.87 ± 0.24 | 31.39 ± 0.55 | | | |
llama-benchy (0.3.3)
date: 2026-03-05 20:57:16 | latency mode: generation
Please share your launch configuration that helped you achieve these results.
~/spark-vllm-docker/launch-cluster.sh \
exec vllm serve \
"$MODEL" \
--served-model-name "$SERVED_NAME" \
--port 8000 --host 0.0.0.0 \
--distributed-executor-backend ray \
--gpu-memory-utilization 0.82 \
--load-format fastsafetensors \
--attention-backend flashinfer \
--tensor-parallel-size 2 \
--max-num-seqs 6 \
--trust-remote-code \
--enable-chunked-prefill \
--enable-prefix-caching \
--max-num-batched-tokens 8192 \
--max-model-len 65536 \
--enable-auto-tool-choice \
--mm-encoder-tp-mode data \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder
serapis:
tg128 @ d16384 (c4)
You should see 65-75 tokens/s at c4 in tg128 @16384
Could you try:
description: vLLM serving Qwen3.5-122B-INT4-Autoround
model: Intel/Qwen3.5-122B-A10B-int4-AutoRound
container: vllm-node-tf5
build_args:
- '--tf5'
mods:
- mods/fix-qwen3.5-autoround
defaults:
port: 8000
host: 0.0.0.0
tensor_parallel: 2
gpu_memory_utilization: 0.7
max_model_len: 131072
max_num_batched_tokens: 8192
env:
VLLM_MARLIN_USE_ATOMIC_ADD: '1'
command: |
vllm serve Intel/Qwen3.5-122B-A10B-int4-AutoRound \
--max-model-len {max_model_len} \
--gpu-memory-utilization {gpu_memory_utilization} \
--port {port} \
--host {host} \
--load-format fastsafetensors \
--enable-prefix-caching \
--enable-auto-tool-choice \
--tool-call-parser qwen3_xml \
--max-num-batched-tokens {max_num_batched_tokens} \
--trust-remote-code \
--distributed-executor-backend ray \
--reasoning-parser qwen3 \
-tp {tensor_parallel}
recipe_version: '1'
name: Qwen3.5-122B-INT4-Autoround
cluster_only: false
Like the one on spark-arena
This is what I use right now:
model: Intel/Qwen3.5-122B-A10B-int4-AutoRound
max_model_len: 262144
gpu_mem_util: 0.75
tp: 2
max_batch_tok: 8192
max_num_seqs: 4
kv_cache_dtype: auto
This is the benchmark I ran for you: llama-benchy --base-url http://0.0.0.0:8080/v1 --pp 2048 --tg 128 --depth 16384 --concurrency 4
Here is the result:
| model | test | t/s (total) | t/s (req) | peak t/s | peak t/s (req) | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:---------------------------------------|---------------------:|---------------:|-----------------:|--------------:|-----------------:|-------------------:|-------------------:|-------------------:|
| Intel/Qwen3.5-122B-A10B-int4-AutoRound | pp2048 @ d16384 (c4) | 3647.84 ± 9.05 | 1504.22 ± 634.91 | | | 14212.08 ± 4819.14 | 14210.32 ± 4819.14 | 14212.13 ± 4819.11 |
| Intel/Qwen3.5-122B-A10B-int4-AutoRound | tg128 @ d16384 (c4) | 28.34 ± 0.05 | 14.60 ± 7.03 | 108.00 ± 0.00 | 27.00 ± 0.00 | | | |
llama-benchy (0.3.5)
date: 2026-03-16 09:51:23 | latency mode: api
And closer to your recipe:
model: Intel/Qwen3.5-122B-A10B-int4-AutoRound
max_model_len: 131072
gpu_mem_util: 0.7
tp: 2
max_batch_tok: 8192
max_num_seqs: 4
kv_cache_dtype: auto
| model | test | t/s (total) | t/s (req) | peak t/s | peak t/s (req) | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:---------------------------------------|---------------------:|----------------:|-----------------:|--------------:|-----------------:|-------------------:|-------------------:|-------------------:|
| Intel/Qwen3.5-122B-A10B-int4-AutoRound | pp2048 @ d16384 (c4) | 3686.29 ± 18.74 | 1522.02 ± 642.50 | | | 14047.98 ± 4770.82 | 14046.57 ± 4770.82 | 14048.03 ± 4770.81 |
| Intel/Qwen3.5-122B-A10B-int4-AutoRound | tg128 @ d16384 (c4) | 27.94 ± 0.89 | 13.92 ± 6.50 | 106.67 ± 1.89 | 26.75 ± 0.43 | | | |
llama-benchy (0.3.5)
date: 2026-03-16 10:33:07 | latency mode: api
And this is literally running the recipe: ./run-recipe.sh qwen3.5-122b-int4-autoround
llama-benchy --base-url http://0.0.0.0:8000/v1 --pp 2048 --tg 128 --depth 16384 --concurrency 4
| model | test | t/s (total) | t/s (req) | peak t/s | peak t/s (req) | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:---------------------------------------|---------------------:|----------------:|-----------------:|--------------:|-----------------:|-------------------:|-------------------:|-------------------:|
| Intel/Qwen3.5-122B-A10B-int4-AutoRound | pp2048 @ d16384 (c4) | 3606.82 ± 40.90 | 1478.36 ± 620.99 | | | 14429.07 ± 4851.22 | 14427.52 ± 4851.22 | 14429.14 ± 4851.21 |
| Intel/Qwen3.5-122B-A10B-int4-AutoRound | tg128 @ d16384 (c4) | 28.31 ± 0.33 | 14.70 ± 7.13 | 108.00 ± 0.00 | 27.00 ± 0.00 | | | |
llama-benchy (0.3.5)
date: 2026-03-16 10:53:27 | latency mode: api
What modules are you running on the cluster? All I run is Qwen3.5 27b-int4autoround. I don’t feel122B A10B is better than 27B.
After I saw Dgx spark’s price increased $700, I hurry up and pick one GIGABYTE from a store that still has the stock. Its price did not increase yet.
can you do benchmark on Qwen3.5-27B int4-autoround? It ‘s my favorite. I want to see the comparison with 122B-1A10B-int4-AutoRound? thanks.
I only run the 397b autoround model with spark-vllm-docker nowadays
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:---------------------------------------|----------------:|---------------:|-------------:|-----------------:|-----------------:|-----------------:|
| Intel/Qwen3.5-397B-A17B-int4-AutoRound | pp2048 | 1955.36 ± 6.42 | | 1148.05 ± 3.09 | 1047.90 ± 3.09 | 1148.18 ± 3.09 |
| Intel/Qwen3.5-397B-A17B-int4-AutoRound | tg32 | 28.44 ± 0.10 | 29.00 ± 0.00 | | | |
| Intel/Qwen3.5-397B-A17B-int4-AutoRound | pp2048 @ d4096 | 1988.45 ± 4.73 | | 3190.35 ± 7.12 | 3090.20 ± 7.12 | 3190.46 ± 7.13 |
| Intel/Qwen3.5-397B-A17B-int4-AutoRound | tg32 @ d4096 | 28.35 ± 0.04 | 29.00 ± 0.00 | | | |
| Intel/Qwen3.5-397B-A17B-int4-AutoRound | pp2048 @ d8192 | 1975.06 ± 2.62 | | 5285.16 ± 7.05 | 5185.01 ± 7.05 | 5285.26 ± 7.05 |
| Intel/Qwen3.5-397B-A17B-int4-AutoRound | tg32 @ d8192 | 28.35 ± 0.16 | 29.00 ± 0.00 | | | |
| Intel/Qwen3.5-397B-A17B-int4-AutoRound | pp2048 @ d16384 | 1913.61 ± 2.57 | | 9732.59 ± 13.14 | 9632.43 ± 13.14 | 9732.70 ± 13.16 |
| Intel/Qwen3.5-397B-A17B-int4-AutoRound | tg32 @ d16384 | 28.32 ± 0.25 | 29.00 ± 0.00 | | | |
| Intel/Qwen3.5-397B-A17B-int4-AutoRound | pp2048 @ d32768 | 1783.73 ± 3.39 | | 19619.28 ± 37.00 | 19519.12 ± 37.00 | 19619.39 ± 36.99 |
| Intel/Qwen3.5-397B-A17B-int4-AutoRound | tg32 @ d32768 | 27.96 ± 0.10 | 28.33 ± 0.47 | | | |