Just thought I would share the recipe. Its very stable and snappy.
Qwen/Qwen3.6-35B-A3B-FP8 through vLLM with tensor parallel across both nodes.
Results, direct to vLLM API:
Small context:
Prompt tokens: 5,098
Cold TTFT: 0.68s
Cold input speed: ~7,487 tok/s
Output speed: ~79.9 tok/s
Medium context:
Prompt tokens: 40,474
Cold TTFT: 5.23s
Cold input speed: ~7,740 tok/s
Output speed: ~78.5 tok/s
Large context:
Prompt tokens: 80,890
Cold TTFT: 8.49s
Cold input speed: ~9,522 tok/s
Output speed: ~75.3 tok/s
Prefix cache kicked in hard on second runs:
Medium run 2 TTFT: 0.47s
Large run 2 TTFT: 0.99s
So the practical read is:
Cold prefill: ~7.5k–9.5k input tok/s
Warm/prefix cache: massively faster on repeated context
Decode speed: ~75–80 output tok/s
80k context: stable
Model / runtime:
Model: Qwen/Qwen3.6-35B-A3B-FP8
Serving stack: vLLM OpenAI-compatible API
Recipe: spark-vllm-docker
Launch recipe: ./run-recipe.sh qwen3.6-35b-a3b-fp8 --no-ray
Tensor parallel: TP=2 across two GX10 nodes
Backend: NCCL / PYNCCL
Attention backend: FlashInfer
KV cache dtype: FP8
Max model len: 262,144
Prefix caching: enabled
Tool parser: qwen3_xml
Chat template: fixed Qwen3.6 template
Hardware / networking:
Nodes: 2x ASUS GX10 4TB
Combined memory: ~258 GB unified memory
Head node: gx10-9141
Worker node: gx10-5611
Cluster network: 200GbE RoCE direct link
Interface: enp1s0f1np1
MTU: 9000