Deployed Qwen3.6-35B-A3b on dual sparks. Speed randomly drops to 0-5 tok/s when multiple requests are hit. both gpus are consuming 105gb memory each, anyone knows how I can improve this random speed going up/ down? This is how i served the model :
recipe_version: “1”
name: Qwen36-35B-A3B
description: vLLM serving Qwen3.6-35B-A3B
HuggingFace model to download (optional, for --download-model)
model: Qwen/Qwen3.6-35B-A3B
#solo_only: true
Container image to use
container: vllm-node
mods:
- mods/fix-qwen3.6-chat-template
Default settings (can be overridden via CLI)
defaults:
port: 8000
host: 0.0.0.0
tensor_parallel: 2
gpu_memory_utilization: 0.85
max_model_len: 262144
max_num_batched_tokens: 16384
max_num_seqs: 8
Environment variables
env:
VLLM_MARLIN_USE_ATOMIC_ADD: 1
The vLLM serve command template
command: |
vllm serve Qwen/Qwen3.6-35B-A3B
–host {host}
–port {port}
–max-model-len {max_model_len}
–max-num-batched-tokens {max_num_batched_tokens}
–gpu-memory-utilization {gpu_memory_utilization}
–enable-auto-tool-choice
–tool-call-parser qwen3_xml
–reasoning-parser qwen3
–max-num-seqs {max_num_seqs}
–default-chat-template-kwargs ‘{{“preserve_thinking”: true, “enable_thinking”: false}}’
–attention-backend flashinfer
–enable-prefix-caching
-tp {tensor_parallel}
–distributed-executor-backend ray
O 08-03 12:53:35 [loggers.py:310] Engine 000: Avg prompt throughput: 594.1 tokens/s, Avg generation throughput: 32.4 tokens/s, Running: 2 reqs, Waiting: 1 reqs, GPU KV cache usage: 1.8%, Prefix cache hit rate: 0.0%
(APIServer pid=39) INFO 08-03 12:53:45 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 2 reqs, Waiting: 1 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 0.0%
(APIServer pid=39) INFO 08-03 12:53:55 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 2 reqs, Waiting: 1 reqs, GPU KV cache usage: 5.7%, Prefix cache hit rate: 0.0%
(APIServer pid=39) INFO 08-03 12:54:05 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 2 reqs, Waiting: 1 reqs, GPU KV cache usage: 7.7%, Prefix cache hit rate: 0.0%
(APIServer pid=39) INFO 08-03 12:54:15 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 2 reqs, Waiting: 1 reqs, GPU KV cache usage: 8.7%, Prefix cache hit rate: 0.0%
(APIServer pid=39) INFO 08-03 12:54:25 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.9%, Prefix cache hit rate: 0.0%
(APIServer pid=39) INFO 08-03 12:54:35 [loggers.py:310] Engine 000: Avg prompt throughput: 16543.6 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.0%, Prefix cache hit rate: 0.0%