Qwen3.5 Flash Attention performance inconsistencies

I ran benchmarks of Qwen3.5 35b through 0.8b models using all different quants and compared performance with flash attention enabled vs disabled. Using llama.cpp and Unsloth’s UD_K_XL versions.

Specifically for the Qwen3.5 9B and Qwen3.5 35B-A3B, both had weird behavior where the BF16 and Q8 versions performed worse with flash attention enabled with context sizes of 0 through 10k. All other Qwen3.5 models and quants saw performance gains with flash attention.

Any thoughts on why this occurred or how I can try to figure out why thats the case?

Here is perf for TPS of the Qwen3.5 35B-A3B, solid lines are FA disabled, dashes are FA enabled

Benchmark commands:

./llama-bench --model \ -p 1000 -n 50 -d 0,1000,2000,3000,4000,5000,6000,7000,8000,9000,10000
./llama-bench --model \ -p 1000 -n 50 -fa 1 -d 0,1000,2000,3000,4000,5000,6000,7000,8000,9000,10000

llama.cpp cmake config options:

cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON -DGGML_CUDA=ON -DGGML_CURL=ON -DGGML_RPC=ON -DCMAKE_CUDA_ARCHITECTURES=121a-real -DGGML_CUDA_F16=ON

could you try llama-benchy (our community tool) GitHub - eugr/llama-benchy: llama-benchy - llama-bench style benchmarking tool for all backends · GitHub

Our community have been publishing benchmark results for Qwen 3.5 and other models on vLLM and SGLang at: https://spark-arena.com, your contributions are welcome.