I ran benchmarks of Qwen3.5 35b through 0.8b models using all different quants and compared performance with flash attention enabled vs disabled. Using llama.cpp and Unsloth’s UD_K_XL versions.
Specifically for the Qwen3.5 9B and Qwen3.5 35B-A3B, both had weird behavior where the BF16 and Q8 versions performed worse with flash attention enabled with context sizes of 0 through 10k. All other Qwen3.5 models and quants saw performance gains with flash attention.
Any thoughts on why this occurred or how I can try to figure out why thats the case?
Here is perf for TPS of the Qwen3.5 35B-A3B, solid lines are FA disabled, dashes are FA enabled
Benchmark commands:
./llama-bench --model \ -p 1000 -n 50 -d 0,1000,2000,3000,4000,5000,6000,7000,8000,9000,10000
./llama-bench --model \ -p 1000 -n 50 -fa 1 -d 0,1000,2000,3000,4000,5000,6000,7000,8000,9000,10000
llama.cpp cmake config options:
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON -DGGML_CUDA=ON -DGGML_CURL=ON -DGGML_RPC=ON -DCMAKE_CUDA_ARCHITECTURES=121a-real -DGGML_CUDA_F16=ON
