Been playing around with llama.cpp RPC distributed mode with sparkrun and I wanted to share some benchmarks/info. Realistically, I much prefer sglang and vllm for deployments; however, since I don’t have enough hardware to quantize models like Qwen3.5 397B BF16 myself… and it doesn’t feel worth it to rent equipment when I know they’ll be available soon enough anyway… so it is tempting to work with llama.cpp and GGUF quantized models to try them out in the meantime.
I did this stuff on 4x Spark cluster using Q_6_K GGUF quant–technically with a smaller quant (e.g. Q_3_K_M), should be able to do the same with 2x Sparks. The sparkrun-official registry has both in there for other people to take a look and try their own experiments. Run sparkrun setup update and sparkrun recipe update to make sure you have the latest version and latest recipes downloaded from registries. (Or install sparkrun with uvx sparkrun setup install if you haven’t already).
Launch: sparkrun run qwen3.5-397b-gguf-q6k-llama-cpp --tp 4 --o split_mode=layer
Details: 4x Sparks, Q_6_K unsloth QQUF quant, RPC over 100Gbps Ethernet (unbonded CX7), split mode: layer (meaning that llama.cpp helps us divide it over 4x sparks but it basically runs sequentially through the four so it just helps with the lack of RAM, not performance)
Benchmark: uvx llama-benchy --base-url "http://10.24.11.13:8000/v1" --pp 512 2048 8192 16384 32768 65535 --tg 32 128 --runs 5 --model "qwen3.5-397b"
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:-------------|--------:|-------------:|------------:|---------------------:|---------------------:|---------------------:|
| qwen3.5-397b | pp512 | 69.79 ± 2.59 | | 6476.48 ± 429.80 | 6475.07 ± 429.80 | 6476.52 ± 429.81 |
| qwen3.5-397b | tg32 | 5.02 ± 0.18 | 7.40 ± 0.49 | | | |
| qwen3.5-397b | pp512 | 71.14 ± 2.24 | | 6595.32 ± 362.76 | 6593.91 ± 362.76 | 6595.36 ± 362.76 |
| qwen3.5-397b | tg128 | 6.30 ± 0.34 | 8.00 ± 0.00 | | | |
| qwen3.5-397b | pp2048 | 72.36 ± 0.79 | | 25583.74 ± 596.07 | 25582.33 ± 596.07 | 25583.78 ± 596.07 |
| qwen3.5-397b | tg32 | 4.85 ± 0.35 | 7.40 ± 0.49 | | | |
| qwen3.5-397b | pp2048 | 72.06 ± 0.71 | | 25204.35 ± 755.50 | 25202.94 ± 755.50 | 25204.39 ± 755.50 |
| qwen3.5-397b | tg128 | 6.44 ± 0.26 | 8.00 ± 0.00 | | | |
| qwen3.5-397b | pp8192 | 73.92 ± 1.88 | | 99003.67 ± 3377.85 | 99002.26 ± 3377.85 | 99003.71 ± 3377.86 |
| qwen3.5-397b | tg32 | 5.19 ± 0.54 | 7.40 ± 0.49 | | | |
| qwen3.5-397b | pp8192 | 77.34 ± 0.38 | | 95272.14 ± 1255.03 | 95270.74 ± 1255.03 | 95272.19 ± 1255.02 |
| qwen3.5-397b | tg128 | 6.08 ± 0.13 | 7.80 ± 0.40 | | | |
| qwen3.5-397b | pp16384 | 71.55 ± 1.42 | | 208078.79 ± 4737.01 | 208077.38 ± 4737.01 | 208078.83 ± 4737.01 |
| qwen3.5-397b | tg32 | 4.89 ± 0.45 | 7.20 ± 0.40 | | | |
| qwen3.5-397b | pp16384 | 71.65 ± 1.31 | | 208173.17 ± 3948.48 | 208171.76 ± 3948.48 | 208173.20 ± 3948.48 |
| qwen3.5-397b | tg128 | 5.70 ± 0.42 | 7.40 ± 0.49 | | | |
| qwen3.5-397b | pp32768 | 65.65 ± 1.94 | | 451942.23 ± 13109.72 | 451940.83 ± 13109.72 | 451942.27 ± 13109.72 |
| qwen3.5-397b | tg32 | 4.48 ± 0.27 | 6.40 ± 0.49 | | | |
| qwen3.5-397b | pp32768 | 64.48 ± 0.24 | | 460478.57 ± 5472.94 | 460477.16 ± 5472.94 | 460478.61 ± 5472.94 |
| qwen3.5-397b | tg128 | 5.12 ± 0.08 | 7.00 ± 0.00 | | | |
llama-benchy (0.3.1)
date: 2026-02-18 23:16:06 | latency mode: api
Note: RPC failures when went to 65535 – head node + 1x worker failed; 2x workers remained up.
Launch: sparkrun run qwen3.5-397b-gguf-q6k-llama-cpp --tp 4 --o split_mode=row
Details: 4x Sparks, Q_6_K unsloth QQUF quant, RPC over 100Gbps Ethernet (unbonded CX7), split mode: row (meaning that we sorta try to get some performance benefit despite the significant overhead compared to NCCL)
Benchmark: uvx llama-benchy --base-url "http://10.24.11.13:8000/v1" --pp 512 2048 8192 16384 32768 65535 --tg 32 128 --runs 5 --model "qwen3.5-397b"
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|:-------------|--------:|-------------:|------------:|---------------------:|---------------------:|---------------------:|
| qwen3.5-397b | pp512 | 74.11 ± 1.38 | | 6327.44 ± 309.88 | 6325.60 ± 309.88 | 6327.47 ± 309.88 |
| qwen3.5-397b | tg32 | 5.60 ± 0.34 | 8.40 ± 0.49 | | | |
| qwen3.5-397b | pp512 | 74.53 ± 1.11 | | 6230.79 ± 285.99 | 6228.96 ± 285.99 | 6230.83 ± 285.99 |
| qwen3.5-397b | tg128 | 7.03 ± 0.32 | 9.00 ± 0.00 | | | |
| qwen3.5-397b | pp2048 | 76.62 ± 0.71 | | 24053.95 ± 982.84 | 24052.12 ± 982.84 | 24054.00 ± 982.84 |
| qwen3.5-397b | tg32 | 5.40 ± 0.15 | 8.00 ± 0.00 | | | |
| qwen3.5-397b | pp2048 | 77.77 ± 0.47 | | 24313.82 ± 1067.11 | 24311.99 ± 1067.11 | 24313.87 ± 1067.11 |
| qwen3.5-397b | tg128 | 6.82 ± 0.28 | 8.80 ± 0.40 | | | |
| qwen3.5-397b | pp8192 | 77.56 ± 1.45 | | 95965.24 ± 1751.05 | 95963.41 ± 1751.05 | 95965.29 ± 1751.06 |
| qwen3.5-397b | tg32 | 5.62 ± 0.36 | 7.80 ± 0.40 | | | |
| qwen3.5-397b | pp8192 | 80.13 ± 0.43 | | 93460.38 ± 794.44 | 93458.55 ± 794.44 | 93460.42 ± 794.44 |
| qwen3.5-397b | tg128 | 6.51 ± 0.11 | 8.00 ± 0.00 | | | |
| qwen3.5-397b | pp16384 | 74.57 ± 1.42 | | 200170.48 ± 3864.25 | 200168.65 ± 3864.25 | 200170.51 ± 3864.25 |
| qwen3.5-397b | tg32 | 5.31 ± 0.47 | 7.60 ± 0.49 | | | |
| qwen3.5-397b | pp16384 | 74.85 ± 0.47 | | 199896.99 ± 1363.42 | 199895.16 ± 1363.42 | 199897.03 ± 1363.42 |
| qwen3.5-397b | tg128 | 6.15 ± 0.27 | 7.80 ± 0.40 | | | |
| qwen3.5-397b | pp32768 | 68.37 ± 2.01 | | 434767.45 ± 12431.53 | 434765.62 ± 12431.53 | 434767.49 ± 12431.52 |
| qwen3.5-397b | tg32 | 5.33 ± 0.45 | 7.20 ± 0.40 | | | |
| qwen3.5-397b | pp32768 | 66.54 ± 0.71 | | 447694.01 ± 7620.90 | 447692.17 ± 7620.90 | 447694.04 ± 7620.90 |
| qwen3.5-397b | tg128 | 5.64 ± 0.18 | 7.00 ± 0.00 | | | |
llama-benchy (0.3.1)
date: 2026-02-19 09:37:50 | latency mode: api
Note: RPC failures when went to 65535 – head node + 1x worker failed; 2x workers remained up.
Unrelated but also a shameless plug, there is a really bizarre joy of instructing Claude Code to start/stop models or tell me what’s running on the cluster. Despite the similarities with other similar “launcher” code out there, sparkrun does have some key differences (little details but from years of doing this stuff on HPC clusters)… but the tab-completion at CLI and Claude code plugin are pretty awesome… https://sparkrun.dev
Hopefully sparkrun makes it easier for people to participate in benchmarking and experimentation – and in this case, I hope that opening up vllm, sglang, and llama.cpp with a relatively streamlined approach (plus recipe registries) will help jumpstart more sharing of knowledge how to tune stuff on the Spark.
…And now I’m off to see what I can do with Qwen 3.5 FP8 on sglang since that’s been released.