# New tool: llama-benchy - llama-bench style benchmarking for ANY LLM backend (vLLM, SGLang, llama.cpp, etc.) **URL:** \ **Category:** DGX Spark / GB10 Projects\ **Tags:** llama\ **Created:** [January 6, 2026, 7:57pm UTC](https://forums.developer.nvidia.com/t/new-tool-llama-benchy-llama-bench-style-benchmarking-for-any-llm-backend-vllm-sglang-llama-cpp-etc/356698 "2026-01-06T19:57:52Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![eugr](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/eugr/32/449615_2.png) [@eugr](https://forums.developer.nvidia.com/u/eugr)\ **Post date:** [January 6, 2026, 7:57pm UTC](https://forums.developer.nvidia.com/t/new-tool-llama-benchy-llama-bench-style-benchmarking-for-any-llm-backend-vllm-sglang-llama-cpp-etc/356698/1 "2026-01-06T19:57:52Z")
## Why I built this? I’ve been happily using llama-bench to benchmark local models performance running in llama.cpp. One great feature is that it can help to evaluate performance at different context lengths and present the output in a table format that is easy to digest. However, llama.cpp is not the only inference engine I use, I also use SGLang and vLLM. But llama-bench can only work with llama.cpp, and other benchmarking tools that I found are more focused on concurrency and total throughput. Also, llama-bench performs measurements using the C++ engine directly which is not representative of the end user experience which can be quite different in practice. vLLM has its own powerful benchmarking tool, but while it can be used with other inference engines, there are a few issues: - You can’t easily measure how prompt processing speed degrades as context grows. You can use `vllm bench sweep serve`, but it only works well with vLLM with prefix caching disabled on the server. Even with random prompts it will reuse the same prompt between multiple runs which will hit the cache in `llama-server` for instance. So you will get very low median TTFT times and very high prompt processing speeds. - The TTFT measurement it uses is not actually until the first usable token, it’s until the very first data chunk from the server which may not contain any generated tokens in /v1/chat/completions mode. - Random dataset is the only ones that allows to specify an arbitrary number of tokens, but randomly generated token sequence doesn’t let you adequately measure speculative decoding/MTP. As of today, I haven’t been able to find any existing benchmarking tool that brings llama-bench style measurements at different context lengths to any OpenAI-compatible endpoint. ## What is llama-benchy? It’s a CLI benchmarking tool that measures: - Prompt Processing (pp) and Token Generation (tg) speeds at different context lengths. - Allows to benchmark context prefill and follow up prompt separately. - Reports additional metrics, like time to first response, estimated prompt processing time and end-to-end time to first token. It works with any OpenAI-compatible endpoint that exposes /v1/chat/completions and also: - Supports configurable prompt length (`--pp`), generation length (`--tg`), and context depth (`--depth`). - Can run multiple iterations (`--runs`) and report mean ± std. - Uses HuggingFace tokenizers for accurate token counts. - Downloads a book from Project Gutenberg to use as source text for prompts to ensure better benchmarking of spec.decoding/MTP models. - Supports executing a command after each run (e.g., to clear cache). - Configurable latency measurement mode to estimate server/network overhead and provide more accurate prompt processing numbers. ## Quick Demo Benchmarking MiniMax 2.1 AWQ running on my dual Spark cluster with up to 100000 context: ```bash # Run without installation uvx llama-benchy --base-url http://spark:8888/v1 --model cyankiwi/MiniMax-M2.1-AWQ-4bit --depth 0 4096 8192 16384 32768 65535 100000 --adapt-prompt --latency-mode generation --enable-prefix-caching ``` Output: | model | test | t/s | ttfr (ms) | est\_ppt (ms) | e2e\_ttft (ms) | | --- | --- | --- | --- | --- | --- | | cyankiwi/MiniMax-M2.1-AWQ-4bit | pp2048 | 3544.10 ± 37.29 | 688.41 ± 6.09 | 577.93 ± 6.09 | 688.45 ± 6.10 | | cyankiwi/MiniMax-M2.1-AWQ-4bit | tg32 | 36.11 ± 0.06 | | | | | cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx\_pp @ d4096 | 3150.63 ± 7.84 | 1410.55 ± 3.24 | 1300.06 ± 3.24 | 1410.58 ± 3.24 | | cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx\_tg @ d4096 | 34.36 ± 0.08 | | | | | cyankiwi/MiniMax-M2.1-AWQ-4bit | pp2048 @ d4096 | 2562.47 ± 21.71 | 909.77 ± 6.75 | 799.29 ± 6.75 | 909.81 ± 6.75 | | cyankiwi/MiniMax-M2.1-AWQ-4bit | tg32 @ d4096 | 33.41 ± 0.05 | | | | | cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx\_pp @ d8192 | 2832.52 ± 12.34 | 3002.66 ± 12.57 | 2892.18 ± 12.57 | 3002.70 ± 12.57 | | cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx\_tg @ d8192 | 31.38 ± 0.06 | | | | | cyankiwi/MiniMax-M2.1-AWQ-4bit | pp2048 @ d8192 | 2261.83 ± 10.69 | 1015.96 ± 4.29 | 905.48 ± 4.29 | 1016.00 ± 4.29 | | cyankiwi/MiniMax-M2.1-AWQ-4bit | tg32 @ d8192 | 30.55 ± 0.08 | | | | | cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx\_pp @ d16384 | 2473.70 ± 2.15 | 6733.76 ± 5.76 | 6623.28 ± 5.76 | 6733.80 ± 5.75 | | cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx\_tg @ d16384 | 27.89 ± 0.04 | | | | | cyankiwi/MiniMax-M2.1-AWQ-4bit | pp2048 @ d16384 | 1824.55 ± 6.32 | 1232.96 ± 3.89 | 1122.48 ± 3.89 | 1233.00 ± 3.89 | | cyankiwi/MiniMax-M2.1-AWQ-4bit | tg32 @ d16384 | 27.21 ± 0.04 | | | | | cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx\_pp @ d32768 | 2011.11 ± 2.40 | 16403.98 ± 19.43 | 16293.50 ± 19.43 | 16404.03 ± 19.43 | | cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx\_tg @ d32768 | 22.09 ± 0.07 | | | | | cyankiwi/MiniMax-M2.1-AWQ-4bit | pp2048 @ d32768 | 1323.21 ± 4.62 | 1658.25 ± 5.41 | 1547.77 ± 5.41 | 1658.29 ± 5.41 | | cyankiwi/MiniMax-M2.1-AWQ-4bit | tg32 @ d32768 | 21.81 ± 0.07 | | | | | cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx\_pp @ d65535 | 1457.71 ± 0.26 | 45067.98 ± 7.94 | 44957.50 ± 7.94 | 45068.01 ± 7.94 | | cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx\_tg @ d65535 | 15.72 ± 0.04 | | | | | cyankiwi/MiniMax-M2.1-AWQ-4bit | pp2048 @ d65535 | 840.36 ± 2.35 | 2547.54 ± 6.79 | 2437.06 ± 6.79 | 2547.60 ± 6.80 | | cyankiwi/MiniMax-M2.1-AWQ-4bit | tg32 @ d65535 | 15.63 ± 0.02 | | | | | cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx\_pp @ d100000 | 1130.05 ± 1.89 | 88602.31 ± 148.70 | 88491.83 ± 148.70 | 88602.37 ± 148.70 | | cyankiwi/MiniMax-M2.1-AWQ-4bit | ctx\_tg @ d100000 | 12.14 ± 0.02 | | | | | cyankiwi/MiniMax-M2.1-AWQ-4bit | pp2048 @ d100000 | 611.01 ± 2.50 | 3462.39 ± 13.73 | 3351.90 ± 13.73 | 3462.42 ± 13.73 | | cyankiwi/MiniMax-M2.1-AWQ-4bit | tg32 @ d100000 | 12.05 ± 0.03 | | | | llama-benchy (0.1.0) date: 2026-01-06 11:44:49 | latency mode: generation ## GitHub [https://github.com/eugr/llama-benchy](https://github.com/eugr/llama-benchy) ---
**Author:** ![raphael.amorim](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/raphael.amorim/32/446733_2.png) [@raphael.amorim](https://forums.developer.nvidia.com/u/raphael.amorim)\ **Post date:** [January 6, 2026, 8:46pm UTC](https://forums.developer.nvidia.com/t/new-tool-llama-benchy-llama-bench-style-benchmarking-for-any-llm-backend-vllm-sglang-llama-cpp-etc/356698/2 "2026-01-06T20:46:09Z")
Thanks for one more amazing contribution @eugr ---
**Author:** ![eugr](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/eugr/32/449615_2.png) [@eugr](https://forums.developer.nvidia.com/u/eugr)\ **Post date:** [February 6, 2026, 12:22am UTC](https://forums.developer.nvidia.com/t/new-tool-llama-benchy-llama-bench-style-benchmarking-for-any-llm-backend-vllm-sglang-llama-cpp-etc/356698/3 "2026-02-06T00:22:23Z")
New major update is out - v0.2.0. It brings: - concurrency support - added JSON and CSV outputs. JSON output contains values for individual runs in addition to mean/std values. - ability to save results to a file Example of concurrency testing. This is for model running on a single Spark with the following parameters: - gpu\_memory\_utilization: 0.7 - max\_model\_len: 202752 - max\_num\_batched\_tokens: 4096 - max\_num\_seqs: 64 ```bash llama-benchy \ --base-url http://spark:8888/v1 \ --model cyankiwi/GLM-4.7-Flash-AWQ-4bit \ --served-model-name glm-4.7-flash \ --depth 0 4096 \ --adapt-prompt \ --concurrency 1 2 10 ``` Maximum supported concurrency is: ```auto GPU KV cache size: 1,239,088 tokens Maximum concurrency for 202,752 tokens per request: 6.11x ``` | model | test | t/s (total) | t/s (req) | ttfr (ms) | est\_ppt (ms) | e2e\_ttft (ms) | | --- | --- | --- | --- | --- | --- | --- | | cyankiwi/GLM-4.7-Flash-AWQ-4bit | pp2048 (c1) | 5326.13 ± 38.60 | 5326.13 ± 38.60 | 336.01 ± 5.86 | 331.75 ± 5.86 | 336.14 ± 5.82 | | cyankiwi/GLM-4.7-Flash-AWQ-4bit | tg32 (c1) | 41.75 ± 0.03 | 41.75 ± 0.03 | | | | | cyankiwi/GLM-4.7-Flash-AWQ-4bit | pp2048 (c2) | 5772.72 ± 310.43 | 2953.01 ± 141.62 | 632.95 ± 22.99 | 628.68 ± 22.99 | 633.00 ± 22.99 | | cyankiwi/GLM-4.7-Flash-AWQ-4bit | tg32 (c2) | 73.74 ± 1.31 | 37.38 ± 0.63 | | | | | cyankiwi/GLM-4.7-Flash-AWQ-4bit | pp2048 (c10) | 6118.58 ± 28.10 | 897.80 ± 400.77 | 2336.27 ± 705.73 | 2332.00 ± 705.73 | 2336.31 ± 705.72 | | cyankiwi/GLM-4.7-Flash-AWQ-4bit | tg32 (c10) | 87.65 ± 2.76 | 15.33 ± 4.11 | | | | | cyankiwi/GLM-4.7-Flash-AWQ-4bit | pp2048 @ d4096 (c1) | 5066.12 ± 9.45 | 5066.12 ± 9.45 | 1097.99 ± 18.53 | 1093.72 ± 18.53 | 1098.06 ± 18.53 | | cyankiwi/GLM-4.7-Flash-AWQ-4bit | tg32 @ d4096 (c1) | 39.27 ± 0.11 | 39.27 ± 0.11 | | | | | cyankiwi/GLM-4.7-Flash-AWQ-4bit | pp2048 @ d4096 (c2) | 5274.22 ± 15.83 | 2665.63 ± 38.23 | 2042.41 ± 37.06 | 2038.14 ± 37.06 | 2042.50 ± 37.06 | | cyankiwi/GLM-4.7-Flash-AWQ-4bit | tg32 @ d4096 (c2) | 68.02 ± 0.32 | 34.73 ± 0.66 | | | | | cyankiwi/GLM-4.7-Flash-AWQ-4bit | pp2048 @ d4096 (c10) | 5217.03 ± 15.20 | 1051.40 ± 608.75 | 6666.76 ± 2782.40 | 6662.49 ± 2782.40 | 6666.81 ± 2782.39 | | cyankiwi/GLM-4.7-Flash-AWQ-4bit | tg32 @ d4096 (c10) | 31.33 ± 0.21 | 8.14 ± 5.03 | | | | llama-benchy (0.2.0) date: 2026-02-05 15:49:22 | latency mode: api Please note that that **t/s (total)** is measured for the entire concurrent batch run, and does not indicate peak throughput. Peak throughput will be added in the next release. ---
**Author:** ![cosinus](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/cosinus/32/454111_2.png) [@cosinus](https://forums.developer.nvidia.com/u/cosinus)\ **Post date:** [February 6, 2026, 4:41pm UTC](https://forums.developer.nvidia.com/t/new-tool-llama-benchy-llama-bench-style-benchmarking-for-any-llm-backend-vllm-sglang-llama-cpp-etc/356698/4 "2026-02-06T16:41:11Z")
Very nice! BTW are you familiar with aiperf? > **[GitHub - ai-dynamo/aiperf: AIPerf is a comprehensive benchmarking tool that...](https://github.com/ai-dynamo/aiperf)** > > AIPerf is a comprehensive benchmarking tool that measures the performance of generative AI models served by your preferred inference solution. Playing around with it lately. May a source of inspiration. ---
**Author:** ![eugr](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/eugr/32/449615_2.png) [@eugr](https://forums.developer.nvidia.com/u/eugr)\ **Post date:** [February 6, 2026, 5:59pm UTC](https://forums.developer.nvidia.com/t/new-tool-llama-benchy-llama-bench-style-benchmarking-for-any-llm-backend-vllm-sglang-llama-cpp-etc/356698/5 "2026-02-06T17:59:50Z")
Yes, I’ve looked at it, as well as a few others, but it’s closer to `vllm bench serve` - lots of functionality, but I wanted to create something like llama-bench, but for all backends, so we could see the effect of context on follow up requests. And have a simple to digest output. ---
**Author:** ![wdhong](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@wdhong](https://forums.developer.nvidia.com/u/wdhong)\ **Post date:** [February 9, 2026, 3:01am UTC](https://forums.developer.nvidia.com/t/new-tool-llama-benchy-llama-bench-style-benchmarking-for-any-llm-backend-vllm-sglang-llama-cpp-etc/356698/6 "2026-02-09T03:01:39Z")
Thank you for the share, eugr. I’m curious if the **oss-120b** benchmark results on GitHub were obtained on a **single DGX Spark** setup? ---
**Author:** ![eugr](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/eugr/32/449615_2.png) [@eugr](https://forums.developer.nvidia.com/u/eugr)\ **Post date:** [February 9, 2026, 5:15am UTC](https://forums.developer.nvidia.com/t/new-tool-llama-benchy-llama-bench-style-benchmarking-for-any-llm-backend-vllm-sglang-llama-cpp-etc/356698/7 "2026-02-09T05:15:07Z")
It’s a mix. Most were dual Sparks, with the exception of concurrency example which was for a single Spark. The numbers for dual sparks are lower than they should be, I was running some other stuff on those sparks at that time, and it slowed things down a bit. On a good day I see up to 77-78 t/s on single requests with low context. ---
**Author:** ![raphael.amorim](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/raphael.amorim/32/446733_2.png) [@raphael.amorim](https://forums.developer.nvidia.com/u/raphael.amorim)\ **Post date:** [February 13, 2026, 1:11am UTC](https://forums.developer.nvidia.com/t/new-tool-llama-benchy-llama-bench-style-benchmarking-for-any-llm-backend-vllm-sglang-llama-cpp-etc/356698/8 "2026-02-13T01:11:31Z")
Feel free to check latest numbers for some models here as well > [@Introducing the Spark Arena](https://forums.developer.nvidia.com/t/introducing-the-spark-arena/360319): > > Hi everyone 👋 Over the past weeks, I’ve been working with @eugr on adding structured, reproducible “benchmark recipes” to our [community Docker runtime for LLMs](https://github.com/eugr/spark-vllm-docker). @eugr also added new export formats to [Llama-Benchy](https://github.com/eugr/llama-benchy). Those are foundations for a better knowledge sharing experience, but we still needed a common platform to publish our experiments. The problem we keep seeing on DGX Spark threads is not lack of experimentation. It’s lack of reproducibility and indexing shared experiments. Introducin… ---
**Author:** ![eugr](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/eugr/32/449615_2.png) [@eugr](https://forums.developer.nvidia.com/u/eugr)\ **Post date:** [March 12, 2026, 6:32am UTC](https://forums.developer.nvidia.com/t/new-tool-llama-benchy-llama-bench-style-benchmarking-for-any-llm-backend-vllm-sglang-llama-cpp-etc/356698/9 "2026-03-12T06:32:29Z")
PSA: llama-benchy will now try to infer HF model name (and served-model-name if applicable) from the endpoint if `--model` parameter is not specified. Works with vLLM and SGLang. While it works with llama.cpp if the server was launched with `-hf` parameter (passing HF model name), depending on the GGUF model repository, it may not be able to load a tokenizer. In this case you can pass HF model name in non-GGUF format that has a tokenizer in the repository. It will still work, but may report slightly incorrect results depending on how different is the tokenizer from the default gpt2 one. ---
**Author:** ![space11](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@space11](https://forums.developer.nvidia.com/u/space11)\ **Post date:** [March 12, 2026, 9:55am UTC](https://forums.developer.nvidia.com/t/new-tool-llama-benchy-llama-bench-style-benchmarking-for-any-llm-backend-vllm-sglang-llama-cpp-etc/356698/10 "2026-03-12T09:55:55Z")
thanks for doing what you are doing @eugr , more helpful than nvidia staff - hope they send you some sparks for free as a thank you at least ---
**Author:** ![martinB78](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/martinb78/32/437495_2.png) [@martinB78](https://forums.developer.nvidia.com/u/martinB78)\ **Post date:** [April 15, 2026, 9:39am UTC](https://forums.developer.nvidia.com/t/new-tool-llama-benchy-llama-bench-style-benchmarking-for-any-llm-backend-vllm-sglang-llama-cpp-etc/356698/11 "2026-04-15T09:39:32Z")
I want to batch benchmark all my LLMs and use lama-benchy for it. Most are served by vLLM some are lama.cpp and I have not looked into SGLang yet. The setup is liteLLM → Lama-Swap → vLLM or lama.cpp Guess there is already a batch benchmark setting in lama-benchy but I could not find it. So I have let the AI write a Benchmark batch script for me.. it works.. you just need to change the lama-swap destination [http://localhost:28080](http://localhost:28080) to the one you use. **benchmark-models.sh — Benchmark all llama-swap models using llama-benchy** Uses llama-benchy ([GitHub - eugr/llama-benchy: llama-benchy - llama-bench style benchmarking tool for all backends · GitHub](https://github.com/eugr/llama-benchy)) for standardized LLM performance measurement. Results are comparable with other DGX Spark users who use the same tool. **What it measures:** pp (prompt processing) = how fast the model reads your input (tok/s) tg (token generation) = how fast the model writes its reply (tok/s) TTFT (time to first token) = how long before you see the first word Usage: ` ./benchmark-models.sh` # Benchmark all models ` ./benchmark-models.sh Qwen3.5 Nemotron` # Only models matching names ` ./benchmark-models.sh --quick Nemotron` # Fast single-run test ` ./benchmark-models.sh --full` # Full sweep with depths ` ./benchmark-models.sh --runs 5 Qwen3.5-35B` # Custom run count the command line output looks like this ```auto ============================================================ DGX Spark Model Benchmark powered by llama-benchy (github.com/eugr/llama-benchy) ============================================================ Endpoint : http://localhost:28080 Mode : standard Settings : pp=512 2048 tg=128 256 depth=0 runs=3 Date : 2026-04-15 12:26 Results : /home/sparky/Docker/dgx-spark_lite-llm_llama-swap_vllm_llama-cpp_ollama/test-results/benchmarks pp = prompt processing (how fast the model reads your input) tg = token generation (how fast the model writes its reply) Found 15 model(s) to benchmark. ============================================================ [1/15] GPT-OSS-120B ============================================================ Loading model via llama-swap... Model ready (loaded in 209.411370441s) Profile: Medium Log (50-page document baseline) Running llama-benchy (pp=2048 tg=128 depth=0 16384 runs=3)... llama-benchy results (copy this to share on forums): | model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:-------------|----------------:|----------------:|-------------:|-----------------:|-----------------:|-----------------:| | GPT-OSS-120B | pp2048 | 3468.07 ± 76.96 | | 577.07 ± 17.35 | 528.72 ± 17.35 | 690.45 ± 23.61 | | GPT-OSS-120B | tg128 | 27.67 ± 0.07 | 29.00 ± 0.00 | | | | | GPT-OSS-120B | pp2048 @ d16384 | 2781.32 ± 87.36 | | 5959.68 ± 157.80 | 5911.33 ± 157.80 | 6081.95 ± 154.08 | | GPT-OSS-120B | tg128 @ d16384 | 25.02 ± 0.74 | 26.00 ± 0.82 | | | | Context Depth Analysis: --- depth=0 — Baseline (no context) --- Reading speed: 3,899 +/- 30 tok/s Writing speed: 27.8 tok/s (peak: 29) -> Fast — comfortable for interactive chat Time to first token: 686ms -> Short pause before response starts --- depth=16,384 — Medium Log (~50 pages) --- Reading speed: 2,981 +/- 55 tok/s Writing speed: 25.8 tok/s (peak: 27) -> -7.3% vs baseline (moderate slowdown) -> Fast — comfortable for interactive chat Time to first token: 5678ms -> Noticeable wait (5.7s) Overall impact: depth 0 -> 16,384 = -7.3% generation speed Conclusion: Model handles deep context well on this hardware JSON data : /home/sparky/Docker/dgx-spark_lite-llm_llama-swap_vllm_llama-cpp_ollama/test-results/benchmarks/GPT-OSS-120B_20260416_095536.json Forum table: /home/sparky/Docker/dgx-spark_lite-llm_llama-swap_vllm_llama-cpp_ollama/test-results/benchmarks/GPT-OSS-120B_20260416_095536.md ``` the GPT-OSS-120B\_20260415\_122622.md file content looks like this ```auto | model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:-------------|-------:|-----------------:|-------------:|---------------:|---------------:|----------------:| | GPT-OSS-120B | pp512 | 1888.57 ± 21.46 | | 283.45 ± 5.95 | 234.77 ± 5.95 | 384.97 ± 9.69 | | GPT-OSS-120B | tg128 | 31.26 ± 0.07 | 32.00 ± 0.00 | | | | | GPT-OSS-120B | pp512 | 1956.00 ± 122.30 | | 293.77 ± 6.04 | 245.09 ± 6.04 | 395.22 ± 6.88 | | GPT-OSS-120B | tg256 | 31.16 ± 0.07 | 32.00 ± 0.00 | | | | | GPT-OSS-120B | pp2048 | 3936.35 ± 107.89 | | 513.56 ± 6.30 | 464.88 ± 6.30 | 615.79 ± 10.97 | | GPT-OSS-120B | tg128 | 30.94 ± 0.06 | 32.00 ± 0.00 | | | | | GPT-OSS-120B | pp2048 | 3937.06 ± 25.26 | | 512.60 ± 15.64 | 463.92 ± 15.64 | 616.41 ± 15.08 | | GPT-OSS-120B | tg256 | 30.79 ± 0.06 | 32.00 ± 0.00 | | | | ``` at the end you get a summary ```auto ============================================================ BENCHMARK REPORT — DGX Spark ============================================================ Date: 2026-04-16 17:46 | Mode: medium-log | Runs: 3 Depths tested: 0 16384 Models tested: 15 | Passed: 15 Failed: 0 Total benchmark time: 77.8 min Model Read (pp) Write (tg) Peak TTFT Deep ctx tok/s tok/s tok/s ms degradation ---------------------------------------------------------------------------------------------------------- GPT-OSS-120B 4356 +/-46 32.3 33 595 -8.4% @16k Mistral-Small-24B-Instruct-2501 1955 +/-188 4.6 5 1166 -3.7% @16k Nemotron-3-Nano-30B-A3B-NVFP4 8392 +/-267 42.1 +/-1.7 45 290 — Nemotron-3-Nano-4B-FP8 7077 +/-1566 38.8 40 340 — Nemotron-3-Nano-4B-Q4_K_M-GGUF 2783 +/-76 49.7 51 760 — Nemotron-3-Super-120B-A12B-NVFP4 1725 +/-37 14.8 16 1221 -1.6% @16k Qwen3-Coder-Next-FP8-Dynamic 1986 +/-806 32.0 +/-1.2 34 1278 -0.4% @16k Qwen3-Coder-Next-int4-AutoRound 3268 +/-627 66.3 +/-1.1 68 644 -3.6% @16k Qwen3-Omni-30B-A3B-Instruct 4445 +/-1426 30.3 32 584 -15.0% @16k Qwen3-VL-30B-A3B-Instruct-FP8 6335 +/-3014 51.5 53 508 -12.2% @16k Qwen3.5-122B-A10B-int4-AutoRound 1760 +/-266 25.6 27 1210 -2.3% @16k Qwen3.5-35B-A3B-FP8 4166 +/-1748 21.2 22 689 -0.5% @16k Qwen3.5-35B-A3B-Uncensored-HauhauCS-Agg... 1742 +/-39 56.7 59 1160 — TheDrummer_GLM-Steam-106B-A12B-v1-Q4_K_... 538 +/-5 16.5 17 3443 — ikiru/dolphin-mistral-24b-venice-editio... 1512 +/-19 13.6 14 2115 -12.6% @16k --------------------------------------------------------------- How to read this table: Read (pp) = How fast the model reads your prompt (higher = better). Write (tg) = How fast the model types its answer at depth=0 baseline. This is the speed you feel when chatting. Humans read at ~4 tok/s, so 20+ feels smooth. Peak = Fastest burst speed observed in a 1-second window. TTFT = Time until the first word appears (lower = better). Deep ctx = Speed change at max tested depth vs baseline. >-15% = unified memory bandwidth bottleneck. --------------------------------------------------------------- Results saved to: Report : /home/sparky/Docker/dgx-spark_lite-llm_llama-swap_vllm_llama-cpp_ollama/test-results/benchmarks/report_20260416_162852.txt JSON data : /home/sparky/Docker/dgx-spark_lite-llm_llama-swap_vllm_llama-cpp_ollama/test-results/benchmarks/*_20260416_162852.json Forum tables: /home/sparky/Docker/dgx-spark_lite-llm_llama-swap_vllm_llama-cpp_ollama/test-results/benchmarks/*_20260416_162852.md ``` here is the script > **benchmark-models.sh** > > ```auto > # ============================================================================= > # benchmark-models.sh — Benchmark all llama-swap models using llama-benchy > # > # Uses llama-benchy (https://github.com/eugr/llama-benchy) for standardized > # LLM performance measurement. Results are comparable with other DGX Spark > # users who use the same tool. > # > # Benchmark profiles simulate real-world document analysis workloads: > # > # "Medium Log" (default) — 50-page document, pp2048 + tg128 @ depth 16384 > # "Massive Log" (--stress) — 100+ page log, pp2048 + tg128 @ depth 32768 > # "Extreme" (--extreme)— 200+ page corpus, pp2048 + tg128 @ depth 65535 > # > # Each profile includes a depth=0 baseline so you can see the performance > # delta as unified memory pressure increases. > # > # Usage: > # ./benchmark-models.sh # Medium Log (default) > # ./benchmark-models.sh --stress # Medium + Massive Log > # ./benchmark-models.sh --extreme # All three depth levels > # ./benchmark-models.sh --quick Nemotron # Fast smoke test > # ./benchmark-models.sh --runs 5 Qwen3.5-35B # Custom run count > # ./benchmark-models.sh Qwen3.5 Nemotron # Only matching models > # ============================================================================= > > set -euo pipefail > > LLAMA_SWAP_URL="${LLAMA_SWAP_URL:-http://localhost:28080}" > RESULTS_DIR="$(dirname "$(readlink -f "$0")")/test-results/benchmarks" > TIMEOUT=1800 > > # Colors > GREEN='\033[0;32m' > RED='\033[0;31m' > YELLOW='\033[1;33m' > CYAN='\033[0;36m' > BOLD='\033[1m' > DIM='\033[2m' > NC='\033[0m' > > # Defaults: "Medium Log" baseline > PP="2048" > TG="128" > DEPTH="0 16384" > RUNS=3 > MODE="medium-log" > FILTERS=() > > # Parse arguments > while [[$# -gt 0]]; do > case "$1" in > --quick) > PP="512" > TG="128" > DEPTH="0" > RUNS=1 > MODE="quick" > shift > ;; > --stress) > # Medium Log + Massive Log > PP="2048" > TG="128" > DEPTH="0 16384 32768" > RUNS=3 > MODE="stress" > shift > ;; > --extreme) > # All three: Medium + Massive + Extreme limit > PP="2048" > TG="128" > DEPTH="0 16384 32768 65535" > RUNS=3 > MODE="extreme" > shift > ;; > --full) > # Comprehensive sweep (original broad test) > PP="512 2048" > TG="128 256 512" > DEPTH="0 16384 32768" > RUNS=3 > MODE="full" > shift > ;; > --runs) > RUNS="$2" > shift 2 > ;; > --runs=*) > RUNS="${1#*=}" > shift > ;; > --help|-h) > cat <<'HELPEOF' > Usage: benchmark-models.sh [OPTIONS] [FILTER...] > > Benchmark your llama-swap models using llama-benchy. > Results use the standard llama-benchy format for comparison > with other DGX Spark users on the NVIDIA forums. > > Profiles (simulating real-world document workloads): > > (default) "Medium Log" — pp2048, tg128 @ depth 0 + 16384 > Simulates a ~50-page document. The depth=0 baseline shows > raw speed; depth=16384 shows the cost of a full KV cache. > > --stress "Massive Log" — adds depth 32768 > Doubles the context to simulate a massive error log. > Watch for tg tok/s drop vs. the medium baseline — that's > where shared memory bandwidth starts to bottleneck. > > --extreme "Extreme Limit" — adds depth 65535 > Pushes to ~100 pages of text. Strictly to see if the > system can process it without crashing or heavy swap. > > --quick Smoke test — pp512, tg128, depth 0, 1 run > --full Broad sweep — pp512+2048, tg128+256+512, depths 0-32k > > Other options: > --runs N Override number of runs (default: 3) > --help Show this help > > Filters: > Add model name fragments to only test matching models. > Example: ./benchmark-models.sh --stress Qwen3.5 Nemotron > > Environment: > LLAMA_SWAP_URL llama-swap endpoint (default: http://localhost:28080) > > What the numbers mean: > pp tok/s = Prompt Processing speed. How fast the model reads your input. > Higher is better. Typically 500-5000+ tok/s on DGX Spark. > tg tok/s = Token Generation speed. How fast the model writes its reply. > Higher is better. This is the number you "feel" when chatting. > Typically 15-50+ tok/s on DGX Spark depending on model size. > TTFT = Time To First Token. The delay before the model starts replying. > Lower is better. Measured in milliseconds. > > Key insight: > Compare tg tok/s across depths. A big drop from depth=16384 to depth=32768 > means you've found the unified memory bandwidth bottleneck between the ARM > CPU and the Blackwell GPU on DGX Spark. > HELPEOF > exit 0 > ;; > *) > FILTERS+=("$1") > shift > ;; > esac > done > > mkdir -p "$RESULTS_DIR" > TIMESTAMP=$(date +%Y%m%d_%H%M%S) > REPORT_FILE="$RESULTS_DIR/report_${TIMESTAMP}.txt" > > log() { echo -e "$1" | tee -a "$REPORT_FILE"; } > > unload_all() { > curl -sf -X POST "$LLAMA_SWAP_URL/unload" > /dev/null 2>&1 || true > sleep 5 > } > > # Warm up: send a tiny request to make llama-swap load the model > warmup_model() { > local model="$1" > log " Loading model via llama-swap..." > local start end elapsed > start=$(date +%s.%N) > > local response > response=$(curl -s --max-time "$TIMEOUT" \ > -X POST "$LLAMA_SWAP_URL/v1/chat/completions" \ > -H "Content-Type: application/json" \ > -d "$(jq -n --arg model "$model" '{ > model: $model, > messages: [{role: "user", content: "Say OK"}], > max_tokens: 5 > }')" 2>&1) || true > > end=$(date +%s.%N) > elapsed=$(echo "scale=1; $end - $start" | bc) > > local err > err=$(echo "$response" | jq -r '.error.message // empty' 2>/dev/null) > if [[-n "$err"]]; then > log " ${RED}FAILED to load: $err${NC}" > return 1 > fi > > log " Model ready (loaded in ${elapsed}s)" > return 0 > } > > # Run llama-benchy and capture results > run_benchy() { > local model="$1" > local safe_name="${model//\//_}" > local json_file="$RESULTS_DIR/${safe_name}_${TIMESTAMP}.json" > local md_file="$RESULTS_DIR/${safe_name}_${TIMESTAMP}.md" > > # Show profile description > case "$MODE" in > medium-log) log " Profile: Medium Log (50-page document baseline)" ;; > stress) log " Profile: Massive Log (stress test — watch for bandwidth bottleneck)" ;; > extreme) log " Profile: Extreme Limit (push to ~100 pages, crash/swap detection)" ;; > quick) log " Profile: Quick smoke test" ;; > full) log " Profile: Full comprehensive sweep" ;; > esac > log " Running llama-benchy (pp=$PP tg=$TG depth=$DEPTH runs=$RUNS)..." > log "" > > # --- Run 1: Save JSON for data parsing --- > local cmd_json="uvx llama-benchy" > cmd_json+=" --base-url $LLAMA_SWAP_URL/v1" > cmd_json+=" --model $model" > cmd_json+=" --pp $PP" > cmd_json+=" --tg $TG" > cmd_json+=" --depth $DEPTH" > cmd_json+=" --runs $RUNS" > cmd_json+=" --latency-mode generation" > cmd_json+=" --no-warmup" > cmd_json+=" --skip-coherence" > cmd_json+=" --save-result ${json_file}" > cmd_json+=" --format json" > > local output exit_code=0 > output=$(eval "$cmd_json" 2>&1) || exit_code=$? > > if [[$exit_code -ne 0]]; then > log " ${RED}llama-benchy failed:${NC}" > echo "$output" | tail -15 | tee -a "$REPORT_FILE" > return 1 > fi > > # --- Run 2: Get the markdown table (for sharing on forums) --- > local cmd_md="uvx llama-benchy" > cmd_md+=" --base-url $LLAMA_SWAP_URL/v1" > cmd_md+=" --model $model" > cmd_md+=" --pp $PP" > cmd_md+=" --tg $TG" > cmd_md+=" --depth $DEPTH" > cmd_md+=" --runs $RUNS" > cmd_md+=" --latency-mode generation" > cmd_md+=" --no-warmup" > cmd_md+=" --skip-coherence" > cmd_md+=" --save-result ${md_file}" > cmd_md+=" --format md" > > local md_output > md_output=$(eval "$cmd_md" 2>&1) || true > > # Show the full saved markdown file (the format people share on forums) > if [[-f "$md_file"]]; then > log " ${CYAN}llama-benchy results (copy this to share on forums):${NC}" > log "" > cat "$md_file" | tee -a "$REPORT_FILE" > log "" > else > # Fallback: show table lines from stdout if file wasn't created > local table_lines > table_lines=$(echo "$md_output" | grep -E '^\|') > if [[-n "$table_lines"]]; then > log " ${CYAN}llama-benchy results:${NC}" > log "" > echo "$table_lines" | tee -a "$REPORT_FILE" > log "" > fi > fi > > # --- Parse JSON and show friendly explanation --- > if [[-f "$json_file"]]; then > python3 < import json > > with open('$json_file') as f: > data = json.load(f) > > benchmarks = data.get('benchmarks', []) > if not benchmarks: > print(" (no benchmark data found)") > exit(0) > > # Check if all benchmarks have null results (model failed silently) > all_null = all( > b.get('pp_throughput') is None and b.get('tg_throughput') is None > for b in benchmarks > ) > if all_null: > print(" (model returned no usable results — it may not support this benchmark)") > exit(0) > > # Group benchmarks by depth for comparison > depth_results = {} > for b in benchmarks: > depth = b.get('context_size', 0) > pp_obj = b.get('pp_throughput') or {} > tg_obj = b.get('tg_throughput') or {} > pk_obj = b.get('peak_throughput') or {} > e2e_obj = b.get('e2e_ttft') or {} > pp_mean = pp_obj.get('mean', 0) or 0 > tg_mean = tg_obj.get('mean', 0) or 0 > if pp_mean > 0 or tg_mean > 0: > depth_results[depth] = { > 'pp_mean': pp_mean, 'pp_std': (pp_obj.get('std', 0) or 0), > 'tg_mean': tg_mean, 'tg_std': (tg_obj.get('std', 0) or 0), > 'pk_mean': (pk_obj.get('mean', 0) or 0), > 'e2e_mean': (e2e_obj.get('mean', 0) or 0), > } > > # Depth label mapping > depth_labels = { > 0: "Baseline (no context)", > 16384: "Medium Log (~50 pages)", > 32768: "Massive Log (~100 pages)", > 65535: "Extreme Limit (~200 pages)", > } > > print(" Context Depth Analysis:") > print("") > > baseline_tg = None > for depth in sorted(depth_results.keys()): > r = depth_results[depth] > label = depth_labels.get(depth, f"depth {depth}") > pp_mean, pp_std = r['pp_mean'], r['pp_std'] > tg_mean, tg_std = r['tg_mean'], r['tg_std'] > pk_mean = r['pk_mean'] > e2e_mean = r['e2e_mean'] > > print(f" --- depth={depth:,} — {label} ---") > > # Reading speed > if pp_mean > 0: > if pp_std > 0.5: > print(f" Reading speed: {pp_mean:,.0f} +/- {pp_std:,.0f} tok/s") > else: > print(f" Reading speed: {pp_mean:,.0f} tok/s") > > # Writing speed > if tg_mean > 0: > if tg_std > 0.5: > print(f" Writing speed: {tg_mean:.1f} +/- {tg_std:.1f} tok/s (peak: {pk_mean:.0f})") > else: > print(f" Writing speed: {tg_mean:.1f} tok/s (peak: {pk_mean:.0f})") > > # Show degradation from baseline > if baseline_tg is None: > baseline_tg = tg_mean > elif baseline_tg > 0: > pct = ((tg_mean - baseline_tg) / baseline_tg) * 100 > if pct < -15: > print(f" {chr(9888)} {pct:+.1f}% vs baseline — BANDWIDTH BOTTLENECK DETECTED") > elif pct < -5: > print(f" -> {pct:+.1f}% vs baseline (moderate slowdown)") > else: > print(f" -> {pct:+.1f}% vs baseline (minimal impact)") > > # What it feels like > if tg_mean >= 40: > feel = "Very fast — feels instant, smooth streaming" > elif tg_mean >= 25: > feel = "Fast — comfortable for interactive chat" > elif tg_mean >= 15: > feel = "Good — readable streaming with slight pauses" > elif tg_mean >= 8: > feel = "Moderate — noticeable wait, but usable" > else: > feel = "Slow — may feel sluggish for chat" > print(f" -> {feel}") > > # TTFT > if e2e_mean > 0: > print(f" Time to first token: {e2e_mean:.0f}ms") > if e2e_mean < 200: > print(f" -> Feels instant") > elif e2e_mean < 500: > print(f" -> Barely noticeable delay") > elif e2e_mean < 2000: > print(f" -> Short pause before response starts") > else: > print(f" -> Noticeable wait ({e2e_mean/1000:.1f}s)") > print("") > > # Summary comparison if multiple depths > if len(depth_results) > 1 and baseline_tg and baseline_tg > 0: > max_depth = max(depth_results.keys()) > deepest_tg = depth_results[max_depth]['tg_mean'] > total_pct = ((deepest_tg - baseline_tg) / baseline_tg) * 100 > print(f" Overall impact: depth 0 -> {max_depth:,} = {total_pct:+.1f}% generation speed") > if total_pct < -20: > print(f" Conclusion: Significant unified memory bandwidth bottleneck at depth {max_depth:,}") > elif total_pct < -10: > print(f" Conclusion: Moderate bandwidth pressure — usable but noticeably slower") > else: > print(f" Conclusion: Model handles deep context well on this hardware") > print("") > > PYEOF > fi > > log " ${DIM}JSON data : $json_file${NC}" > log " ${DIM}Forum table: $md_file${NC}" > return 0 > } > > # --------------- MAIN --------------- > log "" > log "${BOLD}============================================================${NC}" > log "${BOLD} DGX Spark Model Benchmark${NC}" > log "${BOLD} powered by llama-benchy (github.com/eugr/llama-benchy)${NC}" > log "${BOLD}============================================================${NC}" > log "" > log " Endpoint : $LLAMA_SWAP_URL" > log " Mode : $MODE" > log " Settings : pp=$PP tg=$TG depth=$DEPTH runs=$RUNS" > log " Date : $(date '+%Y-%m-%d %H:%M')" > log " Results : $RESULTS_DIR" > log "" > > # Mode descriptions > case "$MODE" in > medium-log) > log " ${CYAN}Profile: Medium Log Baseline${NC}" > log " ${DIM}Simulates a ~50-page document (depth=16384). Establishes your${NC}" > log " ${DIM}baseline generation speed with a moderately full KV cache.${NC}" > ;; > stress) > log " ${CYAN}Profile: Massive Log Stress Test${NC}" > log " ${DIM}Doubles context to simulate a massive error log (depth=32768).${NC}" > log " ${DIM}Watch for tg tok/s drop — that's the unified memory bottleneck.${NC}" > ;; > extreme) > log " ${CYAN}Profile: Extreme Limit Test${NC}" > log " ${DIM}Pushes to ~200 pages (depth=65535). Tests if the system can${NC}" > log " ${DIM}process it without crashing or heavy swap paging.${NC}" > ;; > quick) > log " ${DIM}Quick smoke test — just checking if models respond.${NC}" > ;; > full) > log " ${DIM}Full comprehensive sweep — broad pp/tg/depth combinations.${NC}" > ;; > esac > log "" > log " ${DIM}pp = prompt processing (how fast the model reads your input)${NC}" > log " ${DIM}tg = token generation (how fast the model writes its reply)${NC}" > log " ${DIM}depth = pre-filled context tokens (simulates document size)${NC}" > log "" > > # Check llama-benchy is available > if ! uvx llama-benchy --help > /dev/null 2>&1; then > log "${RED}Error: llama-benchy not available via uvx.${NC}" > log "Install with: pip install llama-benchy OR uv pip install llama-benchy" > exit 1 > fi > > # Fetch model list > MODELS=$(curl -sf "$LLAMA_SWAP_URL/v1/models" | jq -r '.data[].id' | sort) > MODEL_COUNT=$(echo "$MODELS" | wc -l) > > # Apply filters > if [[${#FILTERS[@]} -gt 0 ]]; then > log "Filtering models matching: ${FILTERS[*]}" > FILTERED="" > for m in $MODELS; do > for f in "${FILTERS[@]}"; do > if [["$m" == *"$f"*]]; then > FILTERED="${FILTERED}${m}\n" > fi > done > done > MODELS=$(echo -e "$FILTERED" | grep -v '^$' | sort -u) > MODEL_COUNT=$(echo "$MODELS" | wc -l) > fi > > log "Found ${BOLD}${MODEL_COUNT}${NC} model(s) to benchmark." > log "" > > PASS=0 > FAIL=0 > IDX=0 > TOTAL_START=$(date +%s.%N) > > # Collect results for final summary > declare -A SUMMARY_PP SUMMARY_TG SUMMARY_PEAK SUMMARY_TTFT SUMMARY_STATUS SUMMARY_DEGRADATION > > for MODEL in $MODELS; do > IDX=$((IDX + 1)) > log "${BOLD}============================================================${NC}" > log "${BOLD} [$IDX/$MODEL_COUNT] $MODEL${NC}" > log "${BOLD}============================================================${NC}" > log "" > > # Unload previous model to get a clean measurement > unload_all > > # Load this model > if ! warmup_model "$MODEL"; then > FAIL=$((FAIL + 1)) > SUMMARY_STATUS[$MODEL]="FAIL" > log "" > continue > fi > > # Benchmark it > if run_benchy "$MODEL"; then > PASS=$((PASS + 1)) > SUMMARY_STATUS[$MODEL]="OK" > > # Extract numbers for final summary from JSON > local_json="$RESULTS_DIR/${MODEL//\//_}_${TIMESTAMP}.json" > if [[-f "$local_json"]]; then > eval "$(python3 < import json > with open('$local_json') as f: > data = json.load(f) > > # Collect baseline (depth=0) and deepest results > baseline_tg = None > deepest_tg = None > max_depth = 0 > > for b in data.get('benchmarks', []): > depth = b.get('context_size', 0) > pp = b.get('pp_throughput') or {} > tg = b.get('tg_throughput') or {} > pk = b.get('peak_throughput') or {} > e2e = b.get('e2e_ttft') or {} > pp_mean = pp.get('mean', 0) or 0 > pp_std = pp.get('std', 0) or 0 > tg_mean = tg.get('mean', 0) or 0 > tg_std = tg.get('std', 0) or 0 > pk_mean = pk.get('mean', 0) or 0 > e2e_mean = e2e.get('mean', 0) or 0 > > if pp_mean == 0 and tg_mean == 0: > continue > > if depth == 0: > baseline_tg = tg_mean > # Use baseline for summary row > if pp_std > 0.5: > print(f"SUMMARY_PP['$MODEL']='{pp_mean:.0f} +/-{pp_std:.0f}'") > else: > print(f"SUMMARY_PP['$MODEL']='{pp_mean:.0f}'") > if tg_std > 0.5: > print(f"SUMMARY_TG['$MODEL']='{tg_mean:.1f} +/-{tg_std:.1f}'") > else: > print(f"SUMMARY_TG['$MODEL']='{tg_mean:.1f}'") > print(f"SUMMARY_PEAK['$MODEL']='{pk_mean:.0f}'") > if e2e_mean > 0: > print(f"SUMMARY_TTFT['$MODEL']='{e2e_mean:.0f}'") > > if depth > max_depth: > max_depth = depth > deepest_tg = tg_mean > > # Calculate degradation > if baseline_tg and deepest_tg and baseline_tg > 0 and max_depth > 0: > pct = ((deepest_tg - baseline_tg) / baseline_tg) * 100 > print(f"SUMMARY_DEGRADATION['$MODEL']='{pct:+.1f}% @{max_depth//1024}k'") > > PYEOF > )" 2>/dev/null > fi > else > FAIL=$((FAIL + 1)) > SUMMARY_STATUS[$MODEL]="FAIL" > fi > log "" > done > > # Final cleanup > unload_all > > TOTAL_END=$(date +%s.%N) > TOTAL_ELAPSED=$(echo "$TOTAL_END - $TOTAL_START" | bc) > TOTAL_MIN=$(echo "scale=1; $TOTAL_ELAPSED / 60" | bc) > > # ============================================= > # FINAL REPORT > # ============================================= > log "" > log "${BOLD}============================================================${NC}" > log "${BOLD} BENCHMARK REPORT — DGX Spark${NC}" > log "${BOLD}============================================================${NC}" > log "" > log " Date: $(date '+%Y-%m-%d %H:%M') | Mode: $MODE | Runs: $RUNS" > log " Depths tested: $DEPTH" > log " Models tested: $MODEL_COUNT | Passed: ${GREEN}$PASS${NC} Failed: ${RED}$FAIL${NC}" > log " Total benchmark time: ${TOTAL_MIN} min" > log "" > log " ${BOLD}$(printf '%-42s %14s %12s %8s %8s %14s' 'Model' 'Read (pp)' 'Write (tg)' 'Peak' 'TTFT' 'Deep ctx')${NC}" > log " ${DIM}$(printf '%-42s %14s %12s %8s %8s %14s' '' 'tok/s' 'tok/s' 'tok/s' 'ms' 'degradation')${NC}" > log " $(printf '%.0s-' {1..106})" > > for MODEL in $MODELS; do > local_name="$MODEL" > [[${#local_name} -gt 42]] && local_name="${local_name:0:39}..." > > status="${SUMMARY_STATUS[$MODEL]:-FAIL}" > pp="${SUMMARY_PP[$MODEL]:-—}" > tg="${SUMMARY_TG[$MODEL]:-—}" > peak="${SUMMARY_PEAK[$MODEL]:-—}" > ttft="${SUMMARY_TTFT[$MODEL]:-—}" > degrad="${SUMMARY_DEGRADATION[$MODEL]:-—}" > > if [["$status" == "OK"]]; then > printf -v line " %-42s %14s %12s %8s %8s %14s" "$local_name" "$pp" "$tg" "$peak" "$ttft" "$degrad" > log "${GREEN}${line}${NC}" > else > printf -v line " %-42s %14s %12s %8s %8s %14s" "$local_name" "FAIL" "—" "—" "—" "—" > log "${RED}${line}${NC}" > fi > done > > log "" > log " ${DIM}---------------------------------------------------------------${NC}" > log " ${DIM}How to read this table:${NC}" > log " ${DIM}${NC}" > log " ${DIM} Read (pp) = How fast the model reads your prompt (higher = better).${NC}" > log " ${DIM} Write (tg) = How fast the model types its answer at depth=0 baseline.${NC}" > log " ${DIM} This is the speed you feel when chatting.${NC}" > log " ${DIM} Humans read at ~4 tok/s, so 20+ feels smooth.${NC}" > log " ${DIM} Peak = Fastest burst speed observed in a 1-second window.${NC}" > log " ${DIM} TTFT = Time until the first word appears (lower = better).${NC}" > log " ${DIM} Deep ctx = Speed change at max tested depth vs baseline.${NC}" > log " ${DIM} >-15% = unified memory bandwidth bottleneck.${NC}" > log " ${DIM}---------------------------------------------------------------${NC}" > log "" > log " Results saved to:" > log " Report : $REPORT_FILE" > log " JSON data : $RESULTS_DIR/*_${TIMESTAMP}.json" > log " Forum tables: $RESULTS_DIR/*_${TIMESTAMP}.md" > log "" > log " ${DIM}Tip: To share on NVIDIA forums, copy the llama-benchy tables${NC}" > log " ${DIM}from the .md files — they use the standard format everyone knows.${NC}" > log "${BOLD}============================================================${NC}" > > ``` **What test settings make the most sense.. on a single spark?** would be nice to have something like a benchmark tool that sends the results to a database and where you can see if your results are good or bad compared to others with the same hardware. Similar to “3D Mark” back in the days… ---
**Author:** ![eugr](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/eugr/32/449615_2.png) [@eugr](https://forums.developer.nvidia.com/u/eugr)\ **Post date:** [April 15, 2026, 11:10pm UTC](https://forums.developer.nvidia.com/t/new-tool-llama-benchy-llama-bench-style-benchmarking-for-any-llm-backend-vllm-sglang-llama-cpp-etc/356698/12 "2026-04-15T23:10:23Z")
> [@martinB78](#): > > would be nice to have something like a benchmark tool that sends the results to a database and where you can see if your results are good or bad compared to others with the same hardware. Look at [https://spark-arena.com/](https://spark-arena.com/) :) ---
**Author:** ![martinB78](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/martinb78/32/437495_2.png) [@martinB78](https://forums.developer.nvidia.com/u/martinB78)\ **Post date:** [April 19, 2026, 11:14am UTC](https://forums.developer.nvidia.com/t/new-tool-llama-benchy-llama-bench-style-benchmarking-for-any-llm-backend-vllm-sglang-llama-cpp-etc/356698/13 "2026-04-19T11:14:14Z")
@eugr nice tip! I’ve been busy since then. The script grew quite a bit — sharing the updated version. I see you got your own Model loader [sparkrun](https://github.com/spark-arena/sparkrun) which looks cool - but its incompatible with my current setup. I like the auto load feature of lama-swap so it does not matter what I use (Visual Studio Code, Open Web UI, Openclaw…) I just pick the model I like to use in that software and lama-swap unloads the current one and loads the one I selected in the app. Without me having to manually run a command and load the model. Still I would like to participate in the [leaderboard](https://spark-arena.com/leaderboard) and submit results. So I told the AI to update the benchmark-models.sh to run the benchmarks as they are required for the Leader board and save the settings and results so they can be submitted. And to not flood the leader board with slow results check if there is a faster one already and if that is the case skip the submission. Also I noticed sometimes the model is Fast but just spits out nonsense “!!!” so I added a little Coherence check where I can see a part of the output to check if that model is working or not. ````auto Coherence check: "Here's a classic "Hello, World!" script in Python:```pythonprint("Hello, World!")```Run it by saving it as `hello.py` and executing in your terminal:`" ```` **everything from here on is AI generated** **What’s new in benchmark-models.sh since the last post:** * * * **Coherence check before benchmarking** The script now does a warmup request and checks the response before wasting time running llama-benchy. It detects two failure modes: - Empty response → `FAIL` - Repetition loop (e.g. model stuck outputting `n8n n8n n8n...`) → `INCOHERENT` Both get skipped cleanly in the summary table — yellow for INCOHERENT, red for FAIL, so you can tell at a glance whether a model crashed or just went degenerate. * * * **More benchmark profiles** ```auto ./benchmark-models.sh # Medium Log (default) — depth 0 + 16384 ./benchmark-models.sh --stress # + depth 32768 ./benchmark-models.sh --extreme # + depth 65535 ./benchmark-models.sh --quick Nemotron # Fast smoke test (pp512, 1 run) ./benchmark-models.sh --runs 5 Qwen3 # Custom run count ``` * * * **`--arena` mode — [spark-arena.com](http://spark-arena.com) submission files** Since eugr pointed me at [spark-arena.com](http://spark-arena.com), I added a `--arena` flag that runs the exact leaderboard benchmark spec and generates the submission files automatically: ```auto ./benchmark-models.sh --arena Qwen3-Coder-Next-int4-AutoRound ``` This runs the official profile: - depths: `0 4096 8192 16384 32768 65535 100000` - concurrency: `1 2 5 10` - prefix caching enabled And outputs per model in `test-results/arena-submission///`: - **`results.csv`** — llama-benchy output in the arena CSV format - **`recipe.yaml`** — pre-filled spark-arena recipe YAML with correct container, env vars, quantization flags etc. The script also tracks personal bests in `arena-best-results.json`. After each run it compares — if it’s a new best, it auto-downloads `spark-arena-cli` and prints the exact submit command: ```auto ★ NEW PERSONAL BEST — 66.3 tg tok/s (was 60.4) spark-arena-cli benchmark test-results/arena-submission/.../recipe.yaml ``` * * * **Latest numbers on a single DGX Spark (GB10 Blackwell)** All vLLM unless noted. Depth=0, concurrency=1, 3 runs. | Model | tg tok/s | pp tok/s | TTFT | Deep ctx (-@16k) | | --- | --- | --- | --- | --- | | Qwen3-Coder-Next-int4-AutoRound | **66.3** | 3,268 | 644ms | -3.6% | | Qwen3.5-35B Uncensored Q4\_K\_M (llama.cpp) | 56.8 | 1,974 | 1160ms | — | | Qwen3-VL-30B-FP8 | 52.5 | 5,827 | 508ms | -12.2% | | Nemotron-3-Nano-30B NVFP4 | 44.0 | 3,529 | 290ms | — | | Nemotron-3-Nano-4B FP8 | 40.0 | 9,677 | 340ms | — | | GPT-OSS-120B MXFP4 | 33.1 | 4,624 | 595ms | -8.4% | | Qwen3-Coder-Next-FP8-Dynamic | 32.6 | 1,835 | 1278ms | -0.4% | | Qwen3-Omni-30B | 30.6 | 2,807 | 584ms | -15.0% ⚠️ | | Qwen3.5-122B int4-AutoRound | 25.6 | 1,760 | 1210ms | -2.3% | | Nemotron-3-Super-120B NVFP4 | 14.8 | 1,725 | 1221ms | -1.6% | **Qwen3-Coder-Next-int4-AutoRound** is the clear speed king at 66 tg tok/s — nearly double the FP8 variant of the same model (32 tok/s). The int4-AutoRound quantization on the 480B MoE actually outperforms everything else in generation speed, which surprised me. The 30B Omni model hit the -15% threshold at depth 16k — that’s the unified memory bandwidth bottleneck showing up. **It happens that the DGX Spark crashes or reboots during a test so there is a resume function for that.** **`benchmark-models.sh --resume`** : - Every session writes a checkpoint to `test-results/checkpoints/session_TIMESTAMP.json` - Each model is written to the checkpoint **before** warmup starts, so a crash mid-model is visible - `--resume` reads the last session, skips completed models, and tells you which model was in-progress when it crashed - The final report always shows the checkpoint path and a resume hint if any models failed * * * **Script is below and on [GitHub (in my stack repo)](https://github.com/mARTin-B78/dgx-spark_lite-llm_llama-swap_vllm_llama-cpp_ollama):** > **benchmark-models.sh** > > #!/bin/bash > > # ============================================================================= > > # benchmark-models.sh — Benchmark all llama-swap models using llama-benchy > > # > > # Uses llama-benchy ([GitHub - eugr/llama-benchy: llama-benchy - llama-bench style benchmarking tool for all backends · GitHub](https://github.com/eugr/llama-benchy)) for standardized > > # LLM performance measurement. Results are comparable with other DGX Spark > > # users who use the same tool. > > # > > # Benchmark profiles simulate real-world document analysis workloads: > > # > > # “Medium Log” (default) — 50-page document, pp2048 + tg128 @ depth 16384 > > # “Massive Log” (–stress) — 100+ page log, pp2048 + tg128 @ depth 32768 > > # “Extreme” (–extreme)— 200+ page corpus, pp2048 + tg128 @ depth 65535 > > # > > # Each profile includes a depth=0 baseline so you can see the performance > > # delta as unified memory pressure increases. > > # > > # Usage: > > # ./benchmark-models.sh # Medium Log (default) > > # ./benchmark-models.sh --stress # Medium + Massive Log > > # ./benchmark-models.sh --extreme # All three depth levels > > # ./benchmark-models.sh --quick Nemotron # Fast smoke test > > # ./benchmark-models.sh --runs 5 Qwen3.5-35B # Custom run count > > # ./benchmark-models.sh Qwen3.5 Nemotron # Only matching models > > # ============================================================================= > > set -euo pipefail > > LLAMA\_SWAP\_URL=“${LLAMA\_SWAP\_URL:-[http://localhost:28080](http://localhost:28080)}” > SCRIPT\_DIR=“$(dirname “$(readlink -f “$0”)”)” > RESULTS\_DIR=“$SCRIPT\_DIR/test-results/benchmarks” > ARENA\_BEST\_FILE=“$SCRIPT\_DIR/test-results/arena-best-results.json” > TIMEOUT=1800 > > # spark-arena-cli: installed to ~/.local/bin on first --arena run > > SPARK\_CLI=“${HOME}/.local/bin/spark-arena-cli” > > # Colors > > GREEN=‘\033[0;32m’ > RED=‘\033[0;31m’ > YELLOW=‘\033[1;33m’ > CYAN=‘\033[0;36m’ > BOLD=‘\033[1m’ > DIM=‘\033[2m’ > NC=‘\033[0m’ > > # Defaults: “Medium Log” baseline > > PP=“2048” > TG=“128” > DEPTH=“0 16384” > RUNS=3 > MODE=“medium-log” > CONCURRENCY=“” # space-separated list; empty = not passed to llama-benchy > ARENA\_DIR=“” # set by --arena mode > FILTERS=() > > # Crash-resume tracking > > CHECKPOINT\_DIR=“$SCRIPT\_DIR/test-results/checkpoints” > LAST\_SESSION\_FILE=“$SCRIPT\_DIR/test-results/.last-session” > CHECKPOINT\_FILE=“” > RESUME=false > SKIP\_MODELS=() > > # Parse arguments > > while [[$# -gt 0]]; do > case “$1” in > –quick) > PP=“512” > TG=“128” > DEPTH=“0” > RUNS=1 > MODE=“quick” > shift > ;; > –stress) > # Medium Log + Massive Log > PP=“2048” > TG=“128” > DEPTH=“0 16384 32768” > RUNS=3 > MODE=“stress” > shift > ;; > –extreme) > # All three: Medium + Massive + Extreme limit > PP=“2048” > TG=“128” > DEPTH=“0 16384 32768 65535” > RUNS=3 > MODE=“extreme” > shift > ;; > –full) > # Comprehensive sweep (original broad test) > PP=“512 2048” > TG=“128 256 512” > DEPTH=“0 16384 32768” > RUNS=3 > MODE=“full” > shift > ;; > –arena) > # Spark-Arena leaderboard submission profile > # [Spark Arena - LLM Leaderboard](https://spark-arena.com/admin) > PP=“2048” > TG=“128” > DEPTH=“0 4096 8192 16384 32768 65535 100000” > CONCURRENCY=“1 2 5 10” > RUNS=3 > MODE=“arena” > shift > ;; > –runs) > RUNS=“$2” > shift 2 > ;; > –runs=_) > RUNS="${1#_=}" > shift > ;; > –resume) > RESUME=true > shift > ;; > –help|-h) > cat \<\<‘HELPEOF’ > Usage: benchmark-models.sh [OPTIONS] [FILTER…] > > Benchmark your llama-swap models using llama-benchy. > Results use the standard llama-benchy format for comparison > with other DGX Spark users on the NVIDIA forums. > > Profiles (simulating real-world document workloads): > > (default) “Medium Log” — pp2048, tg128 @ depth 0 + 16384 > Simulates a ~50-page document. The depth=0 baseline shows > raw speed; depth=16384 shows the cost of a full KV cache. > > –stress “Massive Log” — adds depth 32768 > Doubles the context to simulate a massive error log. > Watch for tg tok/s drop vs. the medium baseline — that’s > where shared memory bandwidth starts to bottleneck. > > –extreme “Extreme Limit” — adds depth 65535 > Pushes to ~100 pages of text. Strictly to see if the > system can process it without crashing or heavy swap. > > –quick Smoke test — pp512, tg128, depth 0, 1 run > –full Broad sweep — pp512+2048, tg128+256+512, depths 0-32k > –arena Spark-Arena leaderboard profile — exact spec from [Spark Arena - LLM Leaderboard](http://spark-arena.com/admin) > Saves results.csv + recipe.yaml per model to test-results/arena-submission/ > Depths: 0 4096 8192 16384 32768 65535 100000 | Concurrency: 1 2 5 10 > > Other options: > –runs N Override number of runs (default: 3) > –resume Resume from the last interrupted session (skip already-completed models) > –help Show this help > > Filters: > Add model name fragments to only test matching models. > Example: ./benchmark-models.sh --stress Qwen3.5 Nemotron > > Environment: > LLAMA\_SWAP\_URL llama-swap endpoint (default: [http://localhost:28080](http://localhost:28080)) > > What the numbers mean: > pp tok/s = Prompt Processing speed. How fast the model reads your input. > Higher is better. Typically 500-5000+ tok/s on DGX Spark. > tg tok/s = Token Generation speed. How fast the model writes its reply. > Higher is better. This is the number you “feel” when chatting. > Typically 15-50+ tok/s on DGX Spark depending on model size. > TTFT = Time To First Token. The delay before the model starts replying. > Lower is better. Measured in milliseconds. > > Key insight: > Compare tg tok/s across depths. A big drop from depth=16384 to depth=32768 > means you’ve found the unified memory bandwidth bottleneck between the ARM > CPU and the Blackwell GPU on DGX Spark. > HELPEOF > exit 0 > ;; > \*) > FILTERS+=(“$1”) > shift > ;; > esac > done > > mkdir -p “$RESULTS\_DIR” > TIMESTAMP=$(date +%Y%m%d\_%H%M%S) > REPORT\_FILE=“$RESULTS\_DIR/report\_${TIMESTAMP}.txt” > > if [[“$MODE” == “arena”]]; then > ARENA\_DIR=“$SCRIPT\_DIR/test-results/arena-submission/${TIMESTAMP}” > mkdir -p “$ARENA\_DIR” > fi > > log() { echo -e “$1” | tee -a “$REPORT\_FILE”; } > > unload\_all() { > curl -sf -X POST “$LLAMA\_SWAP\_URL/unload” \> /dev/null 2\>&1 || true > sleep 5 > } > > # Warm up: send a tiny request to make llama-swap load the model > > warmup\_model() { > local model=“$1” > log " Loading model via llama-swap…" > local start end elapsed > start=$(date +%s.%N) > > ``` > local response_file="/tmp/response_${TIMESTAMP}.json" > local http_code > http_code=$(curl -s -w "%{http_code}" -o "$response_file" --max-time "$TIMEOUT" \ > -X POST "$LLAMA_SWAP_URL/v1/chat/completions" \ > -H "Content-Type: application/json" \ > -d "$(jq -n --arg model "$model" '{ > model: $model, > messages: [{role: "user", content: "Write a short python hello world script."}], > max_tokens: 50 > }')" 2>/dev/null) || http_code=0 > > end=$(date +%s.%N) > elapsed=$(echo "scale=1; $end - $start" | bc) > > if [["$http_code" -ne 200]]; then > local err_msg > err_msg=$(jq -r '.error.message // empty' "$response_file" 2>/dev/null) > if [[-n "$err_msg"]]; then > log " ${RED}FAILED to load: $err_msg (HTTP $http_code)${NC}" > else > log " ${RED}FAILED to load (HTTP $http_code)${NC}" > fi > rm -f "$response_file" > return 1 > fi > > local content > content=$(jq -r '(.choices[0].message.reasoning_content // "") + (.choices[0].message.reasoning // "") + (.choices[0].message.content // "")' "$response_file" 2>/dev/null | tr -d '\n' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//') > rm -f "$response_file" > > if [[-z "$content"]]; then > log " ${CYAN}Coherence check:${NC} ${RED}FAILED — empty response${NC}" > WARMUP_FAIL_REASON="EMPTY" > return 1 > fi > > # Detect repetition loop: split into words, check if any single word > # makes up >60% of total words (e.g. "n8n n8n n8n..." or "the the the...") > local word_count most_freq_count most_freq_word > word_count=$(echo "$content" | wc -w) > if [["$word_count" -ge 5]]; then > most_freq_word=$(echo "$content" | tr ' ' '\n' | sort | uniq -c | sort -rn | awk 'NR==1{print $2}') > most_freq_count=$(echo "$content" | tr ' ' '\n' | grep -cFx "$most_freq_word" 2>/dev/null || echo 0) > local pct=$(( most_freq_count * 100 / word_count )) > if [["$pct" -ge 60]]; then > log " ${CYAN}Coherence check:${NC} ${RED}FAILED — repetition loop (\"${most_freq_word}\" = ${pct}% of output)${NC}" > log " ${RED}Skipping benchmark — model output is incoherent.${NC}" > WARMUP_FAIL_REASON="INCOHERENT" > return 1 > fi > fi > > log " ${CYAN}Coherence check:${NC} \"${content:0:150}\"" > log " Model ready (loaded in ${elapsed}s)" > WARMUP_FAIL_REASON="" > return 0 > > ``` > > } > > # Run llama-benchy and capture results > > run\_benchy() { > local model=“$1” > local safe\_name=“${model////_}" > local json\_file="$RESULTS\_DIR/${safe\_name}_${TIMESTAMP}.json” > local md\_file=“$RESULTS\_DIR/${safe\_name}\_${TIMESTAMP}.md” > > ``` > # Show profile description > case "$MODE" in > medium-log) log " Profile: Medium Log (50-page document baseline)" ;; > stress) log " Profile: Massive Log (stress test — watch for bandwidth bottleneck)" ;; > extreme) log " Profile: Extreme Limit (push to ~100 pages, crash/swap detection)" ;; > quick) log " Profile: Quick smoke test" ;; > full) log " Profile: Full comprehensive sweep" ;; > arena) log " Profile: Spark-Arena leaderboard (7 depths × 4 concurrency levels)" ;; > esac > local concurrency_display="" > [[-n "$CONCURRENCY"]] && concurrency_display=" concurrency=$CONCURRENCY" > log " Running llama-benchy (pp=$PP tg=$TG depth=$DEPTH runs=$RUNS${concurrency_display})..." > log "" > > # Build shared base flags (used by all runs) > local base_flags="" > base_flags+=" --base-url $LLAMA_SWAP_URL/v1" > base_flags+=" --model $model" > base_flags+=" --pp $PP" > base_flags+=" --tg $TG" > base_flags+=" --depth $DEPTH" > base_flags+=" --runs $RUNS" > base_flags+=" --latency-mode generation" > base_flags+=" --no-warmup" > base_flags+=" --skip-coherence" > [[-n "$CONCURRENCY"]] && base_flags+=" --concurrency $CONCURRENCY" > [["$MODE" == "arena"]] && base_flags+=" --enable-prefix-caching" > > # --- Run 1: Save JSON for data parsing --- > local cmd_json="uvx llama-benchy${base_flags} --save-result ${json_file} --format json" > > local output exit_code=0 > output=$(eval "$cmd_json" 2>&1) || exit_code=$? > > if [[$exit_code -ne 0]]; then > log " ${RED}llama-benchy failed:${NC}" > echo "$output" | tail -15 | tee -a "$REPORT_FILE" > return 1 > fi > > # --- Arena mode: save submission CSV (separate run, same params) --- > if [["$MODE" == "arena" && -n "$ARENA_DIR"]]; then > local arena_model_dir="$ARENA_DIR/${safe_name}" > mkdir -p "$arena_model_dir" > local csv_file="$arena_model_dir/results.csv" > local cmd_csv="uvx llama-benchy${base_flags} --save-result ${csv_file} --format csv" > log " ${CYAN}Saving arena submission CSV...${NC}" > local csv_output csv_exit=0 > csv_output=$(eval "$cmd_csv" 2>&1) || csv_exit=$? > if [[$csv_exit -ne 0]]; then > log " ${YELLOW}Warning: CSV run failed — JSON data still saved${NC}" > else > log " ${DIM}Arena CSV : $csv_file${NC}" > fi > fi > > # --- Run 2: Get the markdown table (for sharing on forums) --- > local cmd_md="uvx llama-benchy${base_flags} --save-result ${md_file} --format md" > > local md_output > md_output=$(eval "$cmd_md" 2>&1) || true > > # Show the full saved markdown file (the format people share on forums) > if [[-f "$md_file"]]; then > log " ${CYAN}llama-benchy results (copy this to share on forums):${NC}" > log "" > cat "$md_file" | tee -a "$REPORT_FILE" > log "" > else > # Fallback: show table lines from stdout if file wasn't created > local table_lines > table_lines=$(echo "$md_output" | grep -E '^\|') > if [[-n "$table_lines"]]; then > log " ${CYAN}llama-benchy results:${NC}" > log "" > echo "$table_lines" | tee -a "$REPORT_FILE" > log "" > fi > fi > > # --- Parse JSON and show friendly explanation --- > if [[-f "$json_file"]]; then > python3 < > ``` > > import json > > with open(‘$json\_file’) as f: > data = json.load(f) > > benchmarks = data.get(‘benchmarks’, ) > if not benchmarks: > print(" (no benchmark data found)") > exit(0) > > # Check if all benchmarks have null results (model failed silently) > > all\_null = all( > b.get(‘pp\_throughput’) is None and b.get(‘tg\_throughput’) is None > for b in benchmarks > ) > if all\_null: > print(" (model returned no usable results — it may not support this benchmark)") > exit(0) > > # Group benchmarks by depth for comparison > > depth\_results = {} > for b in benchmarks: > depth = b.get(‘context\_size’, 0) > pp\_obj = b.get(‘pp\_throughput’) or {} > tg\_obj = b.get(‘tg\_throughput’) or {} > pk\_obj = b.get(‘peak\_throughput’) or {} > e2e\_obj = b.get(‘e2e\_ttft’) or {} > pp\_mean = pp\_obj.get(‘mean’, 0) or 0 > tg\_mean = tg\_obj.get(‘mean’, 0) or 0 > if pp\_mean \> 0 or tg\_mean \> 0: > depth\_results[depth] = { > ‘pp\_mean’: pp\_mean, ‘pp\_std’: (pp\_obj.get(‘std’, 0) or 0), > ‘tg\_mean’: tg\_mean, ‘tg\_std’: (tg\_obj.get(‘std’, 0) or 0), > ‘pk\_mean’: (pk\_obj.get(‘mean’, 0) or 0), > ‘e2e\_mean’: (e2e\_obj.get(‘mean’, 0) or 0), > } > > # Depth label mapping > > depth\_labels = { > 0: “Baseline (no context)”, > 16384: “Medium Log (~50 pages)”, > 32768: “Massive Log (~100 pages)”, > 65535: “Extreme Limit (~200 pages)”, > } > > print(" Context Depth Analysis:“) > print(”") > > baseline\_tg = None > for depth in sorted(depth\_results.keys()): > r = depth\_results[depth] > label = depth\_labels.get(depth, f"depth {depth}") > pp\_mean, pp\_std = r[‘pp\_mean’], r[‘pp\_std’] > tg\_mean, tg\_std = r[‘tg\_mean’], r[‘tg\_std’] > pk\_mean = r[‘pk\_mean’] > e2e\_mean = r[‘e2e\_mean’] > > ``` > print(f" --- depth={depth:,} — {label} ---") > > # Reading speed > if pp_mean > 0: > if pp_std > 0.5: > print(f" Reading speed: {pp_mean:,.0f} +/- {pp_std:,.0f} tok/s") > else: > print(f" Reading speed: {pp_mean:,.0f} tok/s") > > # Writing speed > if tg_mean > 0: > if tg_std > 0.5: > print(f" Writing speed: {tg_mean:.1f} +/- {tg_std:.1f} tok/s (peak: {pk_mean:.0f})") > else: > print(f" Writing speed: {tg_mean:.1f} tok/s (peak: {pk_mean:.0f})") > > # Show degradation from baseline > if baseline_tg is None: > baseline_tg = tg_mean > elif baseline_tg > 0: > pct = ((tg_mean - baseline_tg) / baseline_tg) * 100 > if pct < -15: > print(f" {chr(9888)} {pct:+.1f}% vs baseline — BANDWIDTH BOTTLENECK DETECTED") > elif pct < -5: > print(f" -> {pct:+.1f}% vs baseline (moderate slowdown)") > else: > print(f" -> {pct:+.1f}% vs baseline (minimal impact)") > > # What it feels like > if tg_mean >= 40: > feel = "Very fast — feels instant, smooth streaming" > elif tg_mean >= 25: > feel = "Fast — comfortable for interactive chat" > elif tg_mean >= 15: > feel = "Good — readable streaming with slight pauses" > elif tg_mean >= 8: > feel = "Moderate — noticeable wait, but usable" > else: > feel = "Slow — may feel sluggish for chat" > print(f" -> {feel}") > > # TTFT > if e2e_mean > 0: > print(f" Time to first token: {e2e_mean:.0f}ms") > if e2e_mean < 200: > print(f" -> Feels instant") > elif e2e_mean < 500: > print(f" -> Barely noticeable delay") > elif e2e_mean < 2000: > print(f" -> Short pause before response starts") > else: > print(f" -> Noticeable wait ({e2e_mean/1000:.1f}s)") > print("") > > ``` > > # Summary comparison if multiple depths > > if len(depth\_results) \> 1 and baseline\_tg and baseline\_tg \> 0: > max\_depth = max(depth\_results.keys()) > deepest\_tg = depth\_results[max\_depth][‘tg\_mean’] > total\_pct = ((deepest\_tg - baseline\_tg) / baseline\_tg) \* 100 > print(f" Overall impact: depth 0 → {max\_depth:,} = {total\_pct:+.1f}% generation speed") > if total\_pct \< -20: > print(f" Conclusion: Significant unified memory bandwidth bottleneck at depth {max\_depth:,}“) > elif total\_pct \< -10: > print(f” Conclusion: Moderate bandwidth pressure — usable but noticeably slower") > else: > print(f" Conclusion: Model handles deep context well on this hardware") > print(“”) > > PYEOF > fi > > ``` > log " ${DIM}JSON data : $json_file${NC}" > log " ${DIM}Forum table: $md_file${NC}" > return 0 > > ``` > > } > > # --------------------------------------------------------------------------- > > # spark-arena-cli helpers > > # --------------------------------------------------------------------------- > > # Download spark-arena-cli binary if it isn’t already on PATH / ~/.local/bin > > install\_spark\_arena\_cli() { > if command -v spark-arena-cli &\>/dev/null; then > SPARK\_CLI=“$(command -v spark-arena-cli)” > return 0 > fi > if [[-x “$SPARK\_CLI”]]; then > return 0 > fi > local arch > arch=$(uname -m) > local bin\_name > [[“$arch” == “aarch64” || “$arch” == “arm64”]] > && bin\_name=“spark-arena-cli-0.1.0-linux-arm64” > || bin\_name=“spark-arena-cli-0.1.0-linux-amd64” > local url=“[https://github.com/spark-arena/spark-arena-cli/releases/download/v0.1.0/${bin\_name}](https://github.com/spark-arena/spark-arena-cli/releases/download/v0.1.0/$%7Bbin_name%7D)” > log " ${DIM}Downloading spark-arena-cli from GitHub releases…${NC}" > mkdir -p “$(dirname “$SPARK\_CLI”)” > if curl -fsSL “$url” -o “$SPARK\_CLI” 2\>/dev/null; then > chmod +x “$SPARK\_CLI” > log " ${DIM}Installed to $SPARK\_CLI${NC}" > # Add to PATH for this session > export PATH=“$(dirname “$SPARK\_CLI”):$PATH” > else > log " ${YELLOW}Warning: could not download spark-arena-cli — manual install needed${NC}" > SPARK\_CLI=“” > fi > } > > # Read our personal best tg tok/s (depth=0, concurrency=1) for a model from the history file > > get\_personal\_best\_tg() { > local model=“$1” > if [[! -f “$ARENA\_BEST\_FILE”]]; then echo “0”; return; fi > python3 -c " > import json, sys > try: > d = json.load(open(‘$ARENA\_BEST\_FILE’)) > entry = d.get(‘$model’, {}) > print(entry.get(‘tg\_mean’, 0)) > except: > print(0) > " 2\>/dev/null || echo “0” > } > > # Save current result as personal best for a model > > save\_personal\_best() { > local model=“$1” > local json\_file=“$2” > python3 - “$model” “$json\_file” “$ARENA\_BEST\_FILE” \<\<‘PYEOF’ > import json, sys, os > model, bench\_json, best\_file = sys.argv[1], sys.argv[2], sys.argv[3] > > try: > data = json.load(open(bench\_json)) > except Exception as e: > sys.exit(0) > > # Find depth=0, concurrency=1 entry > > baseline = None > for b in data.get(“benchmarks”, ): > if b.get(“context\_size”) == 0 and b.get(“concurrency”) == 1: > baseline = b > break > if not baseline: > # Fallback: first entry with context\_size=0 > for b in data.get(“benchmarks”, ): > if b.get(“context\_size”) == 0: > baseline = b > break > if not baseline: > sys.exit(0) > > tg = (baseline.get(“tg\_throughput”) or {}).get(“mean”, 0) or 0 > pp = (baseline.get(“pp\_throughput”) or {}).get(“mean”, 0) or 0 > e2e = (baseline.get(“e2e\_ttft”) or {}).get(“mean”, 0) or 0 > > try: > best = json.load(open(best\_file)) if os.path.exists(best\_file) else {} > except: > best = {} > > best[model] = { > “tg\_mean”: round(tg, 2), > “pp\_mean”: round(pp, 1), > “ttft\_ms”: round(e2e, 1), > “timestamp”: data.get(“timestamp”, “”), > “depth”: 0, > “concurrency”: 1, > } > with open(best\_file, “w”) as f: > json.dump(best, f, indent=2) > PYEOF > } > > # Compare current result vs personal best; if better, print submission info > > check\_and\_suggest\_submit() { > local model=“$1” > local json\_file=“$2” > local recipe\_file=“$3” > > ``` > local prev_best > prev_best=$(get_personal_best_tg "$model") > > local current_tg > current_tg=$(python3 -c " > > ``` > > import json > data = json.load(open(‘$json\_file’)) > for b in data.get(‘benchmarks’, ): > if b.get(‘context\_size’) == 0 and b.get(‘concurrency’) == 1: > tg = (b.get(‘tg\_throughput’) or {}).get(‘mean’, 0) or 0 > print(round(tg, 2)) > exit() > for b in data.get(‘benchmarks’, ): > if b.get(‘context\_size’) == 0: > tg = (b.get(‘tg\_throughput’) or {}).get(‘mean’, 0) or 0 > print(round(tg, 2)) > exit() > print(0) > " 2\>/dev/null || echo “0”) > > ``` > local is_better=0 > python3 -c "exit(0 if float('$current_tg') > float('$prev_best') else 1)" 2>/dev/null && is_better=1 > > if [["$is_better" -eq 1]]; then > if python3 -c "exit(0 if float('$prev_best') > 0 else 1)" 2>/dev/null; then > log " ${GREEN}★ NEW PERSONAL BEST${NC} — ${current_tg} tg tok/s (was ${prev_best})" > else > log " ${GREEN}★ FIRST ARENA RESULT${NC} — ${current_tg} tg tok/s @ depth=0 concurrency=1" > fi > save_personal_best "$model" "$json_file" > > # Try to get leaderboard context (best-effort scrape — may be empty) > log " ${CYAN}Submission candidate${NC} — consider submitting to spark-arena.com/leaderboard" > log "" > > install_spark_arena_cli > > if [[-n "$SPARK_CLI" && -x "$SPARK_CLI"]]; then > local is_logged_in=0 > "$SPARK_CLI" benchmark --help &>/dev/null && { > # Probe login state: spark-arena-cli prints a warning if not configured > local probe > probe=$(echo "exit" | timeout 3 "$SPARK_CLI" 2>&1 || true) > echo "$probe" | grep -q "Warning: Configuration not found" || is_logged_in=1 > } > > if [["$is_logged_in" -eq 1]]; then > log " ${GREEN}spark-arena-cli is logged in.${NC} Run this to submit officially:" > log " ${BOLD} $SPARK_CLI benchmark $recipe_file${NC}" > log " ${DIM} (This re-runs the benchmark via sparkrun and auto-uploads results)${NC}" > else > log " ${YELLOW}spark-arena-cli installed but not logged in.${NC} To submit:" > log " ${DIM} 1. $SPARK_CLI login # authenticate via Google/GitHub${NC}" > log " ${DIM} 2. $SPARK_CLI setup # configure sparkrun + llama-benchy${NC}" > log " ${DIM} 3. $SPARK_CLI benchmark $recipe_file${NC}" > fi > else > log " ${DIM}To submit to spark-arena, install spark-arena-cli:${NC}" > log " ${DIM} curl -fsSL https://github.com/spark-arena/spark-arena-cli/releases/download/v0.1.0/spark-arena-cli-0.1.0-linux-amd64 -o ~/.local/bin/spark-arena-cli && chmod +x ~/.local/bin/spark-arena-cli${NC}" > log " ${DIM} spark-arena-cli login${NC}" > log " ${DIM} spark-arena-cli benchmark $recipe_file${NC}" > fi > log "" > else > log " ${DIM}tg ${current_tg} tok/s — personal best is ${prev_best} tok/s (no improvement, skipping submission)${NC}" > fi > > ``` > > } > > # Generate a spark-arena recipe.yaml for a model > > generate\_recipe\_yaml() { > local model=“$1” > local safe\_name=“${model////\_}” > local out\_dir=“$ARENA\_DIR/${safe\_name}” > mkdir -p “$out\_dir” > > ``` > python3 - "$model" "$out_dir/recipe.yaml" <<'PYEOF' > > ``` > > import sys, textwrap > > model\_name = sys.argv[1] > out\_path = sys.argv[2] > > # Per-model recipe metadata. > > # container: the local Docker image tag we actually use. > > # hf\_model: canonical HuggingFace model ID for the submission. > > RECIPES = { > “Qwen3.5-35B-A3B-FP8”: { > “hf\_model”: “Qwen/Qwen3.5-35B-A3B-Instruct”, > “description”: “Qwen3.5 35B MoE FP8-dynamic — reasoning + tool use with MTP-2 speculation”, > “container”: “vllm-node:Version\_1”, > “tp”: 1, “gpu\_mem”: 0.7, “max\_len”: 131072, > “env”: { > “VLLM\_MARLIN\_USE\_ATOMIC\_ADD”: “1”, > “VLLM\_ENABLE\_CUDAGRAPH\_GC”: “1”, > “VLLM\_USE\_FLASHINFER\_SAMPLER”: “1”, > }, > “extras”: [ > “–kv-cache-dtype fp8”, > “–load-format fastsafetensors”, > “–attention-backend FLASHINFER”, > “–enable-prefix-caching”, > “–enable-chunked-prefill”, > “–max-num-batched-tokens 4096”, > ‘–speculative-config '{“method”:“mtp”,“num\_speculative\_tokens”:2}'’, > “–enable-auto-tool-choice”, > “–tool-call-parser qwen3\_xml”, > “–reasoning-parser qwen3”, > ], > }, > “Qwen3.5-122B-A10B-int4-AutoRound”: { > “hf\_model”: “Qwen/Qwen3.5-122B-A10B-Instruct”, > “description”: “Qwen3.5 122B MoE INT4 AutoRound — large hybrid reasoning model”, > “container”: “vllm-node-tf5:latest”, > “tp”: 1, “gpu\_mem”: 0.75, “max\_len”: 40960, > “env”: {“VLLM\_MARLIN\_USE\_ATOMIC\_ADD”: “1”}, > “extras”: [ > “–trust-remote-code”, > “–enforce-eager”, > “–kv-cache-dtype fp8”, > “–enable-prefix-caching”, > “–enable-auto-tool-choice”, > “–tool-call-parser qwen3\_xml”, > “–reasoning-parser qwen3”, > ], > }, > “Qwen3-VL-30B-A3B-Instruct-FP8”: { > “hf\_model”: “Qwen/Qwen3-VL-30B-A3B-Instruct”, > “description”: “Qwen3-VL 30B MoE FP8 — vision-language model”, > “container”: “spark-vllm:Version\_1”, > “tp”: 1, “gpu\_mem”: 0.60, “max\_len”: 32768, > “env”: {}, > “extras”: [ > “–trust-remote-code”, > “–kv-cache-dtype fp8”, > “–load-format fastsafetensors”, > “–enable-prefix-caching”, > “–limit-mm-per-prompt ‘{"image": 2}’”, > “–enable-auto-tool-choice”, > “–tool-call-parser qwen3\_coder”, > ], > }, > “Qwen3-Omni-30B-A3B-Instruct”: { > “hf\_model”: “Qwen/Qwen3-Omni-30B-A3B-Instruct”, > “description”: “Qwen3-Omni 30B MoE — audio + vision + text multimodal”, > “container”: “vllm-node:Version\_1”, > “tp”: 1, “gpu\_mem”: 0.75, “max\_len”: 32768, > “env”: {}, > “extras”: [ > “–trust-remote-code”, > “–load-format fastsafetensors”, > “–enable-prefix-caching”, > “–limit-mm-per-prompt ‘{"image": 2, "audio": 2}’”, > “–enable-auto-tool-choice”, > “–tool-call-parser qwen3\_coder”, > ], > }, > “Qwen3-Coder-Next-FP8-Dynamic”: { > “hf\_model”: “Qwen/Qwen3-Coder-480B-A22B-FP8-Dynamic”, > “description”: “Qwen3-Coder-Next 480B MoE FP8-Dynamic — coding specialist”, > “container”: “vllm-node-tf5:latest”, > “tp”: 1, “gpu\_mem”: 0.75, “max\_len”: 32768, > “env”: {}, > “extras”: [ > “–kv-cache-dtype fp8”, > “–load-format fastsafetensors”, > “–attention-backend flashinfer”, > “–enable-auto-tool-choice”, > “–tool-call-parser qwen3\_coder”, > ], > }, > “Qwen3-Coder-Next-int4-AutoRound”: { > “hf\_model”: “Qwen/Qwen3-Coder-480B-A22B-Instruct”, > “description”: “Qwen3-Coder-Next 480B MoE INT4 AutoRound — coding + throughput optimized”, > “container”: “vllm-node-tf5:latest”, > “tp”: 1, “gpu\_mem”: 0.6, “max\_len”: 32768, > “env”: { > “VLLM\_MARLIN\_USE\_ATOMIC\_ADD”: “1”, > “VLLM\_ALLOW\_LONG\_MAX\_MODEL\_LEN”: “1”, > “VLLM\_USE\_FLASHINFER\_MOE\_FP8”: “1”, > }, > “extras”: [ > “–language-model-only”, > “–enable-chunked-prefill”, > “–max-num-batched-tokens 49152”, > “–max-num-seqs 384”, > “–kv-cache-dtype fp8”, > “–load-format fastsafetensors”, > “–optimization-level 3”, > “–performance-mode throughput”, > “–enable-auto-tool-choice”, > “–tool-call-parser qwen3\_coder”, > ], > }, > “Nemotron-3-Nano-4B-FP8”: { > “hf\_model”: “nvidia/Nemotron-3-Nano-4B-Instruct”, > “description”: “Nemotron-3-Nano 4B FP8 — ultra-fast orchestrator / routing model”, > “container”: “spark-vllm:Version\_1”, > “tp”: 1, “gpu\_mem”: 0.5, “max\_len”: 8192, > “env”: {}, > “extras”: [ > “–kv-cache-dtype fp8”, > “–enforce-eager”, > “–trust-remote-code”, > “–load-format fastsafetensors”, > “–enable-prefix-caching”, > “–tool-call-parser qwen3\_coder”, > “–reasoning-parser nemotron\_v3”, > “–enable-auto-tool-choice”, > ], > }, > “Nemotron-3-Nano-30B-A3B-NVFP4”: { > “hf\_model”: “nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4”, > “description”: “Nemotron-3-Nano 30B NVFP4 — Blackwell-native MoE with nano\_v3 reasoning”, > “container”: “vllm-node:Version\_1”, > “tp”: 1, “gpu\_mem”: 0.65, “max\_len”: 131072, > “env”: { > “VLLM\_USE\_FLASHINFER\_MOE\_FP4”: “1”, > “VLLM\_FLASHINFER\_MOE\_BACKEND”: “throughput”, > }, > “extras”: [ > “–kv-cache-dtype fp8”, > “–enforce-eager”, > “–trust-remote-code”, > “–quantization modelopt\_fp4”, > “–enable-auto-tool-choice”, > “–tool-call-parser qwen3\_coder”, > “–reasoning-parser nano\_v3”, > ], > }, > “Nemotron-3-Super-120B-A12B-NVFP4”: { > “hf\_model”: “nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4”, > “description”: “Nemotron-3-Super 120B NVFP4 — large reasoning model with CUTLASS MoE”, > “container”: “spark-vllm:Version\_1”, > “tp”: 1, “gpu\_mem”: 0.7, “max\_len”: 65536, > “env”: { > “VLLM\_FLASHINFER\_ALLREDUCE\_BACKEND”: “trtllm”, > “VLLM\_ALLOW\_LONG\_MAX\_MODEL\_LEN”: “1”, > }, > “extras”: [ > “–kv-cache-dtype fp8”, > “–moe-backend cutlass”, > “–trust-remote-code”, > “–enable-prefix-caching”, > “–load-format fastsafetensors”, > “–tool-call-parser qwen3\_coder”, > “–enable-auto-tool-choice”, > “–reasoning-parser nemotron\_v3”, > ], > }, > “GPT-OSS-120B”: { > “hf\_model”: “openai/gpt-oss-120b”, > “description”: “OpenAI GPT-OSS 120B MXFP4 — open-weights GPT model”, > “container”: “vllm-node-mxfp4”, > “tp”: 1, “gpu\_mem”: 0.7, “max\_len”: 65536, > “env”: {“VLLM\_USE\_FLASHINFER\_MOE\_MXFP4\_MXFP8”: “1”}, > “extras”: [ > “–quantization mxfp4”, > “–kv-cache-dtype fp8”, > “–max-num-batched-tokens 8192”, > “–enable-prefix-caching”, > “–load-format fastsafetensors”, > “–tool-call-parser openai”, > “–reasoning-parser openai\_gptoss”, > “–enable-auto-tool-choice”, > ], > }, > “Mistral-Small-24B-Instruct-2501”: { > “hf\_model”: “mistralai/Mistral-Small-24B-Instruct-2501”, > “description”: “Mistral Small 24B — fast roleplay and instruction following”, > “container”: “vllm-node:Version\_1”, > “tp”: 1, “gpu\_mem”: 0.7, “max\_len”: 32768, > “env”: {}, > “extras”: [ > “–trust-remote-code”, > “–enforce-eager”, > “–enable-auto-tool-choice”, > “–tool-call-parser mistral”, > ], > }, > “Qwen3.5-35B-A3B-Uncensored-HauhauCS-Aggressive-Q4\_K\_M-GGUF”: { > “hf\_model”: “HauhauCS/Qwen3.5-35B-A3B-Uncensored-Aggressive-GGUF”, > “description”: “Qwen3.5 35B MoE Q4\_K\_M GGUF — uncensored, llama.cpp serving”, > “container”: “[ghcr.io/martin-b78/llama-cpp-spark:latest](http://ghcr.io/martin-b78/llama-cpp-spark:latest)”, > “tp”: 1, “gpu\_mem”: 0.0, “max\_len”: 16384, > “env”: {“GGML\_CUDA\_ENABLE\_UNIFIED\_MEMORY”: “1”}, > “extras”: [ > “–ctx-size 16384”, > “–n-gpu-layers 99”, > “–parallel 4”, > “–no-mmap”, > ], > “runtime”: “llama-cpp”, > }, > } > > r = RECIPES.get(model\_name) > if not r: > # Unknown model — generate a minimal placeholder recipe > r = { > “hf\_model”: f"\", > “description”: model\_name, > “container”: “\”, > “tp”: 1, “gpu\_mem”: 0.7, “max\_len”: 32768, > “env”: {}, “extras”: , > } > > runtime = r.get(“runtime”, “vllm”) > hf\_model = r[“hf\_model”] > extras = “\n”.join(f" {e} \" for e in r[“extras”]) > env\_block = “” > if r[“env”]: > env\_lines = “\n”.join(f" {k}: ‘{v}’" for k, v in r[“env”].items()) > env\_block = f"env:\n{env\_lines}\n" > > if runtime == “llama-cpp”: > cmd = ( > f"llama-server \\n" > f" -hf {hf\_model} \\n" > f" --host {{host}} --port {{port}} \\n" > + “\n”.join(f" {e} \" for e in r[“extras”]) > + “\n” > ) > else: > cmd = ( > f"vllm serve {hf\_model} \\n" > f" --served-model-name {model\_name} \\n" > f" --host {{host}} --port {{port}} \\n" > f" --tensor-parallel-size {{tensor\_parallel}} \\n" > f" --gpu-memory-utilization {{gpu\_memory\_utilization}} \\n" > f" --max-model-len {{max\_model\_len}} \\n" > + “\n”.join(f" {e} \" for e in r[“extras”]) > + “\n” > ) > > yaml = f""“recipe\_version: ‘1’ > name: {model\_name} > description: {r[‘description’]} > model: {hf\_model} > cluster\_only: false > container: {r[‘container’]} > defaults: > port: 8000 > host: 0.0.0.0 > tensor\_parallel: {r[‘tp’]} > gpu\_memory\_utilization: {r[‘gpu\_mem’]} > max\_model\_len: {r[‘max\_len’]} > {env\_block}command: | > {cmd.rstrip()} > solo\_only: false > “”” > > with open(out\_path, “w”) as f: > f.write(yaml) > > print(f" Recipe YAML: {out\_path}") > PYEOF > > ``` > if [[$? -ne 0]]; then > log " ${YELLOW}Warning: recipe.yaml generation failed for $model${NC}" > fi > > ``` > > } > > # --------------------------------------------------------------------------- > > # Crash-resume checkpoint functions > > # --------------------------------------------------------------------------- > > # Create a new checkpoint file for this session and register it as last-known > > init\_checkpoint() { > mkdir -p “$CHECKPOINT\_DIR” > CHECKPOINT\_FILE=“$CHECKPOINT\_DIR/session\_${TIMESTAMP}.json” > python3 - “$CHECKPOINT\_FILE” “$MODE” “$PP” “$TG” “$DEPTH” “$RUNS” \<\<‘PYEOF’ > import json, sys, datetime > f, mode, pp, tg, depth, runs = sys.argv[1:] > data = { > “session\_id”: f.split(“\_”)[-1].replace(“.json”, “”), > “started\_at”: datetime.datetime.now().isoformat(), > “mode”: mode, > “settings”: {“pp”: pp, “tg”: tg, “depth”: depth, “runs”: int(runs)}, > “models”: , > } > with open(f, “w”) as fh: > json.dump(data, fh, indent=2) > PYEOF > echo “$CHECKPOINT\_FILE” \> “$LAST\_SESSION\_FILE” > } > > # Mark a model as started (written BEFORE warmup so a crash is detectable) > > checkpoint\_model\_start() { > local model=“$1” > [[-z “$CHECKPOINT\_FILE”]] && return > python3 - “$CHECKPOINT\_FILE” “$model” \<\<‘PYEOF’ > import json, sys, datetime > f, model = sys.argv[1:] > with open(f) as fh: > data = json.load(fh) > data[“models”].append({ > “model”: model, > “status”: “started”, > “started\_at”: datetime.datetime.now().isoformat(), > }) > with open(f, “w”) as fh: > json.dump(data, fh, indent=2) > PYEOF > } > > # Mark a model as completed with its result (OK / FAIL / INCOHERENT / EMPTY) > > checkpoint\_model\_done() { > local model=“$1” > local result=“$2” > [[-z “$CHECKPOINT\_FILE”]] && return > python3 - “$CHECKPOINT\_FILE” “$model” “$result” \<\<‘PYEOF’ > import json, sys, datetime > f, model, result = sys.argv[1:] > with open(f) as fh: > data = json.load(fh) > for entry in reversed(data[“models”]): > if entry[“model”] == model and entry.get(“status”) == “started”: > entry[“status”] = “completed” > entry[“result”] = result > entry[“completed\_at”] = datetime.datetime.now().isoformat() > break > with open(f, “w”) as fh: > json.dump(data, fh, indent=2) > PYEOF > } > > # Load a previous checkpoint and populate SKIP\_MODELS (models already done) > > load\_resume\_checkpoint() { > if [[! -f “$LAST\_SESSION\_FILE”]]; then > echo -e “${RED}Error: no previous session found. Run without --resume first.${NC}” \>&2 > exit 1 > fi > local prev\_cp > prev\_cp=$(cat “$LAST\_SESSION\_FILE”) > if [[! -f “$prev\_cp”]]; then > echo -e “${RED}Error: checkpoint file not found: $prev\_cp${NC}” \>&2 > exit 1 > fi > > ``` > log " ${CYAN}Resuming from checkpoint:${NC} $prev_cp" > > local info > info=$(python3 - "$prev_cp" <<'PYEOF' > > ``` > > import json, sys > data = json.load(open(sys.argv[1])) > completed = [e[“model”] for e in data.get(“models”, ) if e.get(“status”) == “completed”] > last\_started = next( > (e[“model”] for e in reversed(data.get(“models”, )) if e.get(“status”) == “started”), > “”, > ) > s = data.get(“settings”, {}) > print(“MODE:” + data.get(“mode”, “”)) > print(“SETTINGS:pp=” + str(s.get(“pp”,“”)) + " tg=" + str(s.get(“tg”,“”)) + > " depth=" + str(s.get(“depth”,“”)) + " runs=" + str(s.get(“runs”,“”))) > print(“LAST\_STARTED:” + last\_started) > for m in completed: > print(“DONE:” + m) > PYEOF > ) > while IFS= read -r line; do > case “$line” in > DONE:_) SKIP\_MODELS+=(“${line#DONE:}”) ;; > MODE:_) log " Previous mode : ${line#MODE:}" ;; > SETTINGS:_) log " Previous settings : ${line#SETTINGS:}" ;; > LAST\_STARTED:_) > local last=“${line#LAST\_STARTED:}” > if [[-n “$last”]]; then > log " ${YELLOW}Crashed while running: $last${NC} — will re-run it." > fi > ;; > esac > done \<\<\< “$info” > > ``` > log " Skipping ${#SKIP_MODELS[@]} already-completed model(s)." > log "" > > ``` > > } > > # --------------- MAIN --------------- > > log “” > log “${BOLD}============================================================${NC}” > log “${BOLD} DGX Spark Model Benchmark${NC}” > log “${BOLD} powered by llama-benchy ([GitHub - eugr/llama-benchy: llama-benchy - llama-bench style benchmarking tool for all backends · GitHub](http://github.com/eugr/llama-benchy))${NC}” > log “${BOLD}============================================================${NC}” > log “” > log " Endpoint : $LLAMA\_SWAP\_URL" > log " Mode : $MODE" > log " Settings : pp=$PP tg=$TG depth=$DEPTH runs=$RUNS" > log " Date : $(date ‘+%Y-%m-%d %H:%M’)" > log " Results : $RESULTS\_DIR" > log “” > > # Mode descriptions > > case “$MODE” in > medium-log) > log " ${CYAN}Profile: Medium Log Baseline${NC}" > log " ${DIM}Simulates a ~50-page document (depth=16384). Establishes your${NC}" > log " ${DIM}baseline generation speed with a moderately full KV cache.${NC}" > ;; > stress) > log " ${CYAN}Profile: Massive Log Stress Test${NC}" > log " ${DIM}Doubles context to simulate a massive error log (depth=32768).${NC}" > log " ${DIM}Watch for tg tok/s drop — that’s the unified memory bottleneck.${NC}" > ;; > extreme) > log " ${CYAN}Profile: Extreme Limit Test${NC}" > log " ${DIM}Pushes to ~200 pages (depth=65535). Tests if the system can${NC}" > log " ${DIM}process it without crashing or heavy swap paging.${NC}" > ;; > quick) > log " ${DIM}Quick smoke test — just checking if models respond.${NC}" > ;; > full) > log " ${DIM}Full comprehensive sweep — broad pp/tg/depth combinations.${NC}" > ;; > arena) > log " ${CYAN}Profile: Spark-Arena Leaderboard Submission${NC}" > log " ${DIM}Official [spark-arena.com](http://spark-arena.com) benchmark spec: 7 depth levels × 4 concurrency${NC}" > log " ${DIM}levels × 3 runs = 84 data points per model.${NC}" > log " ${DIM}Generates results.csv + recipe.yaml per model in:${NC}" > log " ${DIM} $ARENA\_DIR${NC}" > ;; > esac > log “” > log " ${DIM}pp = prompt processing (how fast the model reads your input)${NC}" > log " ${DIM}tg = token generation (how fast the model writes its reply)${NC}" > log " ${DIM}depth = pre-filled context tokens (simulates document size)${NC}" > log “” > > # Initialize crash-resume checkpoint and, if --resume, load previous state > > init\_checkpoint > [[“$RESUME” == true]] && load\_resume\_checkpoint > > # Check llama-benchy is available > > if ! uvx llama-benchy --help \> /dev/null 2\>&1; then > log “${RED}Error: llama-benchy not available via uvx.${NC}” > log “Install with: pip install llama-benchy OR uv pip install llama-benchy” > exit 1 > fi > > # Fetch model list > > MODELS=$(curl -sf “$LLAMA\_SWAP\_URL/v1/models” | jq -r ‘.data.id’ | sort) > MODEL\_COUNT=$(echo “$MODELS” | wc -l) > > # Apply filters > > if [[${#FILTERS[@]} -gt 0 ]]; then > log “Filtering models matching: ${FILTERS[\*]}” > FILTERED=“” > for m in $MODELS; do > for f in “${FILTERS[@]}”; do > if [[“$m” == _“$f”_]]; then > FILTERED=“${FILTERED}${m}\n” > fi > done > done > MODELS=$(echo -e “$FILTERED” | grep -v ‘^$’ | sort -u) > MODEL\_COUNT=$(echo “$MODELS” | wc -l) > fi > > log “Found ${BOLD}${MODEL\_COUNT}${NC} model(s) to benchmark.” > log “” > > PASS=0 > FAIL=0 > IDX=0 > TOTAL\_START=$(date +%s.%N) > > # Collect results for final summary > > declare -A SUMMARY\_PP SUMMARY\_TG SUMMARY\_PEAK SUMMARY\_TTFT SUMMARY\_STATUS SUMMARY\_DEGRADATION > > for MODEL in $MODELS; do > IDX=$((IDX + 1)) > > ``` > # Skip models already completed in a --resume session > if [["$RESUME" == true]]; then > _skip=false > for _done in "${SKIP_MODELS[@]:-}"; do > [["$MODEL" == "$_done"]] && { _skip=true; break; } > done > if [["$_skip" == true]]; then > log " ${DIM}⏭ [$IDX/$MODEL_COUNT] $MODEL — skipped (completed in previous session)${NC}" > continue > fi > fi > > log "${BOLD}============================================================${NC}" > log "${BOLD} [$IDX/$MODEL_COUNT] $MODEL${NC}" > log "${BOLD}============================================================${NC}" > log "" > > # Write checkpoint BEFORE starting so a crash is detectable > checkpoint_model_start "$MODEL" > > # Unload previous model to get a clean measurement > unload_all > > # Load this model > WARMUP_FAIL_REASON="" > if ! warmup_model "$MODEL"; then > FAIL=$((FAIL + 1)) > SUMMARY_STATUS[$MODEL]="${WARMUP_FAIL_REASON:-FAIL}" > checkpoint_model_done "$MODEL" "${WARMUP_FAIL_REASON:-FAIL}" > log "" > continue > fi > > # Benchmark it > if run_benchy "$MODEL"; then > PASS=$((PASS + 1)) > SUMMARY_STATUS[$MODEL]="OK" > checkpoint_model_done "$MODEL" "OK" > # In arena mode: generate recipe YAML and check vs personal best > if [["$MODE" == "arena"]]; then > generate_recipe_yaml "$MODEL" 2>&1 | tee -a "$REPORT_FILE" > local arena_json="$RESULTS_DIR/${MODEL//\//_}_${TIMESTAMP}.json" > local arena_recipe="$ARENA_DIR/${MODEL//\//_}/recipe.yaml" > check_and_suggest_submit "$MODEL" "$arena_json" "$arena_recipe" 2>&1 | tee -a "$REPORT_FILE" > fi > > # Extract numbers for final summary from JSON > local_json="$RESULTS_DIR/${MODEL//\//_}_${TIMESTAMP}.json" > if [[-f "$local_json"]]; then > # Extract metrics using jq for the summary table > # We look for depth 0 (baseline) and the deepest result > baseline_tg=$(jq -r '.benchmarks[] | select(.context_size == 0) | .tg_throughput.mean // empty' "$local_json" | head -n1) > max_depth=$(jq -r '.benchmarks[].context_size' "$local_json" | sort -rn | head -n1) > deepest_tg=$(jq -r ".benchmarks[] | select(.context_size == $max_depth) | .tg_throughput.mean // empty" "$local_json" | head -n1) > > # Baseline metrics for the summary table > SUMMARY_PP[$MODEL]=$(jq -r '.benchmarks[] | select(.context_size == 0) | if .pp_throughput.std > 0.5 then "\(.pp_throughput.mean + 0.5 | floor) +/-\(.pp_throughput.std + 0.5 | floor)" else "\(.pp_throughput.mean + 0.5 | floor)" end' "$local_json" | head -n1) > SUMMARY_TG[$MODEL]=$(jq -r '.benchmarks[] | select(.context_size == 0) | if .tg_throughput.std > 0.5 then "\((.tg_throughput.mean * 10 + 0.5 | floor) / 10) +/-\((.tg_throughput.std * 10 + 0.5 | floor) / 10)" else "\((.tg_throughput.mean * 10 + 0.5 | floor) / 10)" end' "$local_json" | head -n1) > SUMMARY_PEAK[$MODEL]=$(jq -r '.benchmarks[] | select(.context_size == 0) | .peak_throughput.mean + 0.5 | floor' "$local_json" | head -n1) > SUMMARY_TTFT[$MODEL]=$(jq -r '.benchmarks[] | select(.context_size == 0) | .e2e_ttft.mean + 0.5 | floor' "$local_json" | head -n1) > > if [[-n "$baseline_tg" && -n "$deepest_tg" && "$max_depth" -gt 0]]; then > pct=$(echo "scale=1; (($deepest_tg - $baseline_tg) / $baseline_tg) * 100" | bc) > SUMMARY_DEGRADATION[$MODEL]="${pct}% @$((max_depth/1024))k" > fi > fi > else > FAIL=$((FAIL + 1)) > SUMMARY_STATUS[$MODEL]="FAIL" > checkpoint_model_done "$MODEL" "FAIL" > fi > log "" > > ``` > > done > > # Final cleanup > > unload\_all > > TOTAL\_END=$(date +%s.%N) > TOTAL\_ELAPSED=$(echo “$TOTAL\_END - $TOTAL\_START” | bc) > TOTAL\_MIN=$(echo “scale=1; $TOTAL\_ELAPSED / 60” | bc) > > # ============================================= > > # FINAL REPORT > > # ============================================= > > log “” > log “${BOLD}============================================================${NC}” > log “${BOLD} BENCHMARK REPORT — DGX Spark${NC}” > log “${BOLD}============================================================${NC}” > log “” > log " Date: $(date ‘+%Y-%m-%d %H:%M’) | Mode: $MODE | Runs: $RUNS" > log " Depths tested: $DEPTH" > log " Models tested: $MODEL\_COUNT | Passed: ${GREEN}$PASS${NC} Failed: ${RED}$FAIL${NC}" > log " Total benchmark time: ${TOTAL\_MIN} min" > log " Checkpoint : $CHECKPOINT\_FILE" > [[$FAIL -gt 0]] && log " ${YELLOW}Tip: if this was interrupted, resume with: ./benchmark-models.sh --resume${NC}" > log “” > log " ${BOLD}$(printf ‘%-42s %14s %12s %8s %8s %14s’ ‘Model’ ‘Read (pp)’ ‘Write (tg)’ ‘Peak’ ‘TTFT’ ‘Deep ctx’)${NC}" > log " ${DIM}$(printf ‘%-42s %14s %12s %8s %8s %14s’ ‘’ ‘tok/s’ ‘tok/s’ ‘tok/s’ ‘ms’ ‘degradation’)${NC}" > log " $(printf ‘%.0s-’ {1..106})" > > for MODEL in $MODELS; do > local\_name=“$MODEL” > [[${#local\_name} -gt 42]] && local\_name=“${local\_name:0:39}…” > > ``` > status="${SUMMARY_STATUS[$MODEL]:-FAIL}" > pp="${SUMMARY_PP[$MODEL]:-—}" > tg="${SUMMARY_TG[$MODEL]:-—}" > peak="${SUMMARY_PEAK[$MODEL]:-—}" > ttft="${SUMMARY_TTFT[$MODEL]:-—}" > degrad="${SUMMARY_DEGRADATION[$MODEL]:-—}" > > if [["$status" == "OK"]]; then > printf -v line " %-42s %14s %12s %8s %8s %14s" "$local_name" "$pp" "$tg" "$peak" "$ttft" "$degrad" > log "${GREEN}${line}${NC}" > elif [["$status" == "INCOHERENT"]]; then > printf -v line " %-42s %14s %12s %8s %8s %14s" "$local_name" "INCOHERENT" "—" "—" "—" "—" > log "${YELLOW}${line}${NC}" > else > printf -v line " %-42s %14s %12s %8s %8s %14s" "$local_name" "FAIL" "—" "—" "—" "—" > log "${RED}${line}${NC}" > fi > > ``` > > done > > log “” > log " ${DIM}---------------------------------------------------------------${NC}" > log " ${DIM}How to read this table:${NC}" > log " ${DIM}${NC}" > log " ${DIM} Read (pp) = How fast the model reads your prompt (higher = better).${NC}" > log " ${DIM} Write (tg) = How fast the model types its answer at depth=0 baseline.${NC}" > log " ${DIM} This is the speed you feel when chatting.${NC}" > log " ${DIM} Humans read at ~4 tok/s, so 20+ feels smooth.${NC}" > log " ${DIM} Peak = Fastest burst speed observed in a 1-second window.${NC}" > log " ${DIM} TTFT = Time until the first word appears (lower = better).${NC}" > log " ${DIM} Deep ctx = Speed change at max tested depth vs baseline.${NC}" > log " ${DIM} \>-15% = unified memory bandwidth bottleneck.${NC}" > log " ${DIM}---------------------------------------------------------------${NC}" > log “” > log " Results saved to:" > log " Report : $REPORT\_FILE" > log " JSON data : $RESULTS\_DIR/_\_${TIMESTAMP}.json" > log " Forum tables: $RESULTS\_DIR/_\_${TIMESTAMP}.md" > log “” > log " ${DIM}Tip: To share on NVIDIA forums, copy the llama-benchy tables${NC}" > log " ${DIM}from the .md files — they use the standard format everyone knows.${NC}" > > if [[“$MODE” == “arena” && -n “$ARENA\_DIR”]]; then > log “” > log “${BOLD}============================================================${NC}” > log “${BOLD} SPARK-ARENA SUBMISSION FILES${NC}” > log “${BOLD}============================================================${NC}” > log “” > log " Submission directory: ${CYAN}$ARENA\_DIR${NC}" > log “” > log " Per-model folders (one submission per model):" > for MODEL in $MODELS; do > safe=“${MODEL////\_}” > model\_dir=“$ARENA\_DIR/$safe” > if [[-f “$model\_dir/results.csv” && -f “$model\_dir/recipe.yaml”]]; then > log " ${GREEN}✓${NC} $MODEL" > log " recipe.yaml : $model\_dir/recipe.yaml" > log " results.csv : $model\_dir/results.csv" > elif [[“${SUMMARY\_STATUS[$MODEL]:-FAIL}” != “OK” ]]; then > log " ${RED}✗${NC} $MODEL (benchmark failed — no submission files)" > fi > done > log “” > log " ${BOLD}How to submit to [spark-arena.com/admin:${NC}](http://spark-arena.com/admin:$%7BNC%7D)" > log " ${DIM}1. Open [https://spark-arena.com/admin${NC}](https://spark-arena.com/admin$%7BNC%7D)" > log " ${DIM}2. For each model folder above:${NC}" > log " ${DIM} a. Paste or upload the contents of recipe.yaml${NC}" > log " ${DIM} b. Upload results.csv${NC}" > log " ${DIM}3. Submit one entry per model.${NC}" > log “” > log " ${DIM}Note: The recipe.yaml ‘model:’ field uses the canonical HuggingFace ID.${NC}" > log " ${DIM}If your model was downloaded from a different source, update it.${NC}" > fi > log “${BOLD}============================================================${NC}” > > ` Full stack is `LiteLLM → llama-swap → vLLM/llama.cpp`, benchmark script is at: `benchmark-models.sh` in the repo — drop it next to your llama-swap config and it auto-discovers all registered models. Requires `uvx` (comes with `uv`) + `llama-benchy`. No other deps. ---
**Author:** ![dbsci](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/dbsci/32/467918_2.png) [@dbsci](https://forums.developer.nvidia.com/u/dbsci)\ **Post date:** [April 20, 2026, 5:19pm UTC](https://forums.developer.nvidia.com/t/new-tool-llama-benchy-llama-bench-style-benchmarking-for-any-llm-backend-vllm-sglang-llama-cpp-etc/356698/14 "2026-04-20T17:19:14Z")
Cool project. Still might be good for you to consider using sparkrun for arena benchmarks because sparkrun also supports login and upload of results, so that’s one step. (Assuming the person has done login via sparkrun). We can talk offline about integration because sparkrun does let you benchmark a model that’s already running (hidden option for arena benchmarks, but it’s implemented), so you could just point sparkrun at the recipe file and have it run the benchmark and upload directly to spark arena (it’ll still save local copies of files that you could use for local work and arena-best-results.json, etc.). That could basically replace the spark-arena-cli part… plus sparkrun collects other metadata to try to properly classify and register versions of CUDA, NCCL, vllm, etc. as part of the benchmark data. It’s not all exposed today on [spark-arena.com](http://spark-arena.com), but benchmarks uploaded via sparkrun have richer metadata that can be used to properly identify quants, pin specific versions of spark-vllm-docker for future reproduction, and understand changes over time from vllm version, etc. Also, FYI. sparkrun does have a litellm proxy functionality that supports load/unload and dynamically assigning ports to avoid collisions, etc. That being said, you still need to choose what to unload – it doesn’t act as a scheduler to decide what to put where, etc. Anyway, we can talk offline if it makes sense to integrate or not. ---
**Author:** ![cho](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@cho](https://forums.developer.nvidia.com/u/cho)\ **Post date:** [April 20, 2026, 5:31pm UTC](https://forums.developer.nvidia.com/t/new-tool-llama-benchy-llama-bench-style-benchmarking-for-any-llm-backend-vllm-sglang-llama-cpp-etc/356698/15 "2026-04-20T17:31:23Z")
> [@martinB78](#): > > benchmark-models.sh ```auto VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_WORKER_MULTIPROC_METHOD=spawn vllm serve --model /llm/models/DeepSeek-R1-Distill-Qwen-7B --served-model-name DeepSeek-R1-Distill-Qwen-7B --dtype=float16 --enforce-eager --port 8000 --host 0.0.0.0 --trust-remote-code --disable-sliding-window --gpu-memory-util=0.9 --disable-log-requests --block-size 64 --quantization fp8 -tp=1 bash benchmark-models.sh DeepSeek-R1-Distill-Qwen-7B ============================================================ DGX Spark Model Benchmark powered by llama-benchy (github.com/eugr/llama-benchy) ============================================================ Endpoint : http://localhost:8000 Mode : medium-log Settings : pp=2048 tg=128 depth=0 16384 runs=3 Date : 2026-04-21 01:26 Results : /home/intel/Downloads/vllm/test-results/benchmarks Profile: Medium Log Baseline Simulates a ~50-page document (depth=16384). Establishes your baseline generation speed with a moderately full KV cache. pp = prompt processing (how fast the model reads your input) tg = token generation (how fast the model writes its reply) depth = pre-filled context tokens (simulates document size) Error: llama-benchy not available via uvx. Install with: pip install llama-benchy OR uv pip install llama-benchy (.venv) intel@P7:~/Downloads/vllm$ llama-benchy PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used. usage: llama-benchy [-h] [--version] --base-url BASE_URL [--api-key API_KEY] [--model MODEL] [--served-model-name SERVED_MODEL_NAME] [--tokenizer TOKENIZER] [--pp PP [PP ...]] [--tg TG [TG ...]] [--depth DEPTH [DEPTH ...]] [--runs RUNS] [--no-cache] [--post-run-cmd POST_RUN_CMD] [--book-url BOOK_URL] [--latency-mode {api,generation,none}] [--no-warmup] [--skip-coherence] [--adapt-prompt] [--no-adapt-prompt] [--enable-prefix-caching] [--concurrency CONCURRENCY [CONCURRENCY ...]] [--save-result SAVE_RESULT] [--format {md,json,csv}] [--save-total-throughput-timeseries] [--save-all-throughput-timeseries] [--exit-on-first-fail] [--no-results-on-fail] llama-benchy: error: the following arguments are required: --base-url ``` If vLLM is calling a model that has already been downloaded into a specified local folder, the script should be able to handle it correctly. [Can I use locale tokenizer ? · Issue #7 · eugr/llama-benchy](https://github.com/eugr/llama-benchy/issues/7) pip show transformers Name: transformers Version: 5.5.4 Summary: Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training. Home-page: [GitHub - huggingface/transformers: 🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training. · GitHub](https://github.com/huggingface/transformers) Author: The Hugging Face team (past and future) with the help of all our contributors ([Contributors to huggingface/transformers · GitHub](https://github.com/huggingface/transformers/graphs/contributors)) Author-email: [transformers@huggingface.co](mailto:transformers@huggingface.co) License: Apache 2.0 License Location: /home/intel/Downloads/tool-eval-bench/.venv/lib/python3.13/site-packages Requires: huggingface-hub, numpy, packaging, pyyaml, regex, safetensors, tokenizers, tqdm, typer Required-by: llama-benchy ---
**Author:** ![martinB78](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/martinb78/32/437495_2.png) [@martinB78](https://forums.developer.nvidia.com/u/martinB78)\ **Post date:** [April 21, 2026, 7:25am UTC](https://forums.developer.nvidia.com/t/new-tool-llama-benchy-llama-bench-style-benchmarking-for-any-llm-backend-vllm-sglang-llama-cpp-etc/356698/16 "2026-04-21T07:25:35Z")
> [@cho](#): > > If vLLM is calling a model that has already been downloaded into a specified local folder, the script should be able to handle it correctly. thanks cho for the feedback. `llama-benchy` is installed directly in our venv but not `uvx` . The script needs to detect whichever is available. The script now tries in order: 1. `uvx llama-benchy` — “my” setup, no install needed 2. `llama-benchy` — the “your” setup (pip-installed in venv) 3. Fails with clear instructions if neither is found [download the update from github](https://github.com/mARTin-B78/dgx-spark_lite-llm_llama-swap_vllm_llama-cpp_ollama/blob/main/benchmark-models.sh) > **or copy the updated script benchmark-models.sh here** > > ```auto > # ============================================================================= > # benchmark-models.sh — Benchmark all llama-swap models using llama-benchy > # > # Uses llama-benchy (https://github.com/eugr/llama-benchy) for standardized > # LLM performance measurement. Results are comparable with other DGX Spark > # users who use the same tool. > # > # Benchmark profiles simulate real-world document analysis workloads: > # > # "Medium Log" (default) — 50-page document, pp2048 + tg128 @ depth 16384 > # "Massive Log" (--stress) — 100+ page log, pp2048 + tg128 @ depth 32768 > # "Extreme" (--extreme)— 200+ page corpus, pp2048 + tg128 @ depth 65535 > # > # Each profile includes a depth=0 baseline so you can see the performance > # delta as unified memory pressure increases. > # > # Usage: > # ./benchmark-models.sh # Medium Log (default) > # ./benchmark-models.sh --stress # Medium + Massive Log > # ./benchmark-models.sh --extreme # All three depth levels > # ./benchmark-models.sh --quick Nemotron # Fast smoke test > # ./benchmark-models.sh --runs 5 Qwen3.5-35B # Custom run count > # ./benchmark-models.sh Qwen3.5 Nemotron # Only matching models > # ============================================================================= > > set -euo pipefail > > LLAMA_SWAP_URL="${LLAMA_SWAP_URL:-http://localhost:28080}" > SCRIPT_DIR="$(dirname "$(readlink -f "$0")")" > RESULTS_DIR="$SCRIPT_DIR/test-results/benchmarks" > ARENA_BEST_FILE="$SCRIPT_DIR/test-results/arena-best-results.json" > TIMEOUT=1800 > > # spark-arena-cli: installed to ~/.local/bin on first --arena run > SPARK_CLI="${HOME}/.local/bin/spark-arena-cli" > > # Colors > GREEN='\033[0;32m' > RED='\033[0;31m' > YELLOW='\033[1;33m' > CYAN='\033[0;36m' > BOLD='\033[1m' > DIM='\033[2m' > NC='\033[0m' > > # Defaults: "Medium Log" baseline > PP="2048" > TG="128" > DEPTH="0 16384" > RUNS=3 > MODE="medium-log" > CONCURRENCY="" # space-separated list; empty = not passed to llama-benchy > ARENA_DIR="" # set by --arena mode > FILTERS=() > > # Crash-resume tracking > CHECKPOINT_DIR="$SCRIPT_DIR/test-results/checkpoints" > LAST_SESSION_FILE="$SCRIPT_DIR/test-results/.last-session" > CHECKPOINT_FILE="" > RESUME=false > SKIP_MODELS=() > > # Parse arguments > while [[$# -gt 0]]; do > case "$1" in > --quick) > PP="512" > TG="128" > DEPTH="0" > RUNS=1 > MODE="quick" > shift > ;; > --stress) > # Medium Log + Massive Log > PP="2048" > TG="128" > DEPTH="0 16384 32768" > RUNS=3 > MODE="stress" > shift > ;; > --extreme) > # All three: Medium + Massive + Extreme limit > PP="2048" > TG="128" > DEPTH="0 16384 32768 65535" > RUNS=3 > MODE="extreme" > shift > ;; > --full) > # Comprehensive sweep (original broad test) > PP="512 2048" > TG="128 256 512" > DEPTH="0 16384 32768" > RUNS=3 > MODE="full" > shift > ;; > --arena) > # Spark-Arena leaderboard submission profile > # https://spark-arena.com/admin > PP="2048" > TG="128" > DEPTH="0 4096 8192 16384 32768 65535 100000" > CONCURRENCY="1 2 5 10" > RUNS=3 > MODE="arena" > shift > ;; > --runs) > RUNS="$2" > shift 2 > ;; > --runs=*) > RUNS="${1#*=}" > shift > ;; > --resume) > RESUME=true > shift > ;; > --help|-h) > cat <<'HELPEOF' > Usage: benchmark-models.sh [OPTIONS] [FILTER...] > > Benchmark your llama-swap models using llama-benchy. > Results use the standard llama-benchy format for comparison > with other DGX Spark users on the NVIDIA forums. > > Profiles (simulating real-world document workloads): > > (default) "Medium Log" — pp2048, tg128 @ depth 0 + 16384 > Simulates a ~50-page document. The depth=0 baseline shows > raw speed; depth=16384 shows the cost of a full KV cache. > > --stress "Massive Log" — adds depth 32768 > Doubles the context to simulate a massive error log. > Watch for tg tok/s drop vs. the medium baseline — that's > where shared memory bandwidth starts to bottleneck. > > --extreme "Extreme Limit" — adds depth 65535 > Pushes to ~100 pages of text. Strictly to see if the > system can process it without crashing or heavy swap. > > --quick Smoke test — pp512, tg128, depth 0, 1 run > --full Broad sweep — pp512+2048, tg128+256+512, depths 0-32k > --arena Spark-Arena leaderboard profile — exact spec from spark-arena.com/admin > Saves results.csv + recipe.yaml per model to test-results/arena-submission/ > Depths: 0 4096 8192 16384 32768 65535 100000 | Concurrency: 1 2 5 10 > > Other options: > --runs N Override number of runs (default: 3) > --resume Resume from the last interrupted session (skip already-completed models) > --help Show this help > > Filters: > Add model name fragments to only test matching models. > Example: ./benchmark-models.sh --stress Qwen3.5 Nemotron > > Environment: > LLAMA_SWAP_URL llama-swap endpoint (default: http://localhost:28080) > > What the numbers mean: > pp tok/s = Prompt Processing speed. How fast the model reads your input. > Higher is better. Typically 500-5000+ tok/s on DGX Spark. > tg tok/s = Token Generation speed. How fast the model writes its reply. > Higher is better. This is the number you "feel" when chatting. > Typically 15-50+ tok/s on DGX Spark depending on model size. > TTFT = Time To First Token. The delay before the model starts replying. > Lower is better. Measured in milliseconds. > > Key insight: > Compare tg tok/s across depths. A big drop from depth=16384 to depth=32768 > means you've found the unified memory bandwidth bottleneck between the ARM > CPU and the Blackwell GPU on DGX Spark. > HELPEOF > exit 0 > ;; > *) > FILTERS+=("$1") > shift > ;; > esac > done > > mkdir -p "$RESULTS_DIR" > TIMESTAMP=$(date +%Y%m%d_%H%M%S) > REPORT_FILE="$RESULTS_DIR/report_${TIMESTAMP}.txt" > > if [["$MODE" == "arena"]]; then > ARENA_DIR="$SCRIPT_DIR/test-results/arena-submission/${TIMESTAMP}" > mkdir -p "$ARENA_DIR" > fi > > log() { echo -e "$1" | tee -a "$REPORT_FILE"; } > > unload_all() { > curl -sf -X POST "$LLAMA_SWAP_URL/unload" > /dev/null 2>&1 || true > sleep 5 > } > > # Warm up: send a tiny request to make llama-swap load the model > warmup_model() { > local model="$1" > log " Loading model via llama-swap..." > local start end elapsed > start=$(date +%s.%N) > > local response_file="/tmp/response_${TIMESTAMP}.json" > local http_code > http_code=$(curl -s -w "%{http_code}" -o "$response_file" --max-time "$TIMEOUT" \ > -X POST "$LLAMA_SWAP_URL/v1/chat/completions" \ > -H "Content-Type: application/json" \ > -d "$(jq -n --arg model "$model" '{ > model: $model, > messages: [{role: "user", content: "Write a short python hello world script."}], > max_tokens: 50 > }')" 2>/dev/null) || http_code=0 > > end=$(date +%s.%N) > elapsed=$(echo "scale=1; $end - $start" | bc) > > if [["$http_code" -ne 200]]; then > local err_msg > err_msg=$(jq -r '.error.message // empty' "$response_file" 2>/dev/null) > if [[-n "$err_msg"]]; then > log " ${RED}FAILED to load: $err_msg (HTTP $http_code)${NC}" > else > log " ${RED}FAILED to load (HTTP $http_code)${NC}" > fi > rm -f "$response_file" > return 1 > fi > > local content > content=$(jq -r '(.choices[0].message.reasoning_content // "") + (.choices[0].message.reasoning // "") + (.choices[0].message.content // "")' "$response_file" 2>/dev/null | tr -d '\n' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//') > rm -f "$response_file" > > if [[-z "$content"]]; then > log " ${CYAN}Coherence check:${NC} ${RED}FAILED — empty response${NC}" > WARMUP_FAIL_REASON="EMPTY" > return 1 > fi > > # Detect repetition loop: split into words, check if any single word > # makes up >60% of total words (e.g. "n8n n8n n8n..." or "the the the...") > local word_count most_freq_count most_freq_word > word_count=$(echo "$content" | wc -w) > if [["$word_count" -ge 5]]; then > most_freq_word=$(echo "$content" | tr ' ' '\n' | sort | uniq -c | sort -rn | awk 'NR==1{print $2}') > most_freq_count=$(echo "$content" | tr ' ' '\n' | grep -cFx "$most_freq_word" 2>/dev/null || echo 0) > local pct=$(( most_freq_count * 100 / word_count )) > if [["$pct" -ge 60]]; then > log " ${CYAN}Coherence check:${NC} ${RED}FAILED — repetition loop (\"${most_freq_word}\" = ${pct}% of output)${NC}" > log " ${RED}Skipping benchmark — model output is incoherent.${NC}" > WARMUP_FAIL_REASON="INCOHERENT" > return 1 > fi > fi > > log " ${CYAN}Coherence check:${NC} \"${content:0:150}\"" > log " Model ready (loaded in ${elapsed}s)" > WARMUP_FAIL_REASON="" > return 0 > } > > # Run llama-benchy and capture results > run_benchy() { > local model="$1" > local safe_name="${model//\//_}" > local json_file="$RESULTS_DIR/${safe_name}_${TIMESTAMP}.json" > local md_file="$RESULTS_DIR/${safe_name}_${TIMESTAMP}.md" > > # Show profile description > case "$MODE" in > medium-log) log " Profile: Medium Log (50-page document baseline)" ;; > stress) log " Profile: Massive Log (stress test — watch for bandwidth bottleneck)" ;; > extreme) log " Profile: Extreme Limit (push to ~100 pages, crash/swap detection)" ;; > quick) log " Profile: Quick smoke test" ;; > full) log " Profile: Full comprehensive sweep" ;; > arena) log " Profile: Spark-Arena leaderboard (7 depths × 4 concurrency levels)" ;; > esac > local concurrency_display="" > [[-n "$CONCURRENCY"]] && concurrency_display=" concurrency=$CONCURRENCY" > log " Running llama-benchy (pp=$PP tg=$TG depth=$DEPTH runs=$RUNS${concurrency_display})..." > log "" > > # Build shared base flags (used by all runs) > local base_flags="" > base_flags+=" --base-url $LLAMA_SWAP_URL/v1" > base_flags+=" --model $model" > base_flags+=" --pp $PP" > base_flags+=" --tg $TG" > base_flags+=" --depth $DEPTH" > base_flags+=" --runs $RUNS" > base_flags+=" --latency-mode generation" > base_flags+=" --no-warmup" > base_flags+=" --skip-coherence" > [[-n "$CONCURRENCY"]] && base_flags+=" --concurrency $CONCURRENCY" > [["$MODE" == "arena"]] && base_flags+=" --enable-prefix-caching" > > # --- Run 1: Save JSON for data parsing --- > local cmd_json="${BENCHY_CMD}${base_flags} --save-result ${json_file} --format json" > > local output exit_code=0 > output=$(eval "$cmd_json" 2>&1) || exit_code=$? > > if [[$exit_code -ne 0]]; then > log " ${RED}llama-benchy failed:${NC}" > echo "$output" | tail -15 | tee -a "$REPORT_FILE" > return 1 > fi > > # --- Arena mode: save submission CSV (separate run, same params) --- > if [["$MODE" == "arena" && -n "$ARENA_DIR"]]; then > local arena_model_dir="$ARENA_DIR/${safe_name}" > mkdir -p "$arena_model_dir" > local csv_file="$arena_model_dir/results.csv" > local cmd_csv="${BENCHY_CMD}${base_flags} --save-result ${csv_file} --format csv" > log " ${CYAN}Saving arena submission CSV...${NC}" > local csv_output csv_exit=0 > csv_output=$(eval "$cmd_csv" 2>&1) || csv_exit=$? > if [[$csv_exit -ne 0]]; then > log " ${YELLOW}Warning: CSV run failed — JSON data still saved${NC}" > else > log " ${DIM}Arena CSV : $csv_file${NC}" > fi > fi > > # --- Run 2: Get the markdown table (for sharing on forums) --- > local cmd_md="${BENCHY_CMD}${base_flags} --save-result ${md_file} --format md" > > local md_output > md_output=$(eval "$cmd_md" 2>&1) || true > > # Show the full saved markdown file (the format people share on forums) > if [[-f "$md_file"]]; then > log " ${CYAN}llama-benchy results (copy this to share on forums):${NC}" > log "" > cat "$md_file" | tee -a "$REPORT_FILE" > log "" > else > # Fallback: show table lines from stdout if file wasn't created > local table_lines > table_lines=$(echo "$md_output" | grep -E '^\|') > if [[-n "$table_lines"]]; then > log " ${CYAN}llama-benchy results:${NC}" > log "" > echo "$table_lines" | tee -a "$REPORT_FILE" > log "" > fi > fi > > # --- Parse JSON and show friendly explanation --- > if [[-f "$json_file"]]; then > python3 < import json > > with open('$json_file') as f: > data = json.load(f) > > benchmarks = data.get('benchmarks', []) > if not benchmarks: > print(" (no benchmark data found)") > exit(0) > > # Check if all benchmarks have null results (model failed silently) > all_null = all( > b.get('pp_throughput') is None and b.get('tg_throughput') is None > for b in benchmarks > ) > if all_null: > print(" (model returned no usable results — it may not support this benchmark)") > exit(0) > > # Group benchmarks by depth for comparison > depth_results = {} > for b in benchmarks: > depth = b.get('context_size', 0) > pp_obj = b.get('pp_throughput') or {} > tg_obj = b.get('tg_throughput') or {} > pk_obj = b.get('peak_throughput') or {} > e2e_obj = b.get('e2e_ttft') or {} > pp_mean = pp_obj.get('mean', 0) or 0 > tg_mean = tg_obj.get('mean', 0) or 0 > if pp_mean > 0 or tg_mean > 0: > depth_results[depth] = { > 'pp_mean': pp_mean, 'pp_std': (pp_obj.get('std', 0) or 0), > 'tg_mean': tg_mean, 'tg_std': (tg_obj.get('std', 0) or 0), > 'pk_mean': (pk_obj.get('mean', 0) or 0), > 'e2e_mean': (e2e_obj.get('mean', 0) or 0), > } > > # Depth label mapping > depth_labels = { > 0: "Baseline (no context)", > 16384: "Medium Log (~50 pages)", > 32768: "Massive Log (~100 pages)", > 65535: "Extreme Limit (~200 pages)", > } > > print(" Context Depth Analysis:") > print("") > > baseline_tg = None > for depth in sorted(depth_results.keys()): > r = depth_results[depth] > label = depth_labels.get(depth, f"depth {depth}") > pp_mean, pp_std = r['pp_mean'], r['pp_std'] > tg_mean, tg_std = r['tg_mean'], r['tg_std'] > pk_mean = r['pk_mean'] > e2e_mean = r['e2e_mean'] > > print(f" --- depth={depth:,} — {label} ---") > > # Reading speed > if pp_mean > 0: > if pp_std > 0.5: > print(f" Reading speed: {pp_mean:,.0f} +/- {pp_std:,.0f} tok/s") > else: > print(f" Reading speed: {pp_mean:,.0f} tok/s") > > # Writing speed > if tg_mean > 0: > if tg_std > 0.5: > print(f" Writing speed: {tg_mean:.1f} +/- {tg_std:.1f} tok/s (peak: {pk_mean:.0f})") > else: > print(f" Writing speed: {tg_mean:.1f} tok/s (peak: {pk_mean:.0f})") > > # Show degradation from baseline > if baseline_tg is None: > baseline_tg = tg_mean > elif baseline_tg > 0: > pct = ((tg_mean - baseline_tg) / baseline_tg) * 100 > if pct < -15: > print(f" {chr(9888)} {pct:+.1f}% vs baseline — BANDWIDTH BOTTLENECK DETECTED") > elif pct < -5: > print(f" -> {pct:+.1f}% vs baseline (moderate slowdown)") > else: > print(f" -> {pct:+.1f}% vs baseline (minimal impact)") > > # What it feels like > if tg_mean >= 40: > feel = "Very fast — feels instant, smooth streaming" > elif tg_mean >= 25: > feel = "Fast — comfortable for interactive chat" > elif tg_mean >= 15: > feel = "Good — readable streaming with slight pauses" > elif tg_mean >= 8: > feel = "Moderate — noticeable wait, but usable" > else: > feel = "Slow — may feel sluggish for chat" > print(f" -> {feel}") > > # TTFT > if e2e_mean > 0: > print(f" Time to first token: {e2e_mean:.0f}ms") > if e2e_mean < 200: > print(f" -> Feels instant") > elif e2e_mean < 500: > print(f" -> Barely noticeable delay") > elif e2e_mean < 2000: > print(f" -> Short pause before response starts") > else: > print(f" -> Noticeable wait ({e2e_mean/1000:.1f}s)") > print("") > > # Summary comparison if multiple depths > if len(depth_results) > 1 and baseline_tg and baseline_tg > 0: > max_depth = max(depth_results.keys()) > deepest_tg = depth_results[max_depth]['tg_mean'] > total_pct = ((deepest_tg - baseline_tg) / baseline_tg) * 100 > print(f" Overall impact: depth 0 -> {max_depth:,} = {total_pct:+.1f}% generation speed") > if total_pct < -20: > print(f" Conclusion: Significant unified memory bandwidth bottleneck at depth {max_depth:,}") > elif total_pct < -10: > print(f" Conclusion: Moderate bandwidth pressure — usable but noticeably slower") > else: > print(f" Conclusion: Model handles deep context well on this hardware") > print("") > > PYEOF > fi > > log " ${DIM}JSON data : $json_file${NC}" > log " ${DIM}Forum table: $md_file${NC}" > return 0 > } > > # --------------------------------------------------------------------------- > # spark-arena-cli helpers > # --------------------------------------------------------------------------- > > # Download spark-arena-cli binary if it isn't already on PATH / ~/.local/bin > install_spark_arena_cli() { > if command -v spark-arena-cli &>/dev/null; then > SPARK_CLI="$(command -v spark-arena-cli)" > return 0 > fi > if [[-x "$SPARK_CLI"]]; then > return 0 > fi > local arch > arch=$(uname -m) > local bin_name > [["$arch" == "aarch64" || "$arch" == "arm64"]] \ > && bin_name="spark-arena-cli-0.1.0-linux-arm64" \ > || bin_name="spark-arena-cli-0.1.0-linux-amd64" > local url="https://github.com/spark-arena/spark-arena-cli/releases/download/v0.1.0/${bin_name}" > log " ${DIM}Downloading spark-arena-cli from GitHub releases...${NC}" > mkdir -p "$(dirname "$SPARK_CLI")" > if curl -fsSL "$url" -o "$SPARK_CLI" 2>/dev/null; then > chmod +x "$SPARK_CLI" > log " ${DIM}Installed to $SPARK_CLI${NC}" > # Add to PATH for this session > export PATH="$(dirname "$SPARK_CLI"):$PATH" > else > log " ${YELLOW}Warning: could not download spark-arena-cli — manual install needed${NC}" > SPARK_CLI="" > fi > } > > # Read our personal best tg tok/s (depth=0, concurrency=1) for a model from the history file > get_personal_best_tg() { > local model="$1" > if [[! -f "$ARENA_BEST_FILE"]]; then echo "0"; return; fi > python3 -c " > import json, sys > try: > d = json.load(open('$ARENA_BEST_FILE')) > entry = d.get('$model', {}) > print(entry.get('tg_mean', 0)) > except: > print(0) > " 2>/dev/null || echo "0" > } > > # Save current result as personal best for a model > save_personal_best() { > local model="$1" > local json_file="$2" > python3 - "$model" "$json_file" "$ARENA_BEST_FILE" <<'PYEOF' > import json, sys, os > model, bench_json, best_file = sys.argv[1], sys.argv[2], sys.argv[3] > > try: > data = json.load(open(bench_json)) > except Exception as e: > sys.exit(0) > > # Find depth=0, concurrency=1 entry > baseline = None > for b in data.get("benchmarks", []): > if b.get("context_size") == 0 and b.get("concurrency") == 1: > baseline = b > break > if not baseline: > # Fallback: first entry with context_size=0 > for b in data.get("benchmarks", []): > if b.get("context_size") == 0: > baseline = b > break > if not baseline: > sys.exit(0) > > tg = (baseline.get("tg_throughput") or {}).get("mean", 0) or 0 > pp = (baseline.get("pp_throughput") or {}).get("mean", 0) or 0 > e2e = (baseline.get("e2e_ttft") or {}).get("mean", 0) or 0 > > try: > best = json.load(open(best_file)) if os.path.exists(best_file) else {} > except: > best = {} > > best[model] = { > "tg_mean": round(tg, 2), > "pp_mean": round(pp, 1), > "ttft_ms": round(e2e, 1), > "timestamp": data.get("timestamp", ""), > "depth": 0, > "concurrency": 1, > } > with open(best_file, "w") as f: > json.dump(best, f, indent=2) > PYEOF > } > > # Compare current result vs personal best; if better, print submission info > check_and_suggest_submit() { > local model="$1" > local json_file="$2" > local recipe_file="$3" > > local prev_best > prev_best=$(get_personal_best_tg "$model") > > local current_tg > current_tg=$(python3 -c " > import json > data = json.load(open('$json_file')) > for b in data.get('benchmarks', []): > if b.get('context_size') == 0 and b.get('concurrency') == 1: > tg = (b.get('tg_throughput') or {}).get('mean', 0) or 0 > print(round(tg, 2)) > exit() > for b in data.get('benchmarks', []): > if b.get('context_size') == 0: > tg = (b.get('tg_throughput') or {}).get('mean', 0) or 0 > print(round(tg, 2)) > exit() > print(0) > " 2>/dev/null || echo "0") > > local is_better=0 > python3 -c "exit(0 if float('$current_tg') > float('$prev_best') else 1)" 2>/dev/null && is_better=1 > > if [["$is_better" -eq 1]]; then > if python3 -c "exit(0 if float('$prev_best') > 0 else 1)" 2>/dev/null; then > log " ${GREEN}★ NEW PERSONAL BEST${NC} — ${current_tg} tg tok/s (was ${prev_best})" > else > log " ${GREEN}★ FIRST ARENA RESULT${NC} — ${current_tg} tg tok/s @ depth=0 concurrency=1" > fi > save_personal_best "$model" "$json_file" > > # Try to get leaderboard context (best-effort scrape — may be empty) > log " ${CYAN}Submission candidate${NC} — consider submitting to spark-arena.com/leaderboard" > log "" > > install_spark_arena_cli > > if [[-n "$SPARK_CLI" && -x "$SPARK_CLI"]]; then > local is_logged_in=0 > "$SPARK_CLI" benchmark --help &>/dev/null && { > # Probe login state: spark-arena-cli prints a warning if not configured > local probe > probe=$(echo "exit" | timeout 3 "$SPARK_CLI" 2>&1 || true) > echo "$probe" | grep -q "Warning: Configuration not found" || is_logged_in=1 > } > > if [["$is_logged_in" -eq 1]]; then > log " ${GREEN}spark-arena-cli is logged in.${NC} Run this to submit officially:" > log " ${BOLD} $SPARK_CLI benchmark $recipe_file${NC}" > log " ${DIM} (This re-runs the benchmark via sparkrun and auto-uploads results)${NC}" > else > log " ${YELLOW}spark-arena-cli installed but not logged in.${NC} To submit:" > log " ${DIM} 1. $SPARK_CLI login # authenticate via Google/GitHub${NC}" > log " ${DIM} 2. $SPARK_CLI setup # configure sparkrun + llama-benchy${NC}" > log " ${DIM} 3. $SPARK_CLI benchmark $recipe_file${NC}" > fi > else > log " ${DIM}To submit to spark-arena, install spark-arena-cli:${NC}" > log " ${DIM} curl -fsSL https://github.com/spark-arena/spark-arena-cli/releases/download/v0.1.0/spark-arena-cli-0.1.0-linux-amd64 -o ~/.local/bin/spark-arena-cli && chmod +x ~/.local/bin/spark-arena-cli${NC}" > log " ${DIM} spark-arena-cli login${NC}" > log " ${DIM} spark-arena-cli benchmark $recipe_file${NC}" > fi > log "" > else > log " ${DIM}tg ${current_tg} tok/s — personal best is ${prev_best} tok/s (no improvement, skipping submission)${NC}" > fi > } > > # Generate a spark-arena recipe.yaml for a model > generate_recipe_yaml() { > local model="$1" > local safe_name="${model//\//_}" > local out_dir="$ARENA_DIR/${safe_name}" > mkdir -p "$out_dir" > > python3 - "$model" "$out_dir/recipe.yaml" <<'PYEOF' > import sys, textwrap > > model_name = sys.argv[1] > out_path = sys.argv[2] > > # Per-model recipe metadata. > # container: the local Docker image tag we actually use. > # hf_model: canonical HuggingFace model ID for the submission. > RECIPES = { > "Qwen3.5-35B-A3B-FP8": { > "hf_model": "Qwen/Qwen3.5-35B-A3B-Instruct", > "description": "Qwen3.5 35B MoE FP8-dynamic — reasoning + tool use with MTP-2 speculation", > "container": "vllm-node:Version_1", > "tp": 1, "gpu_mem": 0.7, "max_len": 131072, > "env": { > "VLLM_MARLIN_USE_ATOMIC_ADD": "1", > "VLLM_ENABLE_CUDAGRAPH_GC": "1", > "VLLM_USE_FLASHINFER_SAMPLER": "1", > }, > "extras": [ > "--kv-cache-dtype fp8", > "--load-format fastsafetensors", > "--attention-backend FLASHINFER", > "--enable-prefix-caching", > "--enable-chunked-prefill", > "--max-num-batched-tokens 4096", > '--speculative-config \'{"method":"mtp","num_speculative_tokens":2}\'', > "--enable-auto-tool-choice", > "--tool-call-parser qwen3_xml", > "--reasoning-parser qwen3", > ], > }, > "Qwen3.5-122B-A10B-int4-AutoRound": { > "hf_model": "Qwen/Qwen3.5-122B-A10B-Instruct", > "description": "Qwen3.5 122B MoE INT4 AutoRound — large hybrid reasoning model", > "container": "vllm-node-tf5:latest", > "tp": 1, "gpu_mem": 0.75, "max_len": 40960, > "env": {"VLLM_MARLIN_USE_ATOMIC_ADD": "1"}, > "extras": [ > "--trust-remote-code", > "--enforce-eager", > "--kv-cache-dtype fp8", > "--enable-prefix-caching", > "--enable-auto-tool-choice", > "--tool-call-parser qwen3_xml", > "--reasoning-parser qwen3", > ], > }, > "Qwen3-VL-30B-A3B-Instruct-FP8": { > "hf_model": "Qwen/Qwen3-VL-30B-A3B-Instruct", > "description": "Qwen3-VL 30B MoE FP8 — vision-language model", > "container": "spark-vllm:Version_1", > "tp": 1, "gpu_mem": 0.60, "max_len": 32768, > "env": {}, > "extras": [ > "--trust-remote-code", > "--kv-cache-dtype fp8", > "--load-format fastsafetensors", > "--enable-prefix-caching", > "--limit-mm-per-prompt '{\"image\": 2}'", > "--enable-auto-tool-choice", > "--tool-call-parser qwen3_coder", > ], > }, > "Qwen3-Omni-30B-A3B-Instruct": { > "hf_model": "Qwen/Qwen3-Omni-30B-A3B-Instruct", > "description": "Qwen3-Omni 30B MoE — audio + vision + text multimodal", > "container": "vllm-node:Version_1", > "tp": 1, "gpu_mem": 0.75, "max_len": 32768, > "env": {}, > "extras": [ > "--trust-remote-code", > "--load-format fastsafetensors", > "--enable-prefix-caching", > "--limit-mm-per-prompt '{\"image\": 2, \"audio\": 2}'", > "--enable-auto-tool-choice", > "--tool-call-parser qwen3_coder", > ], > }, > "Qwen3-Coder-Next-FP8-Dynamic": { > "hf_model": "Qwen/Qwen3-Coder-480B-A22B-FP8-Dynamic", > "description": "Qwen3-Coder-Next 480B MoE FP8-Dynamic — coding specialist", > "container": "vllm-node-tf5:latest", > "tp": 1, "gpu_mem": 0.75, "max_len": 32768, > "env": {}, > "extras": [ > "--kv-cache-dtype fp8", > "--load-format fastsafetensors", > "--attention-backend flashinfer", > "--enable-auto-tool-choice", > "--tool-call-parser qwen3_coder", > ], > }, > "Qwen3-Coder-Next-int4-AutoRound": { > "hf_model": "Qwen/Qwen3-Coder-480B-A22B-Instruct", > "description": "Qwen3-Coder-Next 480B MoE INT4 AutoRound — coding + throughput optimized", > "container": "vllm-node-tf5:latest", > "tp": 1, "gpu_mem": 0.6, "max_len": 32768, > "env": { > "VLLM_MARLIN_USE_ATOMIC_ADD": "1", > "VLLM_ALLOW_LONG_MAX_MODEL_LEN": "1", > "VLLM_USE_FLASHINFER_MOE_FP8": "1", > }, > "extras": [ > "--language-model-only", > "--enable-chunked-prefill", > "--max-num-batched-tokens 49152", > "--max-num-seqs 384", > "--kv-cache-dtype fp8", > "--load-format fastsafetensors", > "--optimization-level 3", > "--performance-mode throughput", > "--enable-auto-tool-choice", > "--tool-call-parser qwen3_coder", > ], > }, > "Nemotron-3-Nano-4B-FP8": { > "hf_model": "nvidia/Nemotron-3-Nano-4B-Instruct", > "description": "Nemotron-3-Nano 4B FP8 — ultra-fast orchestrator / routing model", > "container": "spark-vllm:Version_1", > "tp": 1, "gpu_mem": 0.5, "max_len": 8192, > "env": {}, > "extras": [ > "--kv-cache-dtype fp8", > "--enforce-eager", > "--trust-remote-code", > "--load-format fastsafetensors", > "--enable-prefix-caching", > "--tool-call-parser qwen3_coder", > "--reasoning-parser nemotron_v3", > "--enable-auto-tool-choice", > ], > }, > "Nemotron-3-Nano-30B-A3B-NVFP4": { > "hf_model": "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4", > "description": "Nemotron-3-Nano 30B NVFP4 — Blackwell-native MoE with nano_v3 reasoning", > "container": "vllm-node:Version_1", > "tp": 1, "gpu_mem": 0.65, "max_len": 131072, > "env": { > "VLLM_USE_FLASHINFER_MOE_FP4": "1", > "VLLM_FLASHINFER_MOE_BACKEND": "throughput", > }, > "extras": [ > "--kv-cache-dtype fp8", > "--enforce-eager", > "--trust-remote-code", > "--quantization modelopt_fp4", > "--enable-auto-tool-choice", > "--tool-call-parser qwen3_coder", > "--reasoning-parser nano_v3", > ], > }, > "Nemotron-3-Super-120B-A12B-NVFP4": { > "hf_model": "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4", > "description": "Nemotron-3-Super 120B NVFP4 — large reasoning model with CUTLASS MoE", > "container": "spark-vllm:Version_1", > "tp": 1, "gpu_mem": 0.7, "max_len": 65536, > "env": { > "VLLM_FLASHINFER_ALLREDUCE_BACKEND": "trtllm", > "VLLM_ALLOW_LONG_MAX_MODEL_LEN": "1", > }, > "extras": [ > "--kv-cache-dtype fp8", > "--moe-backend cutlass", > "--trust-remote-code", > "--enable-prefix-caching", > "--load-format fastsafetensors", > "--tool-call-parser qwen3_coder", > "--enable-auto-tool-choice", > "--reasoning-parser nemotron_v3", > ], > }, > "GPT-OSS-120B": { > "hf_model": "openai/gpt-oss-120b", > "description": "OpenAI GPT-OSS 120B MXFP4 — open-weights GPT model", > "container": "vllm-node-mxfp4", > "tp": 1, "gpu_mem": 0.7, "max_len": 65536, > "env": {"VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8": "1"}, > "extras": [ > "--quantization mxfp4", > "--kv-cache-dtype fp8", > "--max-num-batched-tokens 8192", > "--enable-prefix-caching", > "--load-format fastsafetensors", > "--tool-call-parser openai", > "--reasoning-parser openai_gptoss", > "--enable-auto-tool-choice", > ], > }, > "Mistral-Small-24B-Instruct-2501": { > "hf_model": "mistralai/Mistral-Small-24B-Instruct-2501", > "description": "Mistral Small 24B — fast roleplay and instruction following", > "container": "vllm-node:Version_1", > "tp": 1, "gpu_mem": 0.7, "max_len": 32768, > "env": {}, > "extras": [ > "--trust-remote-code", > "--enforce-eager", > "--enable-auto-tool-choice", > "--tool-call-parser mistral", > ], > }, > "Qwen3.5-35B-A3B-Uncensored-HauhauCS-Aggressive-Q4_K_M-GGUF": { > "hf_model": "HauhauCS/Qwen3.5-35B-A3B-Uncensored-Aggressive-GGUF", > "description": "Qwen3.5 35B MoE Q4_K_M GGUF — uncensored, llama.cpp serving", > "container": "ghcr.io/martin-b78/llama-cpp-spark:latest", > "tp": 1, "gpu_mem": 0.0, "max_len": 16384, > "env": {"GGML_CUDA_ENABLE_UNIFIED_MEMORY": "1"}, > "extras": [ > "--ctx-size 16384", > "--n-gpu-layers 99", > "--parallel 4", > "--no-mmap", > ], > "runtime": "llama-cpp", > }, > } > > r = RECIPES.get(model_name) > if not r: > # Unknown model — generate a minimal placeholder recipe > r = { > "hf_model": f"", > "description": model_name, > "container": "", > "tp": 1, "gpu_mem": 0.7, "max_len": 32768, > "env": {}, "extras": [], > } > > runtime = r.get("runtime", "vllm") > hf_model = r["hf_model"] > extras = "\n".join(f" {e} \\" for e in r["extras"]) > env_block = "" > if r["env"]: > env_lines = "\n".join(f" {k}: '{v}'" for k, v in r["env"].items()) > env_block = f"env:\n{env_lines}\n" > > if runtime == "llama-cpp": > cmd = ( > f"llama-server \\\n" > f" -hf {hf_model} \\\n" > f" --host {{host}} --port {{port}} \\\n" > + "\n".join(f" {e} \\" for e in r["extras"]) > + "\n" > ) > else: > cmd = ( > f"vllm serve {hf_model} \\\n" > f" --served-model-name {model_name} \\\n" > f" --host {{host}} --port {{port}} \\\n" > f" --tensor-parallel-size {{tensor_parallel}} \\\n" > f" --gpu-memory-utilization {{gpu_memory_utilization}} \\\n" > f" --max-model-len {{max_model_len}} \\\n" > + "\n".join(f" {e} \\" for e in r["extras"]) > + "\n" > ) > > yaml = f"""recipe_version: '1' > name: {model_name} > description: {r['description']} > model: {hf_model} > cluster_only: false > container: {r['container']} > defaults: > port: 8000 > host: 0.0.0.0 > tensor_parallel: {r['tp']} > gpu_memory_utilization: {r['gpu_mem']} > max_model_len: {r['max_len']} > {env_block}command: | > {cmd.rstrip()} > solo_only: false > """ > > with open(out_path, "w") as f: > f.write(yaml) > > print(f" Recipe YAML: {out_path}") > PYEOF > > if [[$? -ne 0]]; then > log " ${YELLOW}Warning: recipe.yaml generation failed for $model${NC}" > fi > } > > # --------------------------------------------------------------------------- > # Crash-resume checkpoint functions > # --------------------------------------------------------------------------- > > # Create a new checkpoint file for this session and register it as last-known > init_checkpoint() { > mkdir -p "$CHECKPOINT_DIR" > CHECKPOINT_FILE="$CHECKPOINT_DIR/session_${TIMESTAMP}.json" > python3 - "$CHECKPOINT_FILE" "$MODE" "$PP" "$TG" "$DEPTH" "$RUNS" <<'PYEOF' > import json, sys, datetime > f, mode, pp, tg, depth, runs = sys.argv[1:] > data = { > "session_id": f.split("_")[-1].replace(".json", ""), > "started_at": datetime.datetime.now().isoformat(), > "mode": mode, > "settings": {"pp": pp, "tg": tg, "depth": depth, "runs": int(runs)}, > "models": [], > } > with open(f, "w") as fh: > json.dump(data, fh, indent=2) > PYEOF > echo "$CHECKPOINT_FILE" > "$LAST_SESSION_FILE" > } > > # Mark a model as started (written BEFORE warmup so a crash is detectable) > checkpoint_model_start() { > local model="$1" > [[-z "$CHECKPOINT_FILE"]] && return > python3 - "$CHECKPOINT_FILE" "$model" <<'PYEOF' > import json, sys, datetime > f, model = sys.argv[1:] > with open(f) as fh: > data = json.load(fh) > data["models"].append({ > "model": model, > "status": "started", > "started_at": datetime.datetime.now().isoformat(), > }) > with open(f, "w") as fh: > json.dump(data, fh, indent=2) > PYEOF > } > > # Mark a model as completed with its result (OK / FAIL / INCOHERENT / EMPTY) > checkpoint_model_done() { > local model="$1" > local result="$2" > [[-z "$CHECKPOINT_FILE"]] && return > python3 - "$CHECKPOINT_FILE" "$model" "$result" <<'PYEOF' > import json, sys, datetime > f, model, result = sys.argv[1:] > with open(f) as fh: > data = json.load(fh) > for entry in reversed(data["models"]): > if entry["model"] == model and entry.get("status") == "started": > entry["status"] = "completed" > entry["result"] = result > entry["completed_at"] = datetime.datetime.now().isoformat() > break > with open(f, "w") as fh: > json.dump(data, fh, indent=2) > PYEOF > } > > # Load a previous checkpoint and populate SKIP_MODELS (models already done) > load_resume_checkpoint() { > if [[! -f "$LAST_SESSION_FILE"]]; then > echo -e "${RED}Error: no previous session found. Run without --resume first.${NC}" >&2 > exit 1 > fi > local prev_cp > prev_cp=$(cat "$LAST_SESSION_FILE") > if [[! -f "$prev_cp"]]; then > echo -e "${RED}Error: checkpoint file not found: $prev_cp${NC}" >&2 > exit 1 > fi > > log " ${CYAN}Resuming from checkpoint:${NC} $prev_cp" > > local info > info=$(python3 - "$prev_cp" <<'PYEOF' > import json, sys > data = json.load(open(sys.argv[1])) > completed = [e["model"] for e in data.get("models", []) if e.get("status") == "completed"] > last_started = next( > (e["model"] for e in reversed(data.get("models", [])) if e.get("status") == "started"), > "", > ) > s = data.get("settings", {}) > print("MODE:" + data.get("mode", "")) > print("SETTINGS:pp=" + str(s.get("pp","")) + " tg=" + str(s.get("tg","")) + > " depth=" + str(s.get("depth","")) + " runs=" + str(s.get("runs",""))) > print("LAST_STARTED:" + last_started) > for m in completed: > print("DONE:" + m) > PYEOF > ) > while IFS= read -r line; do > case "$line" in > DONE:*) SKIP_MODELS+=("${line#DONE:}") ;; > MODE:*) log " Previous mode : ${line#MODE:}" ;; > SETTINGS:*) log " Previous settings : ${line#SETTINGS:}" ;; > LAST_STARTED:*) > local last="${line#LAST_STARTED:}" > if [[-n "$last"]]; then > log " ${YELLOW}Crashed while running: $last${NC} — will re-run it." > fi > ;; > esac > done <<< "$info" > > log " Skipping ${#SKIP_MODELS[@]} already-completed model(s)." > log "" > } > > # --------------- MAIN --------------- > log "" > log "${BOLD}============================================================${NC}" > log "${BOLD} DGX Spark Model Benchmark${NC}" > log "${BOLD} powered by llama-benchy (github.com/eugr/llama-benchy)${NC}" > log "${BOLD}============================================================${NC}" > log "" > log " Endpoint : $LLAMA_SWAP_URL" > log " Mode : $MODE" > log " Settings : pp=$PP tg=$TG depth=$DEPTH runs=$RUNS" > log " Date : $(date '+%Y-%m-%d %H:%M')" > log " Results : $RESULTS_DIR" > log "" > > # Mode descriptions > case "$MODE" in > medium-log) > log " ${CYAN}Profile: Medium Log Baseline${NC}" > log " ${DIM}Simulates a ~50-page document (depth=16384). Establishes your${NC}" > log " ${DIM}baseline generation speed with a moderately full KV cache.${NC}" > ;; > stress) > log " ${CYAN}Profile: Massive Log Stress Test${NC}" > log " ${DIM}Doubles context to simulate a massive error log (depth=32768).${NC}" > log " ${DIM}Watch for tg tok/s drop — that's the unified memory bottleneck.${NC}" > ;; > extreme) > log " ${CYAN}Profile: Extreme Limit Test${NC}" > log " ${DIM}Pushes to ~200 pages (depth=65535). Tests if the system can${NC}" > log " ${DIM}process it without crashing or heavy swap paging.${NC}" > ;; > quick) > log " ${DIM}Quick smoke test — just checking if models respond.${NC}" > ;; > full) > log " ${DIM}Full comprehensive sweep — broad pp/tg/depth combinations.${NC}" > ;; > arena) > log " ${CYAN}Profile: Spark-Arena Leaderboard Submission${NC}" > log " ${DIM}Official spark-arena.com benchmark spec: 7 depth levels × 4 concurrency${NC}" > log " ${DIM}levels × 3 runs = 84 data points per model.${NC}" > log " ${DIM}Generates results.csv + recipe.yaml per model in:${NC}" > log " ${DIM} $ARENA_DIR${NC}" > ;; > esac > log "" > log " ${DIM}pp = prompt processing (how fast the model reads your input)${NC}" > log " ${DIM}tg = token generation (how fast the model writes its reply)${NC}" > log " ${DIM}depth = pre-filled context tokens (simulates document size)${NC}" > log "" > > # Initialize crash-resume checkpoint and, if --resume, load previous state > init_checkpoint > [["$RESUME" == true]] && load_resume_checkpoint > > # Detect how llama-benchy is available: uvx (preferred), direct, or missing > if uvx llama-benchy --help > /dev/null 2>&1; then > BENCHY_CMD="uvx llama-benchy" > elif command -v llama-benchy > /dev/null 2>&1; then > BENCHY_CMD="llama-benchy" > else > log "${RED}Error: llama-benchy not found.${NC}" > log "" > log "Install one of these ways:" > log " ${BOLD}uvx${NC} (no install needed): works if 'uv' is installed" > log " ${BOLD}pip install llama-benchy${NC} (install into current venv/system)" > log " ${BOLD}uv pip install llama-benchy${NC}" > exit 1 > fi > log " ${DIM}Using: $BENCHY_CMD${NC}" > > # Fetch model list > MODELS=$(curl -sf "$LLAMA_SWAP_URL/v1/models" | jq -r '.data[].id' | sort) > MODEL_COUNT=$(echo "$MODELS" | wc -l) > > # Apply filters > if [[${#FILTERS[@]} -gt 0 ]]; then > log "Filtering models matching: ${FILTERS[*]}" > FILTERED="" > for m in $MODELS; do > for f in "${FILTERS[@]}"; do > if [["$m" == *"$f"*]]; then > FILTERED="${FILTERED}${m}\n" > fi > done > done > MODELS=$(echo -e "$FILTERED" | grep -v '^$' | sort -u) > MODEL_COUNT=$(echo "$MODELS" | wc -l) > fi > > log "Found ${BOLD}${MODEL_COUNT}${NC} model(s) to benchmark." > log "" > > PASS=0 > FAIL=0 > IDX=0 > TOTAL_START=$(date +%s.%N) > > # Collect results for final summary > declare -A SUMMARY_PP SUMMARY_TG SUMMARY_PEAK SUMMARY_TTFT SUMMARY_STATUS SUMMARY_DEGRADATION > > for MODEL in $MODELS; do > IDX=$((IDX + 1)) > > # Skip models already completed in a --resume session > if [["$RESUME" == true]]; then > _skip=false > for _done in "${SKIP_MODELS[@]:-}"; do > [["$MODEL" == "$_done"]] && { _skip=true; break; } > done > if [["$_skip" == true]]; then > log " ${DIM}⏭ [$IDX/$MODEL_COUNT] $MODEL — skipped (completed in previous session)${NC}" > continue > fi > fi > > log "${BOLD}============================================================${NC}" > log "${BOLD} [$IDX/$MODEL_COUNT] $MODEL${NC}" > log "${BOLD}============================================================${NC}" > log "" > > # Write checkpoint BEFORE starting so a crash is detectable > checkpoint_model_start "$MODEL" > > # Unload previous model to get a clean measurement > unload_all > > # Load this model > WARMUP_FAIL_REASON="" > if ! warmup_model "$MODEL"; then > FAIL=$((FAIL + 1)) > SUMMARY_STATUS[$MODEL]="${WARMUP_FAIL_REASON:-FAIL}" > checkpoint_model_done "$MODEL" "${WARMUP_FAIL_REASON:-FAIL}" > log "" > continue > fi > > # Benchmark it > if run_benchy "$MODEL"; then > PASS=$((PASS + 1)) > SUMMARY_STATUS[$MODEL]="OK" > checkpoint_model_done "$MODEL" "OK" > # In arena mode: generate recipe YAML and check vs personal best > if [["$MODE" == "arena"]]; then > generate_recipe_yaml "$MODEL" 2>&1 | tee -a "$REPORT_FILE" > local arena_json="$RESULTS_DIR/${MODEL//\//_}_${TIMESTAMP}.json" > local arena_recipe="$ARENA_DIR/${MODEL//\//_}/recipe.yaml" > check_and_suggest_submit "$MODEL" "$arena_json" "$arena_recipe" 2>&1 | tee -a "$REPORT_FILE" > fi > > # Extract numbers for final summary from JSON > local_json="$RESULTS_DIR/${MODEL//\//_}_${TIMESTAMP}.json" > if [[-f "$local_json"]]; then > # Extract metrics using jq for the summary table > # We look for depth 0 (baseline) and the deepest result > baseline_tg=$(jq -r '.benchmarks[] | select(.context_size == 0) | .tg_throughput.mean // empty' "$local_json" | head -n1) > max_depth=$(jq -r '.benchmarks[].context_size' "$local_json" | sort -rn | head -n1) > deepest_tg=$(jq -r ".benchmarks[] | select(.context_size == $max_depth) | .tg_throughput.mean // empty" "$local_json" | head -n1) > > # Baseline metrics for the summary table > SUMMARY_PP[$MODEL]=$(jq -r '.benchmarks[] | select(.context_size == 0) | if .pp_throughput.std > 0.5 then "\(.pp_throughput.mean + 0.5 | floor) +/-\(.pp_throughput.std + 0.5 | floor)" else "\(.pp_throughput.mean + 0.5 | floor)" end' "$local_json" | head -n1) > SUMMARY_TG[$MODEL]=$(jq -r '.benchmarks[] | select(.context_size == 0) | if .tg_throughput.std > 0.5 then "\((.tg_throughput.mean * 10 + 0.5 | floor) / 10) +/-\((.tg_throughput.std * 10 + 0.5 | floor) / 10)" else "\((.tg_throughput.mean * 10 + 0.5 | floor) / 10)" end' "$local_json" | head -n1) > SUMMARY_PEAK[$MODEL]=$(jq -r '.benchmarks[] | select(.context_size == 0) | .peak_throughput.mean + 0.5 | floor' "$local_json" | head -n1) > SUMMARY_TTFT[$MODEL]=$(jq -r '.benchmarks[] | select(.context_size == 0) | .e2e_ttft.mean + 0.5 | floor' "$local_json" | head -n1) > > if [[-n "$baseline_tg" && -n "$deepest_tg" && "$max_depth" -gt 0]]; then > pct=$(echo "scale=1; (($deepest_tg - $baseline_tg) / $baseline_tg) * 100" | bc) > SUMMARY_DEGRADATION[$MODEL]="${pct}% @$((max_depth/1024))k" > fi > fi > else > FAIL=$((FAIL + 1)) > SUMMARY_STATUS[$MODEL]="FAIL" > checkpoint_model_done "$MODEL" "FAIL" > fi > log "" > done > > # Final cleanup > unload_all > > TOTAL_END=$(date +%s.%N) > TOTAL_ELAPSED=$(echo "$TOTAL_END - $TOTAL_START" | bc) > TOTAL_MIN=$(echo "scale=1; $TOTAL_ELAPSED / 60" | bc) > > # ============================================= > # FINAL REPORT > # ============================================= > log "" > log "${BOLD}============================================================${NC}" > log "${BOLD} BENCHMARK REPORT — DGX Spark${NC}" > log "${BOLD}============================================================${NC}" > log "" > log " Date: $(date '+%Y-%m-%d %H:%M') | Mode: $MODE | Runs: $RUNS" > log " Depths tested: $DEPTH" > log " Models tested: $MODEL_COUNT | Passed: ${GREEN}$PASS${NC} Failed: ${RED}$FAIL${NC}" > log " Total benchmark time: ${TOTAL_MIN} min" > log " Checkpoint : $CHECKPOINT_FILE" > [[$FAIL -gt 0]] && log " ${YELLOW}Tip: if this was interrupted, resume with: ./benchmark-models.sh --resume${NC}" > log "" > log " ${BOLD}$(printf '%-42s %14s %12s %8s %8s %14s' 'Model' 'Read (pp)' 'Write (tg)' 'Peak' 'TTFT' 'Deep ctx')${NC}" > log " ${DIM}$(printf '%-42s %14s %12s %8s %8s %14s' '' 'tok/s' 'tok/s' 'tok/s' 'ms' 'degradation')${NC}" > log " $(printf '%.0s-' {1..106})" > > for MODEL in $MODELS; do > local_name="$MODEL" > [[${#local_name} -gt 42]] && local_name="${local_name:0:39}..." > > status="${SUMMARY_STATUS[$MODEL]:-FAIL}" > pp="${SUMMARY_PP[$MODEL]:-—}" > tg="${SUMMARY_TG[$MODEL]:-—}" > peak="${SUMMARY_PEAK[$MODEL]:-—}" > ttft="${SUMMARY_TTFT[$MODEL]:-—}" > degrad="${SUMMARY_DEGRADATION[$MODEL]:-—}" > > if [["$status" == "OK"]]; then > printf -v line " %-42s %14s %12s %8s %8s %14s" "$local_name" "$pp" "$tg" "$peak" "$ttft" "$degrad" > log "${GREEN}${line}${NC}" > elif [["$status" == "INCOHERENT"]]; then > printf -v line " %-42s %14s %12s %8s %8s %14s" "$local_name" "INCOHERENT" "—" "—" "—" "—" > log "${YELLOW}${line}${NC}" > else > printf -v line " %-42s %14s %12s %8s %8s %14s" "$local_name" "FAIL" "—" "—" "—" "—" > log "${RED}${line}${NC}" > fi > done > > log "" > log " ${DIM}---------------------------------------------------------------${NC}" > log " ${DIM}How to read this table:${NC}" > log " ${DIM}${NC}" > log " ${DIM} Read (pp) = How fast the model reads your prompt (higher = better).${NC}" > log " ${DIM} Write (tg) = How fast the model types its answer at depth=0 baseline.${NC}" > log " ${DIM} This is the speed you feel when chatting.${NC}" > log " ${DIM} Humans read at ~4 tok/s, so 20+ feels smooth.${NC}" > log " ${DIM} Peak = Fastest burst speed observed in a 1-second window.${NC}" > log " ${DIM} TTFT = Time until the first word appears (lower = better).${NC}" > log " ${DIM} Deep ctx = Speed change at max tested depth vs baseline.${NC}" > log " ${DIM} >-15% = unified memory bandwidth bottleneck.${NC}" > log " ${DIM}---------------------------------------------------------------${NC}" > log "" > log " Results saved to:" > log " Report : $REPORT_FILE" > log " JSON data : $RESULTS_DIR/*_${TIMESTAMP}.json" > log " Forum tables: $RESULTS_DIR/*_${TIMESTAMP}.md" > log "" > log " ${DIM}Tip: To share on NVIDIA forums, copy the llama-benchy tables${NC}" > log " ${DIM}from the .md files — they use the standard format everyone knows.${NC}" > > if [["$MODE" == "arena" && -n "$ARENA_DIR"]]; then > log "" > log "${BOLD}============================================================${NC}" > log "${BOLD} SPARK-ARENA SUBMISSION FILES${NC}" > log "${BOLD}============================================================${NC}" > log "" > log " Submission directory: ${CYAN}$ARENA_DIR${NC}" > log "" > log " Per-model folders (one submission per model):" > for MODEL in $MODELS; do > safe="${MODEL//\//_}" > model_dir="$ARENA_DIR/$safe" > if [[-f "$model_dir/results.csv" && -f "$model_dir/recipe.yaml"]]; then > log " ${GREEN}✓${NC} $MODEL" > log " recipe.yaml : $model_dir/recipe.yaml" > log " results.csv : $model_dir/results.csv" > elif [["${SUMMARY_STATUS[$MODEL]:-FAIL}" != "OK" ]]; then > log " ${RED}✗${NC} $MODEL (benchmark failed — no submission files)" > fi > done > log "" > log " ${BOLD}How to submit to spark-arena.com/admin:${NC}" > log " ${DIM}1. Open https://spark-arena.com/admin${NC}" > log " ${DIM}2. For each model folder above:${NC}" > log " ${DIM} a. Paste or upload the contents of recipe.yaml${NC}" > log " ${DIM} b. Upload results.csv${NC}" > log " ${DIM}3. Submit one entry per model.${NC}" > log "" > log " ${DIM}Note: The recipe.yaml 'model:' field uses the canonical HuggingFace ID.${NC}" > log " ${DIM}If your model was downloaded from a different source, update it.${NC}" > fi > log "${BOLD}============================================================${NC}" > > ``` ---
**Author:** ![martinB78](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/martinb78/32/437495_2.png) [@martinB78](https://forums.developer.nvidia.com/u/martinB78)\ **Post date:** [April 21, 2026, 7:28am UTC](https://forums.developer.nvidia.com/t/new-tool-llama-benchy-llama-bench-style-benchmarking-for-any-llm-backend-vllm-sglang-llama-cpp-etc/356698/17 "2026-04-21T07:28:01Z")
Ok I did the first testrun - took about 8h here is the result. Will upload the files to [https://spark-arena.com](https://spark-arena.com) ```auto e[1m============================================================e[0m e[1m DGX Spark Model Benchmarke[0m e[1m powered by llama-benchy (github.com/eugr/llama-benchy)e[0m e[1m============================================================e[0m Endpoint : http://localhost:28080 Mode : arena Settings : pp=2048 tg=128 depth=0 4096 8192 16384 32768 65535 100000 runs=3 Date : 2026-04-20 18:38 Results : /home/sparky/Docker/dgx-spark_lite-llm_llama-swap_vllm_llama-cpp_ollama/test-results/benchmarks e[0;36mProfile: Spark-Arena Leaderboard Submissione[0m e[2mOfficial spark-arena.com benchmark spec: 7 depth levels × 4 concurrencye[0m e[2mlevels × 3 runs = 84 data points per model.e[0m e[2mGenerates results.csv + recipe.yaml per model in:e[0m e[2m /home/sparky/Docker/dgx-spark_lite-llm_llama-swap_vllm_llama-cpp_ollama/test-results/arena-submission/20260420_183825e[0m e[2mpp = prompt processing (how fast the model reads your input)e[0m e[2mtg = token generation (how fast the model writes its reply)e[0m e[2mdepth = pre-filled context tokens (simulates document size)e[0m Filtering models matching: GPT-OSS-120B Found e[1m1e[0m model(s) to benchmark. e[1m============================================================e[0m e[1m [1/1] GPT-OSS-120Be[0m e[1m============================================================e[0m Loading model via llama-swap... e[0;36mCoherence check:e[0m "The user asks: "Write a short python hello world script." Simple answer: Provide a small Python script that prints "Hello, World!". Possibly include a" Model ready (loaded in 1.600343363s) Profile: Spark-Arena leaderboard (7 depths × 4 concurrency levels) Running llama-benchy (pp=2048 tg=128 depth=0 4096 8192 16384 32768 65535 100000 runs=3 concurrency=1 2 5 10)... e[0;36mSaving arena submission CSV...e[0m e[2mArena CSV : /home/sparky/Docker/dgx-spark_lite-llm_llama-swap_vllm_llama-cpp_ollama/test-results/arena-submission/20260420_183825/GPT-OSS-120B/results.csve[0m e[0;36mllama-benchy results (copy this to share on forums):e[0m | model | test | t/s (total) | t/s (req) | peak t/s | peak t/s (req) | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:-------------|-----------------------:|-----------------:|------------------:|--------------:|-----------------:|----------------------:|----------------------:|----------------------:| | GPT-OSS-120B | pp2048 (c1) | 3947.98 ± 181.15 | 3947.98 ± 181.15 | | | 531.89 ± 19.05 | 455.64 ± 19.05 | 634.40 ± 21.92 | | GPT-OSS-120B | tg128 (c1) | 30.40 ± 0.14 | 30.40 ± 0.14 | 31.33 ± 0.47 | 31.33 ± 0.47 | | | | | GPT-OSS-120B | pp2048 (c2) | 3101.86 ± 47.23 | 2927.56 ± 1067.90 | | | 785.69 ± 261.70 | 709.44 ± 261.70 | 1141.88 ± 19.79 | | GPT-OSS-120B | tg128 (c2) | 53.38 ± 0.57 | 26.89 ± 0.28 | 56.33 ± 0.47 | 28.17 ± 0.37 | | | | | GPT-OSS-120B | pp2048 (c5) | 4058.86 ± 254.53 | 1385.24 ± 1038.82 | | | 1863.99 ± 618.78 | 1787.74 ± 618.78 | 2284.64 ± 158.14 | | GPT-OSS-120B | tg128 (c5) | 85.82 ± 0.93 | 17.27 ± 0.18 | 100.33 ± 4.11 | 20.07 ± 0.85 | | | | | GPT-OSS-120B | pp2048 (c10) | 4570.57 ± 10.53 | 982.42 ± 1014.99 | | | 2856.45 ± 1077.50 | 2780.20 ± 1077.50 | 3953.18 ± 65.14 | | GPT-OSS-120B | tg128 (c10) | 126.44 ± 3.49 | 12.79 ± 0.36 | 153.33 ± 2.36 | 15.33 ± 0.47 | | | | | GPT-OSS-120B | ctx_pp @ d4096 (c1) | 4253.76 ± 6.71 | 4253.76 ± 6.71 | | | 921.66 ± 26.31 | 845.41 ± 26.31 | 1027.30 ± 25.76 | | GPT-OSS-120B | ctx_tg @ d4096 (c1) | 29.27 ± 1.08 | 29.27 ± 1.08 | 30.67 ± 0.47 | 30.67 ± 0.47 | | | | | GPT-OSS-120B | pp2048 @ d4096 (c1) | 3723.75 ± 82.19 | 3723.75 ± 82.19 | | | 626.50 ± 12.06 | 550.25 ± 12.06 | 731.18 ± 12.20 | | GPT-OSS-120B | tg128 @ d4096 (c1) | 29.61 ± 0.14 | 29.61 ± 0.14 | 30.33 ± 0.47 | 30.33 ± 0.47 | | | | | GPT-OSS-120B | ctx_pp @ d4096 (c2) | 3926.50 ± 223.05 | 2923.58 ± 974.45 | | | 1440.51 ± 381.63 | 1364.26 ± 381.63 | 1837.70 ± 97.13 | | GPT-OSS-120B | ctx_tg @ d4096 (c2) | 50.95 ± 2.56 | 25.61 ± 1.42 | 55.33 ± 0.94 | 27.67 ± 0.47 | | | | | GPT-OSS-120B | pp2048 @ d4096 (c2) | 2954.37 ± 68.92 | 2673.38 ± 969.88 | | | 958.21 ± 319.99 | 881.96 ± 319.99 | 1370.17 ± 37.20 | | GPT-OSS-120B | tg128 @ d4096 (c2) | 52.40 ± 0.40 | 26.39 ± 0.20 | 55.33 ± 0.94 | 27.67 ± 0.47 | | | | | GPT-OSS-120B | ctx_pp @ d4096 (c5) | 3959.01 ± 171.25 | 1530.72 ± 948.16 | | | 3122.99 ± 1215.13 | 3046.74 ± 1215.13 | 4519.15 ± 199.15 | | GPT-OSS-120B | ctx_tg @ d4096 (c5) | 82.88 ± 0.91 | 16.78 ± 0.18 | 94.33 ± 0.94 | 18.87 ± 0.34 | | | | | GPT-OSS-120B | pp2048 @ d4096 (c5) | 3508.96 ± 310.49 | 1345.96 ± 929.56 | | | 2180.92 ± 901.07 | 2104.67 ± 901.07 | 2923.07 ± 269.42 | | GPT-OSS-120B | tg128 @ d4096 (c5) | 81.88 ± 0.99 | 16.51 ± 0.22 | 94.00 ± 4.32 | 18.80 ± 0.91 | | | | | GPT-OSS-120B | ctx_pp @ d4096 (c10) | 4219.29 ± 90.49 | 1055.24 ± 967.79 | | | 5357.62 ± 2501.59 | 5281.37 ± 2501.59 | 8267.47 ± 589.06 | | GPT-OSS-120B | ctx_tg @ d4096 (c10) | 101.29 ± 7.74 | 11.83 ± 0.70 | 136.67 ± 4.71 | 13.97 ± 0.18 | | | | | GPT-OSS-120B | pp2048 @ d4096 (c10) | 3872.15 ± 99.58 | 876.29 ± 877.93 | | | 3627.64 ± 1456.42 | 3551.39 ± 1456.42 | 5227.85 ± 119.70 | | GPT-OSS-120B | tg128 @ d4096 (c10) | 113.68 ± 1.25 | 11.54 ± 0.18 | 140.00 ± 0.00 | 14.03 ± 0.18 | | | | | GPT-OSS-120B | ctx_pp @ d8192 (c1) | 4119.74 ± 22.64 | 4119.74 ± 22.64 | | | 1854.50 ± 16.28 | 1778.25 ± 16.28 | 1963.30 ± 11.53 | | GPT-OSS-120B | ctx_tg @ d8192 (c1) | 28.67 ± 0.52 | 28.67 ± 0.52 | 30.00 ± 0.00 | 30.00 ± 0.00 | | | | | GPT-OSS-120B | pp2048 @ d8192 (c1) | 2914.02 ± 95.84 | 2914.02 ± 95.84 | | | 779.81 ± 22.82 | 703.56 ± 22.82 | 886.68 ± 25.42 | | GPT-OSS-120B | tg128 @ d8192 (c1) | 28.66 ± 0.39 | 28.66 ± 0.39 | 30.00 ± 0.00 | 30.00 ± 0.00 | | | | | GPT-OSS-120B | ctx_pp @ d8192 (c2) | 3721.07 ± 180.93 | 2919.03 ± 988.26 | | | 2876.38 ± 942.55 | 2800.13 ± 942.55 | 3888.80 ± 190.13 | | GPT-OSS-120B | ctx_tg @ d8192 (c2) | 50.34 ± 0.82 | 25.35 ± 0.41 | 54.00 ± 0.00 | 27.00 ± 0.00 | | | | | GPT-OSS-120B | pp2048 @ d8192 (c2) | 2703.94 ± 229.07 | 2055.30 ± 668.13 | | | 1170.32 ± 303.39 | 1094.07 ± 303.39 | 1512.09 ± 113.92 | | GPT-OSS-120B | tg128 @ d8192 (c2) | 50.81 ± 0.15 | 25.54 ± 0.02 | 54.00 ± 0.00 | 27.00 ± 0.00 | | | | | GPT-OSS-120B | ctx_pp @ d8192 (c5) | 3817.51 ± 61.91 | 1775.68 ± 1200.80 | | | 5824.30 ± 2715.35 | 5748.05 ± 2715.35 | 9135.03 ± 562.33 | | GPT-OSS-120B | ctx_tg @ d8192 (c5) | 67.24 ± 0.44 | 15.41 ± 0.87 | 85.00 ± 0.00 | 17.13 ± 0.34 | | | | | GPT-OSS-120B | pp2048 @ d8192 (c5) | 2959.28 ± 255.91 | 1023.15 ± 637.72 | | | 2649.12 ± 990.25 | 2572.87 ± 990.25 | 3466.69 ± 317.09 | | GPT-OSS-120B | tg128 @ d8192 (c5) | 79.18 ± 1.02 | 15.97 ± 0.19 | 92.67 ± 1.70 | 18.53 ± 0.50 | | | | | GPT-OSS-120B | ctx_pp @ d8192 (c10) | 3900.45 ± 54.70 | 1128.22 ± 1080.85 | | | 10850.66 ± 5637.60 | 10774.41 ± 5637.60 | 15268.65 ± 3789.88 | | GPT-OSS-120B | ctx_tg @ d8192 (c10) | 56.59 ± 0.92 | 9.14 ± 2.01 | 126.67 ± 4.71 | 12.93 ± 0.63 | | | | | GPT-OSS-120B | pp2048 @ d8192 (c10) | 3209.98 ± 9.62 | 757.23 ± 762.36 | | | 4200.36 ± 1749.65 | 4124.10 ± 1749.65 | 6283.87 ± 61.18 | | GPT-OSS-120B | tg128 @ d8192 (c10) | 106.44 ± 1.48 | 10.81 ± 0.16 | 137.00 ± 3.56 | 13.77 ± 0.42 | | | | | GPT-OSS-120B | ctx_pp @ d16384 (c1) | 3208.08 ± 149.31 | 3208.08 ± 149.31 | | | 4606.14 ± 211.86 | 4529.89 ± 211.86 | 4716.57 ± 209.22 | | GPT-OSS-120B | ctx_tg @ d16384 (c1) | 27.83 ± 0.09 | 27.83 ± 0.09 | 28.67 ± 0.47 | 28.67 ± 0.47 | | | | | GPT-OSS-120B | pp2048 @ d16384 (c1) | 2295.21 ± 113.84 | 2295.21 ± 113.84 | | | 970.78 ± 45.13 | 894.53 ± 45.13 | 1085.98 ± 42.79 | | GPT-OSS-120B | tg128 @ d16384 (c1) | 27.94 ± 0.12 | 27.94 ± 0.12 | 28.67 ± 0.47 | 28.67 ± 0.47 | | | | | GPT-OSS-120B | ctx_pp @ d16384 (c2) | 3167.92 ± 38.67 | 2322.68 ± 694.85 | | | 6923.30 ± 2079.76 | 6847.05 ± 2079.76 | 9087.10 ± 136.80 | | GPT-OSS-120B | ctx_tg @ d16384 (c2) | 45.89 ± 0.83 | 23.28 ± 0.42 | 50.67 ± 0.94 | 25.33 ± 0.47 | | | | | GPT-OSS-120B | pp2048 @ d16384 (c2) | 1989.58 ± 131.27 | 1521.26 ± 576.35 | | | 1597.90 ± 466.71 | 1521.65 ± 466.71 | 2054.69 ± 133.71 | | GPT-OSS-120B | tg128 @ d16384 (c2) | 47.64 ± 0.42 | 23.94 ± 0.28 | 50.00 ± 0.00 | 25.00 ± 0.00 | | | | | GPT-OSS-120B | ctx_pp @ d16384 (c5) | 3141.31 ± 33.66 | 1375.85 ± 876.51 | | | 14368.87 ± 6468.58 | 14292.62 ± 6468.58 | 19893.51 ± 4133.10 | | GPT-OSS-120B | ctx_tg @ d16384 (c5) | 32.68 ± 0.82 | 11.83 ± 3.29 | 78.33 ± 2.36 | 16.07 ± 0.68 | | | | | GPT-OSS-120B | pp2048 @ d16384 (c5) | 2371.31 ± 42.45 | 850.51 ± 697.67 | | | 3523.64 ± 1269.28 | 3447.39 ± 1269.28 | 4308.09 ± 80.52 | | GPT-OSS-120B | tg128 @ d16384 (c5) | 71.13 ± 0.94 | 14.32 ± 0.26 | 83.67 ± 1.89 | 16.73 ± 0.44 | | | | | GPT-OSS-120B | ctx_pp @ d16384 (c10) | 3136.56 ± 38.93 | 862.88 ± 760.58 | | | 26198.68 ± 13026.42 | 26122.43 ± 13026.42 | 32921.75 ± 11570.45 | | GPT-OSS-120B | ctx_tg @ d16384 (c10) | 26.83 ± 0.40 | 5.96 ± 2.70 | 103.67 ± 4.50 | 11.93 ± 1.71 | | | | | GPT-OSS-120B | pp2048 @ d16384 (c10) | 2389.99 ± 33.59 | 574.34 ± 595.16 | | | 5791.76 ± 2479.52 | 5715.51 ± 2479.52 | 8469.52 ± 134.09 | | GPT-OSS-120B | tg128 @ d16384 (c10) | 96.43 ± 1.35 | 9.88 ± 0.14 | 120.00 ± 0.00 | 12.00 ± 0.00 | | | | | GPT-OSS-120B | ctx_pp @ d32768 (c1) | 2265.54 ± 91.99 | 2265.54 ± 91.99 | | | 12934.07 ± 558.65 | 12857.82 ± 558.65 | 13051.20 ± 561.44 | | GPT-OSS-120B | ctx_tg @ d32768 (c1) | 25.61 ± 0.43 | 25.61 ± 0.43 | 27.00 ± 0.00 | 27.00 ± 0.00 | | | | | GPT-OSS-120B | pp2048 @ d32768 (c1) | 1541.87 ± 45.41 | 1541.87 ± 45.41 | | | 1405.65 ± 38.84 | 1329.40 ± 38.84 | 1520.58 ± 41.31 | | GPT-OSS-120B | tg128 @ d32768 (c1) | 26.04 ± 0.11 | 26.04 ± 0.11 | 27.00 ± 0.00 | 27.00 ± 0.00 | | | | | GPT-OSS-120B | ctx_pp @ d32768 (c2) | 2307.66 ± 18.85 | 1670.49 ± 509.79 | | | 19237.48 ± 5796.01 | 19161.23 ± 5796.01 | 24789.07 ± 452.69 | | GPT-OSS-120B | ctx_tg @ d32768 (c2) | 35.17 ± 1.60 | 19.15 ± 1.53 | 42.67 ± 0.94 | 21.83 ± 1.07 | | | | | GPT-OSS-120B | pp2048 @ d32768 (c2) | 1363.74 ± 127.63 | 931.47 ± 289.18 | | | 2456.81 ± 604.13 | 2380.56 ± 604.13 | 3017.05 ± 294.17 | | GPT-OSS-120B | tg128 @ d32768 (c2) | 41.95 ± 0.17 | 21.07 ± 0.08 | 44.67 ± 0.94 | 22.33 ± 0.47 | | | | | GPT-OSS-120B | ctx_pp @ d32768 (c5) | 2309.29 ± 21.36 | 1008.59 ± 622.29 | | | 38716.88 ± 17482.64 | 38640.63 ± 17482.64 | 47536.88 ± 14755.77 | | GPT-OSS-120B | ctx_tg @ d32768 (c5) | 12.65 ± 0.22 | 7.06 ± 4.10 | 65.00 ± 0.00 | 14.27 ± 1.57 | | | | | GPT-OSS-120B | pp2048 @ d32768 (c5) | 1554.37 ± 9.47 | 564.05 ± 351.33 | | | 4821.69 ± 1958.83 | 4745.44 ± 1958.83 | 6549.27 ± 66.56 | | GPT-OSS-120B | tg128 @ d32768 (c5) | 63.59 ± 0.57 | 12.86 ± 0.10 | 74.00 ± 1.41 | 14.80 ± 0.40 | | | | | GPT-OSS-120B | ctx_pp @ d32768 (c10) | 2334.23 ± 6.74 | 664.93 ± 581.02 | | | 68725.40 ± 35276.98 | 68649.15 ± 35276.98 | 78601.84 ± 33602.79 | | GPT-OSS-120B | ctx_tg @ d32768 (c10) | 10.54 ± 0.17 | 3.17 ± 2.34 | 84.00 ± 4.32 | 11.23 ± 2.97 | | | | | GPT-OSS-120B | pp2048 @ d32768 (c10) | 1145.89 ± 704.01 | 253.71 ± 303.06 | | | 30938.70 ± 38848.63 | 30862.45 ± 38848.63 | 36802.27 ± 40701.68 | | GPT-OSS-120B | tg128 @ d32768 (c10) | 57.99 ± 34.12 | 6.59 ± 2.84 | 93.33 ± 9.43 | 10.53 ± 1.77 | | | | | GPT-OSS-120B | ctx_pp @ d65535 (c1) | 1533.82 ± 13.70 | 1533.82 ± 13.70 | | | 37795.50 ± 426.04 | 37719.25 ± 426.04 | 37935.42 ± 425.55 | | GPT-OSS-120B | ctx_tg @ d65535 (c1) | 22.50 ± 0.25 | 22.50 ± 0.25 | 24.00 ± 0.00 | 24.00 ± 0.00 | | | | | GPT-OSS-120B | pp2048 @ d65535 (c1) | 927.84 ± 54.94 | 927.84 ± 54.94 | | | 2290.98 ± 126.09 | 2214.73 ± 126.09 | 2419.17 ± 129.74 | | GPT-OSS-120B | tg128 @ d65535 (c1) | 23.37 ± 0.08 | 23.37 ± 0.08 | 24.00 ± 0.00 | 24.00 ± 0.00 | | | | | GPT-OSS-120B | ctx_pp @ d65535 (c2) | 1512.86 ± 4.52 | 1106.50 ± 347.38 | | | 58159.69 ± 18240.56 | 58083.44 ± 18240.56 | 64523.39 ± 12066.59 | | GPT-OSS-120B | ctx_tg @ d65535 (c2) | 7.84 ± 0.08 | 10.23 ± 6.27 | 34.00 ± 0.00 | 18.00 ± 1.00 | | | | | GPT-OSS-120B | pp2048 @ d65535 (c2) | 875.34 ± 12.74 | 689.16 ± 227.94 | | | 3412.71 ± 1103.36 | 3336.46 ± 1103.36 | 4653.66 ± 73.24 | | GPT-OSS-120B | tg128 @ d65535 (c2) | 35.18 ± 0.28 | 17.68 ± 0.24 | 37.33 ± 0.94 | 18.67 ± 0.47 | | | | | GPT-OSS-120B | ctx_pp @ d65535 (c5) | 1509.22 ± 8.83 | 673.73 ± 417.41 | | | 116009.08 ± 53604.98 | 115932.82 ± 53604.98 | 126089.75 ± 50107.77 | | GPT-OSS-120B | ctx_tg @ d65535 (c5) | 4.03 ± 0.01 | 3.27 ± 3.26 | 50.00 ± 0.00 | 12.80 ± 2.76 | | | | | GPT-OSS-120B | pp2048 @ d65535 (c5) | 928.15 ± 15.73 | 352.99 ± 287.19 | | | 8691.52 ± 3547.57 | 8615.27 ± 3547.57 | 10994.34 ± 205.95 | | GPT-OSS-120B | tg128 @ d65535 (c5) | 51.12 ± 0.36 | 10.37 ± 0.14 | 60.00 ± 0.00 | 12.00 ± 0.00 | | | | | GPT-OSS-120B | ctx_pp @ d65535 (c10) | 1501.82 ± 2.95 | 431.77 ± 376.76 | | | 212424.23 ± 110106.34 | 212347.98 ± 110106.34 | 223317.66 ± 108053.45 | | GPT-OSS-120B | ctx_tg @ d65535 (c10) | 3.51 ± 0.01 | 1.48 ± 1.79 | 60.00 ± 0.00 | 9.67 ± 3.57 | | | | | GPT-OSS-120B | pp2048 @ d65535 (c10) | 53.87 ± 2.55 | 45.96 ± 161.38 | | | 199897.64 ± 115957.85 | 199821.39 ± 115957.85 | 210213.11 ± 114454.39 | | GPT-OSS-120B | tg128 @ d65535 (c10) | 3.46 ± 0.19 | 1.45 ± 1.80 | 60.00 ± 0.00 | 9.63 ± 3.69 | | | | | GPT-OSS-120B | ctx_pp @ d100000 (c1) | 1101.34 ± 3.59 | 1101.34 ± 3.59 | | | 80240.81 ± 338.66 | 80164.56 ± 338.66 | 80387.52 ± 340.30 | | GPT-OSS-120B | ctx_tg @ d100000 (c1) | 19.88 ± 0.10 | 19.88 ± 0.10 | 21.00 ± 0.00 | 21.00 ± 0.00 | | | | | GPT-OSS-120B | pp2048 @ d100000 (c1) | 672.97 ± 7.14 | 672.97 ± 7.14 | | | 3119.84 ± 32.52 | 3043.59 ± 32.52 | 3268.76 ± 29.85 | | GPT-OSS-120B | tg128 @ d100000 (c1) | 21.02 ± 0.22 | 21.02 ± 0.22 | 22.00 ± 0.00 | 22.00 ± 0.00 | | | | | GPT-OSS-120B | ctx_pp @ d100000 (c2) | 1094.12 ± 4.95 | 821.96 ± 274.22 | | | 120909.00 ± 40246.23 | 120832.75 ± 40246.23 | 126257.61 ± 35127.68 | | GPT-OSS-120B | ctx_tg @ d100000 (c2) | 3.09 ± 0.04 | 7.45 ± 5.91 | 28.00 ± 0.00 | 15.50 ± 1.50 | | | | | GPT-OSS-120B | pp2048 @ d100000 (c2) | 609.31 ± 24.61 | 478.03 ± 167.39 | | | 4930.24 ± 1643.24 | 4853.99 ± 1643.24 | 6702.46 ± 274.80 | | GPT-OSS-120B | tg128 @ d100000 (c2) | 29.59 ± 0.62 | 14.89 ± 0.38 | 32.00 ± 0.00 | 16.00 ± 0.00 | | | | | GPT-OSS-120B | ctx_pp @ d100000 (c5) | 1096.31 ± 10.59 | 499.70 ± 316.86 | | | 242068.17 ± 114026.69 | 241991.92 ± 114026.69 | 251381.26 ± 111503.90 | | GPT-OSS-120B | ctx_tg @ d100000 (c5) | 1.87 ± 0.03 | 2.21 ± 2.87 | 39.67 ± 0.47 | 11.53 ± 3.22 | | | | | GPT-OSS-120B | pp2048 @ d100000 (c5) | 648.04 ± 3.95 | 265.59 ± 185.93 | | | 10877.47 ± 4798.09 | 10801.22 ± 4798.09 | 15723.88 ± 128.48 | | GPT-OSS-120B | tg128 @ d100000 (c5) | 42.07 ± 0.75 | 8.52 ± 0.16 | 48.33 ± 2.36 | 9.67 ± 0.47 | | | | | GPT-OSS-120B | ctx_pp @ d100000 (c10) | 1095.53 ± 4.07 | 322.46 ± 291.42 | | | 442570.16 ± 231162.15 | 442493.91 ± 231162.15 | 452724.03 ± 229550.26 | | GPT-OSS-120B | ctx_tg @ d100000 (c10) | 1.68 ± 0.01 | 0.99 ± 1.66 | 50.00 ± 0.00 | 8.70 ± 3.93 | | | | | GPT-OSS-120B | pp2048 @ d100000 (c10) | 24.37 ± 0.11 | 7.13 ± 6.30 | | | 458847.36 ± 239752.42 | 458771.11 ± 239752.42 | 468982.14 ± 237665.45 | | GPT-OSS-120B | tg128 @ d100000 (c10) | 1.64 ± 0.01 | 1.30 ± 2.58 | 45.67 ± 0.47 | 9.20 ± 4.95 | | | | Context Depth Analysis: --- depth=0 — Baseline (no context) --- Reading speed: 5,024 +/- 35 tok/s Writing speed: 137.7 +/- 1.3 tok/s (peak: 163) -> Very fast — feels instant, smooth streaming Time to first token: 3606ms -> Noticeable wait (3.6s) --- depth=4,096 — depth 4096 --- Reading speed: 4,122 +/- 89 tok/s Writing speed: 121.5 +/- 2.1 tok/s (peak: 147) -> -11.7% vs baseline (moderate slowdown) -> Very fast — feels instant, smooth streaming Time to first token: 4913ms -> Noticeable wait (4.9s) --- depth=8,192 — depth 8192 --- Reading speed: 3,345 +/- 57 tok/s Writing speed: 111.5 +/- 3.5 tok/s (peak: 131) ⚠ -19.0% vs baseline — BANDWIDTH BOTTLENECK DETECTED -> Very fast — feels instant, smooth streaming Time to first token: 6030ms -> Noticeable wait (6.0s) --- depth=16,384 — Medium Log (~50 pages) --- Reading speed: 2,478 +/- 137 tok/s Writing speed: 101.3 +/- 5.6 tok/s (peak: 126) ⚠ -26.4% vs baseline — BANDWIDTH BOTTLENECK DETECTED -> Very fast — feels instant, smooth streaming Time to first token: 8217ms -> Noticeable wait (8.2s) --- depth=32,768 — Massive Log (~100 pages) --- Reading speed: 1,768 +/- 23 tok/s Writing speed: 91.7 +/- 0.6 tok/s (peak: 110) ⚠ -33.4% vs baseline — BANDWIDTH BOTTLENECK DETECTED -> Very fast — feels instant, smooth streaming Time to first token: 11475ms -> Noticeable wait (11.5s) --- depth=65,535 — Extreme Limit (~200 pages) --- Reading speed: 87 +/- 26 tok/s Writing speed: 5.3 +/- 1.4 tok/s (peak: 68) ⚠ -96.1% vs baseline — BANDWIDTH BOTTLENECK DETECTED -> Slow — may feel sluggish for chat Time to first token: 123778ms -> Noticeable wait (123.8s) --- depth=100,000 — depth 100000 --- Reading speed: 24 tok/s Writing speed: 1.6 tok/s (peak: 47) ⚠ -98.8% vs baseline — BANDWIDTH BOTTLENECK DETECTED -> Slow — may feel sluggish for chat Time to first token: 468727ms -> Noticeable wait (468.7s) Overall impact: depth 0 -> 100,000 = -98.8% generation speed Conclusion: Significant unified memory bandwidth bottleneck at depth 100,000 e[2mJSON data : /home/sparky/Docker/dgx-spark_lite-llm_llama-swap_vllm_llama-cpp_ollama/test-results/benchmarks/GPT-OSS-120B_20260420_183825.jsone[0m e[2mForum table: /home/sparky/Docker/dgx-spark_lite-llm_llama-swap_vllm_llama-cpp_ollama/test-results/benchmarks/GPT-OSS-120B_20260420_183825.mde[0m Recipe YAML: /home/sparky/Docker/dgx-spark_lite-llm_llama-swap_vllm_llama-cpp_ollama/test-results/arena-submission/20260420_183825/GPT-OSS-120B/recipe.yaml ``` ---
**Author:** ![martinB78](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/martinb78/32/437495_2.png) [@martinB78](https://forums.developer.nvidia.com/u/martinB78)\ **Post date:** [April 21, 2026, 7:43am UTC](https://forums.developer.nvidia.com/t/new-tool-llama-benchy-llama-bench-style-benchmarking-for-any-llm-backend-vllm-sglang-llama-cpp-etc/356698/18 "2026-04-21T07:43:20Z")
So now I can see that my setup sucks compared to the results of other users ![image](https://global.discourse-cdn.com/nvidia/original/4X/0/5/7/057686671a00d62bd772f4ff14700031f6c44b1c.png) View full benchmark at [openai/gpt-oss-120b - Spark Arena Benchmark](https://spark-arena.com/benchmark/sub1776755324864) ---
**Author:** ![0rand](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/0rand/32/512004_2.png) [@0rand](https://forums.developer.nvidia.com/u/0rand)\ **Post date:** [October 3, 2026, 8:22am UTC](https://forums.developer.nvidia.com/t/new-tool-llama-benchy-llama-bench-style-benchmarking-for-any-llm-backend-vllm-sglang-llama-cpp-etc/356698/19 "2026-10-03T08:22:24Z")
Made a fix to Llama-benchy @eugr to support non-vllm engines to show prefill correctly. I was having this issues always when testing other engines (on Mac for example) - benchy is tuned to support a peculiar behaviour of vllm output that is not completely openai-standard. Other fully-aligned openai-endpoints generate bogus prefill numbers in benchy (and teb as a result). This fix solves it as demonstated for TensorFold: ```auto llama-benchy (0.4.1.dev2+gef4927e79) Date: 2026-10-03 10:09:07 Benchmarking model: glm-5.3-flash at http://192.168.1.8:8100/v1 Concurrency levels: [1] Total tokens available in text corpus: 141280 Skipping coherence test (--skip-coherence specified) Measuring latency using mode: api... Average latency (api): 28.95 ms Running test: pp=2048, tg=128, depth=0, concurrency=1 Run 1/3 (batch size 1)... No complete token_ids in response, using stream usage token count Run 2/3 (batch size 1)... Run 3/3 (batch size 1)... Running test: pp=8192, tg=128, depth=0, concurrency=1 Run 1/3 (batch size 1)... Run 2/3 (batch size 1)... Run 3/3 (batch size 1)... Running test: pp=32768, tg=128, depth=0, concurrency=1 Run 1/3 (batch size 1)... Run 2/3 (batch size 1)... Run 3/3 (batch size 1)... Printing results in MD format: | model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:--------------|--------:|---------------:|-------------:|-----------------:|-----------------:|-----------------:| | glm-5.3-flash | pp2048 | 1655.86 ± 9.55 | | 1273.06 ± 7.15 | 1244.11 ± 7.15 | 1273.06 ± 7.15 | | glm-5.3-flash | tg128 | 65.75 ± 3.63 | 66.00 ± 3.74 | | | | | glm-5.3-flash | pp8192 | 1740.88 ± 2.99 | | 4741.52 ± 8.09 | 4712.56 ± 8.09 | 4741.52 ± 8.09 | | glm-5.3-flash | tg128 | 58.61 ± 4.14 | 59.00 ± 4.24 | | | | | glm-5.3-flash | pp32768 | 1744.48 ± 2.72 | | 18819.68 ± 29.34 | 18790.73 ± 29.34 | 18819.68 ± 29.34 | | glm-5.3-flash | tg128 | 62.39 ± 3.28 | 63.00 ± 3.27 | | | | llama-benchy (0.4.1.dev2+gef4927e79) date: 2026-10-03 10:09:07 | latency mode: api ``` > > > \## Problem > > \`first\_response\_ts\` is set on the first chunk that contains \`choices…\` — but OpenAI's own streaming example emits a first chunk whose delta carries only the assistant role: > > \`\`\` > data: {"choices":\[{"delta":{"role":"assistant"}, ...}\]} > \`\`\` > > Servers that follow the OpenAI example faithfully (TensorFold, oMLX, mlx-serve, and any server built from the OpenAI cookbook pattern) write that priming chunk to the socket the instant the stream opens — \*\*before prefill has run\*\*. \`ttfr\` and \`est\_ppt\` then collapse to the socket round-trip (~2–6 ms), and \`pp\_throughput = prompt\_tokens / est\_ppt\` inflates by orders of magnitude. > > Measured live on a 2x DGX Spark TensorFold GLM-5.3-Flash stack (2048-token prompt, \`--no-cache\`, \`cached\_tokens: 0\` verified — the prefill is real): > > \`\`\` > chunk#1 @ 6.6ms FIRST choices chunk, delta={"role": "assistant"} \<- role priming > chunk#2 @ 2717.6ms FIRST content: reasoning\_content "The" > \`\`\` > > Reported: \`pp t/s = 847,691\` (from est\_ppt 2.4 ms). Honest: ~750 t/s (2048 / 2.72 s). vLLM escapes the trap only because it deviates from the OpenAI example and writes nothing until the first token exists — so benchy's \`est\_ppt\` was effectively calibrated on that deviation. > > \## Change > > The first response is now the first chunk carrying generated payload — \`content\`, \`reasoning\_content\`, \`reasoning\` or \`tool\_calls\` — the same event class \`first\_token\_ts\` already waits for, so \`est\_ppt\` and \`ttft\` measure consistently on every engine. vLLM-style streams are unaffected (their first choices chunk already carries content). > > \## Tests > > Four regression tests via the existing fake-SSE harness: > \- role-only first chunk does not set \`first\_response\_ts\` (content chunk does); > \- reasoning delta counts as first response; > \- tool-call delta counts as first response; > \- vLLM-style stream lands both timestamps on the same chunk, unchanged. > > RED/GREEN verified: the three new-behaviour tests fail against the pre-fix \`client.py\` and pass with the fix; full suite 29/29. Live confirmation on the engine above: \`pp2048\` now reports 1,488 t/s (est\_ppt 1,229 ms) instead of 847,691. > > No schema change: \`ttfr\`/\`est\_ppt\` keep their meaning — they now measure it honestly on OpenAI-faithful servers. A/B test - without the fix ```auto llama-benchy --base-url http://192.168.1.8:8100/v1 --model glm-5.3-flash --pp 2048 8192 32768 --tg 128 --depth 0 --concurrency 1 --runs 3 --no-cache --no-warmup --skip-coherence --no-adapt-prompt --latency-mode api --tokenizer /home/xraan/.cache/huggingface/hub/models--Mia-AiLab--GLM-5.3-Flash-EXL3-TR3-4bpw/snapshots/9eaebb7c4e96d983dcd538e18624622ba5b820a8/tokenizer.json llama-benchy (0.4.0) Date: 2026-10-03 10:14:45 Benchmarking model: glm-5.3-flash at http://192.168.1.8:8100/v1 Concurrency levels: [1] Total tokens available in text corpus: 141280 Skipping coherence test (--skip-coherence specified) Measuring latency using mode: api... Average latency (api): 29.06 ms Running test: pp=2048, tg=128, depth=0, concurrency=1 Run 1/3 (batch size 1)... No complete token_ids in response, using stream usage token count Run 2/3 (batch size 1)... Run 3/3 (batch size 1)... Running test: pp=8192, tg=128, depth=0, concurrency=1 Run 1/3 (batch size 1)... Run 2/3 (batch size 1)... Run 3/3 (batch size 1)... Running test: pp=32768, tg=128, depth=0, concurrency=1 Run 1/3 (batch size 1)... Run 2/3 (batch size 1)... Run 3/3 (batch size 1)... Printing results in MD format: | model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:--------------|--------:|-----------------------:|-------------:|--------------:|---------------:|-----------------:| | glm-5.3-flash | pp2048 | 134858.06 ± 0.00 | | 17.61 ± 18.90 | 5.09 ± 7.20 | 1275.23 ± 2.89 | | glm-5.3-flash | tg128 | 62.85 ± 7.64 | 63.33 ± 7.41 | | | | | glm-5.3-flash | tg128 | 58.73 ± 3.95 | 59.33 ± 3.68 | | | | | glm-5.3-flash | pp32768 | 2865382.63 ± 109031.54 | | 40.52 ± 0.45 | 11.46 ± 0.45 | 18850.48 ± 25.09 | | glm-5.3-flash | tg128 | 59.79 ± 0.96 | 60.00 ± 0.82 | | | | llama-benchy (0.4.0) date: 2026-10-03 10:14:45 | latency mode: api ``` Similar results for mlx-serve (metal) With fix: ```auto | model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:---------------------------------------------|--------:|----------------:|-------------:|-------------------:|-------------------:|-------------------:| | xrrand/Qwen3.8-Flash-Next-MLX-Serve-4bit-mtp | pp2048 | 701.28 ± 422.85 | | 8010.38 ± 8398.75 | 7999.36 ± 8398.75 | 8010.38 ± 8398.75 | | xrrand/Qwen3.8-Flash-Next-MLX-Serve-4bit-mtp | tg128 | 75.94 ± 5.24 | 76.33 ± 5.19 | | | | | xrrand/Qwen3.8-Flash-Next-MLX-Serve-4bit-mtp | pp8192 | 995.66 ± 72.57 | | 8293.95 ± 583.39 | 8282.94 ± 583.39 | 8293.95 ± 583.39 | | xrrand/Qwen3.8-Flash-Next-MLX-Serve-4bit-mtp | tg128 | 72.33 ± 5.47 | 72.67 ± 5.25 | | | | | xrrand/Qwen3.8-Flash-Next-MLX-Serve-4bit-mtp | pp32768 | 1142.36 ± 80.25 | | 28841.55 ± 1930.30 | 28830.53 ± 1930.30 | 28841.55 ± 1930.30 | | xrrand/Qwen3.8-Flash-Next-MLX-Serve-4bit-mtp | tg128 | 74.28 ± 8.28 | 74.67 ± 8.18 | | | | llama-benchy (0.4.1.dev2+gef4927e79) date: 2026-10-03 10:20:51 | latency mode: api ``` Without fix ```auto | model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:---------------------------------------------|--------:|-----------------------:|-------------:|-------------:|---------------:|-------------------:| | xrrand/Qwen3.8-Flash-Next-MLX-Serve-4bit-mtp | tg128 | 85.98 ± 4.82 | 86.33 ± 4.92 | | | | | xrrand/Qwen3.8-Flash-Next-MLX-Serve-4bit-mtp | pp8192 | 7495145.98 ± 0.00 | | 9.37 ± 1.19 | 0.36 ± 0.52 | 5618.39 ± 1842.39 | | xrrand/Qwen3.8-Flash-Next-MLX-Serve-4bit-mtp | tg128 | 75.88 ± 0.84 | 76.67 ± 0.94 | | | | | xrrand/Qwen3.8-Flash-Next-MLX-Serve-4bit-mtp | pp32768 | 1876638.02 ± 336099.53 | | 27.91 ± 3.62 | 18.11 ± 3.62 | 18934.88 ± 1199.48 | | xrrand/Qwen3.8-Flash-Next-MLX-Serve-4bit-mtp | tg128 | 68.38 ± 3.62 | 69.00 ± 3.27 | | | | llama-benchy (0.4.0) date: 2026-10-03 10:23:40 | latency mode: api ```