Some initial llama-benchy results using recipe from PR Add DeepSeek V4 Flash recipe by arthur-drozdov · Pull Request #219 · eugr/spark-vllm-docker · GitHub, concurrent runs throttled pretty hard with context over 65535 so wasn’t able to test up to 100k context (however in practice it seems to only be able to handle a single concurrent request for large context).
Decode throughput, tg128, total tok/s:
| depth | c1 | c2 | c5 | c10 |
|---|---|---|---|---|
| 0 | 37.32 | 54.16 | 43.64 | 43.09 |
| 4096 | 27.19 | 40.00 | 25.90 | 27.23 |
| 8192 | 35.21 | 30.82 | 26.72 | 20.52 |
| 16384 | 37.54 | 27.55 | 16.63 | 13.52 |
| 32768 | 33.51 | 19.01 | 12.64 | 2.25 |
| 65535 | 29.50 | 1.11 | 0.74 | - |
Prompt throughput, pp2048, total tok/s:
| depth | c1 | c2 | c5 | c10 |
|---|---|---|---|---|
| 0 | 1097.6 | 977.9 | 1044.3 | 975.4 |
| 4096 | 738.9 | 598.3 | 571.3 | 552.6 |
| 8192 | 613.4 | 548.1 | 460.7 | 411.1 |
| 16384 | 471.1 | 467.9 | 307.7 | 280.2 |
| 32768 | 318.4 | 322.6 | 222.6 | 39.4 |
| 65535 | 176.0 | 17.2 | 11.9 | - |