Hi all,
Recipe for Eugr’s repo (will work with Sparkrun also) to serve the new Laguna model. It’s still running on my own benchmarking tool but seems very good so far.
I tried fiddling with settings but their defaults were optimal.
First:
git pull
./build-and-copy.sh --tf5
Then run (takes ~15 mins on first run, use tensor_parallel: 1 for single spark):
recipe_version: "1"
name: Laguna-S-2.1-NVFP4
description: vLLM serving Laguna S 2.1 in native NVFP4 with DFlash speculation
model: poolside/Laguna-S-2.1-NVFP4
container: vllm-node
defaults:
port: 8000
host: 0.0.0.0
tensor_parallel: 2
gpu_memory_utilization: 0.85
max_model_len: 262144
env:
CUTE_DSL_ARCH: "sm_121a"
MAX_JOBS: "4"
# Replace gpu_memory_utilization config with `--kv-cache-memory=32449423258 to fit into memory
command: |
vllm serve poolside/Laguna-S-2.1-NVFP4 \
--port {port} \
--host {host} \
--tensor-parallel-size {tensor_parallel} \
--gpu-memory-utilization {gpu_memory_utilization} \
--max-model-len {max_model_len} \
--speculative-config '{{"model":"poolside/Laguna-S-2.1-DFlash-NVFP4","num_speculative_tokens":15}}' \
--enable-auto-tool-choice \
--tool-call-parser poolside_v1 \
--reasoning-parser poolside_v1 \
--override-generation-config '{{"temperature":0.7,"top_p":0.95}}' \
--max-num-seqs 32
Benchmarks:
============ Serving Benchmark Result ============
Successful requests: 50
Failed requests: 0
Benchmark duration (s): 83.41
Total input tokens: 12173
Total generated tokens: 10229
Request throughput (req/s): 0.60
Output token throughput (tok/s): 122.63
Peak output token throughput (tok/s): 96.00
Peak concurrent requests: 50.00
Total token throughput (tok/s): 268.58
---------------Time to First Token----------------
Mean TTFT (ms): 3287.16
Median TTFT (ms): 1033.71
P99 TTFT (ms): 14485.45
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 166.99
Median TPOT (ms): 167.42
P99 TPOT (ms): 351.18
---------------Inter-token Latency----------------
Mean ITL (ms): 368.15
Median ITL (ms): 393.69
P99 ITL (ms): 440.30
---------------Speculative Decoding---------------
Acceptance rate (%): 11.71
Acceptance length: 2.76
Drafts: 3717
Draft tokens: 55755
Accepted tokens: 6529
Per-position acceptance (%):
Position 0: 64.89
Position 1: 37.77
Position 2: 22.33
Position 3: 13.80
Position 4: 9.25
Position 5: 6.43
Position 6: 4.90
Position 7: 3.98
Position 8: 2.99
Position 9: 2.34
Position 10: 2.02
Position 11: 1.67
Position 12: 1.35
Position 13: 1.16
Position 14: 0.78
==================================================
I’ll get round to doing the two spark fp8 one at some point but it’s probably default also.
