I’m nearly certain that unless you’re doing document extraction or certain other tasks without thinking enabled, that you need temp greater than zero generally for reasoning models.
The other deal breaker for me is --enable-prefix-caching for multi-turn long context, this settings really matters for end-to-end latency.
For the majority of the tasks, yes.
And about prefix caching, @whpthomas , that will be my next battle. I have a recipe with it working properly but it tanks on longer context (64k+), but I’m in no rush to solve it (yet). I will be trying this DFlash for Qwen3.5-122B-A10B = 80+ tok/s on 1x Spark! soon as well as it allegedly works with Prefix Caching w/o issues.
Would be nice to quote the benchmark versions, and details of the settings. I run various benchmarks over a period of time, and realized they are inconsistent between different versions. Even hardmode.
I couldn’t get Ornith and AgentWorld to bench better than Qwen 3.6 on Tool Calls Bench hardmode , all 35B.
I totally agree – you should qualify everything I post by saying “all this shows is what happened with this specific model, during a specific run” – these are stochastic systems and it often feels like sometimes the model gets out of the wrong side of the bed, fail dismally and the next day, no updates, the model fires up and gets a top score. So yes – all I try to take away from this analysis is what settings are less likely to degrade quality. Not what setting guarantee the highest quality or that one model is better than another.
I use all these tests to guide the configurations I run in production. Even then it can be misleading. -pc-mtp-bf16 was ranking high in benchmarks, but fell over in production. -pc ended up being the fastest and most reliable in production.
What I mean by ‘reliable in production’, is staying on task and finishing a concurrent multi-task multi-modal job.
Confirmed that my posted recipe here + FP8 KV Cache Dtype + Prefix Caching works as expected. It’s a tiny bit slower at higher context pressure, but within the expected values.
I’m still testing for degradation on the KV Cache (I doubt is meaningful) but definitely speeds up the multi-step agent workloads by more than 2x :)
Give it a go and let me know!
May I ask what tool this is you are using to test?
It’s a personal benchmark for mathematical and coding capabilities. Nothing I can yet share since i’m still validating its accuracy but I’m already able to see patters clearly which is nice.
Have experienced same issue with this model as well with the stop.
Interesting, I was coming to the same conclusion. mtp aren’t always lossless. And bf16 aren’t always the best, especially if the KV was calibrated differently. And GenAI are not always deterministics (although that should be obvious to all).
I also experienced that MTP being lossless is a myth. The main model accepts or rejects MTP candidates, and it accepts subpar candidates more often than I like. Seems worse for high number of MTPs, but vastly different across different models.
buggy always crashes after 2-3h usage start inventing non existing tools and fails on tool calls start outputting to text or even not parsed.
That i have always found with all ornith finetunes. They got astroturfed hard … actually worse than base model.
what tool is this btw?
Tool Eval Bench
But I am running some custom scripts, you can get run-test.sh here:
https://github.com/whpthomas/spark-auto-round/tree/master/spark-vllm-docker
I then have the main test script that looks like this:
#!/bin/bash
set -euo pipefail
trap 'echo "Error on line $LINENO"' ERR
TEST_PATH="$HOME/test-35b"
cd ~/spark-vllm-docker
#./run-test.sh --recipe qwen3.6-35b --path "$TEST_PATH"
./run-test.sh --recipe qwen3.6-35b-fp8 --path "$TEST_PATH"
./run-local-test.sh --recipe local-qwen3.6-35b-sar --path "$TEST_PATH"
./run-local-test.sh --recipe local-ornith-1.0-35b-sar --path "$TEST_PATH"
./generate-recipe.sh --test 1 recipes/qwen3.6-35b-fp8.yaml
./run-local-test.sh --recipe qwen3.6-35b-fp8-current --path "$TEST_PATH"
./generate-recipe.sh --test 1 recipes/local-qwen3.6-35b-sar.yaml
./run-local-test.sh --recipe local-qwen3.6-35b-sar-current --path "$TEST_PATH"
./generate-recipe.sh --test 1 recipes/local-ornith-1.0-35b-sar.yaml
./run-local-test.sh --recipe local-ornith-1.0-35b-sar --path "$TEST_PATH"
./generate-recipe.sh --test 2 recipes/qwen3.6-35b-fp8.yaml
./run-local-test.sh --recipe qwen3.6-35b-fp8-current --path "$TEST_PATH"
./generate-recipe.sh --test 2 recipes/local-qwen3.6-35b-sar.yaml
./run-local-test.sh --recipe local-qwen3.6-35b-sar-current --path "$TEST_PATH"
./generate-recipe.sh --test 2 recipes/local-ornith-1.0-35b-sar.yaml
./run-local-test.sh --recipe local-ornith-1.0-35b-sar --path "$TEST_PATH"
./generate-recipe.sh --test 3 recipes/qwen3.6-35b-fp8.yaml
./run-local-test.sh --recipe qwen3.6-35b-fp8-current --path "$TEST_PATH"
./generate-recipe.sh --test 3 recipes/local-qwen3.6-35b-sar.yaml
./run-local-test.sh --recipe local-qwen3.6-35b-sar-current --path "$TEST_PATH"
./generate-recipe.sh --test 3 recipes/local-ornith-1.0-35b-sar.yaml
./run-local-test.sh --recipe local-ornith-1.0-35b-sar --path "$TEST_PATH"
./generate-recipe.sh --test 4 recipes/qwen3.6-35b-fp8.yaml
./run-local-test.sh --recipe qwen3.6-35b-fp8-current --path "$TEST_PATH"
./generate-recipe.sh --test 4 recipes/local-qwen3.6-35b-sar.yaml
./run-local-test.sh --recipe local-qwen3.6-35b-sar-current --path "$TEST_PATH"
./generate-recipe.sh --test 4 recipes/local-ornith-1.0-35b-sar.yaml
./run-local-test.sh --recipe local-ornith-1.0-35b-sar --path "$TEST_PATH"
./generate-recipe.sh --test 5 recipes/qwen3.6-35b-fp8.yaml
./run-local-test.sh --recipe qwen3.6-35b-fp8-current --path "$TEST_PATH"
./generate-recipe.sh --test 5 recipes/local-qwen3.6-35b-sar.yaml
./run-local-test.sh --recipe local-qwen3.6-35b-sar-current --path "$TEST_PATH"
./generate-recipe.sh --test 5 recipes/local-ornith-1.0-35b-sar.yaml
./run-local-test.sh --recipe local-ornith-1.0-35b-sar --path "$TEST_PATH"
./generate-recipe.sh --test 6 recipes/qwen3.6-35b-fp8.yaml
./run-local-test.sh --recipe qwen3.6-35b-fp8-current --path "$TEST_PATH"
./generate-recipe.sh --test 6 recipes/local-qwen3.6-35b-sar.yaml
./run-local-test.sh --recipe local-qwen3.6-35b-sar-current --path "$TEST_PATH"
./generate-recipe.sh --test 6 recipes/local-ornith-1.0-35b-sar.yaml
./run-local-test.sh --recipe local-ornith-1.0-35b-sar --path "$TEST_PATH"
./generate-recipe.sh --test 7 recipes/qwen3.6-35b-fp8.yaml
./run-local-test.sh --recipe qwen3.6-35b-fp8-current --path "$TEST_PATH"
./generate-recipe.sh --test 7 recipes/local-qwen3.6-35b-sar.yaml
./run-local-test.sh --recipe local-qwen3.6-35b-sar-current --path "$TEST_PATH"
./generate-recipe.sh --test 7 recipes/local-ornith-1.0-35b-sar.yaml
./run-local-test.sh --recipe local-ornith-1.0-35b-sar --path "$TEST_PATH"
cd ~/test-35b
python3 ~/scripts/tool-eval-table.py ~/test-35b
And this one uses sed to alter the recipe generate-recipe.sh
#!/bin/bash
set -euo pipefail
# generate-recipe.sh - Generate test variants of recipe YAML files
#
# This script takes a base recipe and generates a test variant with specific
# configuration changes (prefix caching, speculative decoding, bf16, etc.)
#
# Usage:
# ./generate-recipe.sh --test <num> <recipe.yaml>
# ./generate-recipe.sh --list
#
# Examples:
# ./generate-recipe.sh --test 1 recipes/local-qwen3.6-35b-sar.yaml
# ./generate-recipe.sh --test 3 recipes/local-qwen3.6-35b-sar.yaml
# ./generate-recipe.sh --list
#==============================================================================
# TEST CONFIGURATIONS
#==============================================================================
# Format: "suffix|prefix_caching|speculative|kv_dtype|dtype_bf16|num_spec_tokens"
#
# To add a new test variation:
# 1. Add entry to TEST_CONFIGS array below
# 2. If new flag type needed, add handling in transform_recipe()
#==============================================================================
declare -A TEST_CONFIGS=(
[0]="baseline|off|none|default|no|0" # Baseline (no changes)
[1]="pc|on|none|default|no|0" # Prefix caching only
[2]="bf16|off|none|default|yes|0" # BF16 dtype
[3]="bf16-kv8|off|none|fp8_e4m3|yes|0" # BF16 dtype + fp8 kv cache
[4]="mtp|off|mtp|default|no|3" # MTP speculative decoding
[5]="mtp-bf16|off|mtp|default|yes|3" # MTP + BF16
[6]="mtp-pc|on|mtp|default|no|1" # MTP + Prefix caching
[7]="mtp-pc-bf16|on|mtp|default|yes|1" # MTP + Prefix caching + BF16
[8]="dflash|off|dflash|fp8_e4m3|no|5" # DFlash speculative decoding
[9]="dflash-pc|on|dflash|fp8_e4m3|no|5" # DFlash + Prefix caching
)
# MTP speculative config JSON
SPECULATIVE_MTP='{{"method": "mtp", "num_speculative_tokens": {num_speculative_tokens}}}'
# DFlash speculative config JSON
SPECULATIVE_DFLASH='{{"method": "dflash", "model":"z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": {num_speculative_tokens}}}'
#==============================================================================
# FUNCTIONS
#==============================================================================
usage() {
cat << 'EOF'
Usage: generate-recipe.sh [options] <recipe.yaml>
Generate test variants of recipe YAML files.
Options:
--test <num> Generate test variant (0-8)
--list List available test configurations
-h, --help Show this help message
Test Configurations:
0 baseline No changes (baseline)
1 pc Prefix caching enabled
2 bf16 BF16 dtype
3 bf16-kv8 BF16 dtype + fp8 kv cache
4 mtp MTP speculative decoding
5 mtp-bf16 MTP + BF16
6 mtp-pc MTP + Prefix caching
7 mtp-pc-bf16 MTP + Prefix caching + BF16
8 dflash DFlash speculative decoding
9 pc-dflash DFlash + Prefix caching
Examples:
./generate-recipe.sh --test 1 recipes/local-qwen3.6-35b-sar.yaml
./generate-recipe.sh --test 3 recipes/local-qwen3.6-35b-sar.yaml
./generate-recipe.sh --list
Output:
Creates <recipe-basename>-current.yaml in same directory as input.
Overwrites any existing -current.yaml file.
EOF
}
list_tests() {
echo "Available test configurations:"
echo ""
printf "%-6s %-16s %-16s %-14s %-14s %-8s %-8s\n" "Test#" "Suffix" "Prefix Cache" "Speculative" "KV Dtype" "BF16" "#Spec"
echo "------ ---------------- ---------------- -------------- -------------- -------- ------"
for num in $(echo "${!TEST_CONFIGS[@]}" | tr ' ' '\n' | sort -n); do
IFS='|' read -r suffix pc spec kv bf16 num_spec <<< "${TEST_CONFIGS[$num]}"
printf "%-6s %-16s %-16s %-14s %-14s %-8s %-8s\n" "$num" "$suffix" "$pc" "$spec" "$kv" "$bf16" "$num_spec"
done
}
get_test_config() {
local test_num=$1
if [[ -z "${TEST_CONFIGS[$test_num]+x}" ]]; then
echo "Error: Invalid test number: $test_num" >&2
echo "Valid test numbers: $(echo "${!TEST_CONFIGS[@]}" | tr ' ' ', ' | sort -n)" >&2
exit 1
fi
echo "${TEST_CONFIGS[$test_num]}"
}
# Extract the base model name from served_model_name
# e.g., "qwen3.6-35b-sar" -> "qwen3.6-35b-sar"
# e.g., "qwen3.6-35b-sar-pc" -> "qwen3.6-35b-sar"
extract_base_model_name() {
local served_name=$1
# Remove common suffixes to get base name
echo "$served_name" | sed -E 's/-(pc|mtp|dflash|bf16|baseline)(-.*)*$//'
}
# Apply suffix to served_model_name
apply_suffix() {
local suffix=$1
local served_name=$2
if [[ "$suffix" == "baseline" ]]; then
echo "$served_name"
else
echo "${served_name}-${suffix}"
fi
}
# Main transformation function
transform_recipe() {
local input_file=$1
local test_num=$2
# Get test configuration
IFS='|' read -r suffix pc_setting spec_type kv_dtype bf16 num_spec <<< "$(get_test_config "$test_num")"
# Read the input file into a temporary file for processing
local tmp_file
tmp_file=$(mktemp)
cp "$input_file" "$tmp_file"
# Extract current served_model_name
local current_served_name
current_served_name=$(grep -E "^\s*served_model_name:" "$tmp_file" | head -1 | sed 's/.*served_model_name:\s*//')
if [[ -z "$current_served_name" ]]; then
echo "Error: Could not find served_model_name in $input_file" >&2
rm -f "$tmp_file"
exit 1
fi
# Get base model name (remove any existing suffix)
local base_name
base_name=$(extract_base_model_name "$current_served_name")
# Calculate new served_model_name
local new_served_name
new_served_name=$(apply_suffix "$suffix" "$base_name")
# 1. Update served_model_name
sed -i "s|served_model_name:.*|served_model_name: ${new_served_name}|" "$tmp_file"
# 2. Handle prefix caching
if [[ "$pc_setting" == "on" ]]; then
sed -i 's/--no-enable-prefix-caching/--enable-prefix-caching/g' "$tmp_file"
else
sed -i 's/--enable-prefix-caching/--no-enable-prefix-caching/g' "$tmp_file"
fi
# 3. Remove any existing speculative-config line (we'll add it at end)
sed -i '/--speculative-config/d' "$tmp_file"
# 4. Remove any existing --kv-cache-dtype line
sed -i '/--kv-cache-dtype/d' "$tmp_file"
# 5. Remove any existing --dtype line
sed -i '/--dtype bfloat16/d' "$tmp_file"
# 6. Update num_speculative_tokens if present in defaults
if grep -q 'num_speculative_tokens:' "$tmp_file"; then
sed -i "s|num_speculative_tokens:.*|num_speculative_tokens: ${num_spec}|" "$tmp_file"
fi
# 7. Add speculative config defaults if needed (skip if already present)
if [[ "$spec_type" == "mtp" ]]; then
sed -i "/--host {host}/a\\ --speculative-config '${SPECULATIVE_MTP}' \\\\" "$tmp_file"
elif [[ "$spec_type" == "dflash" ]]; then
sed -i "/--host {host}/a\\ --speculative-config '${SPECULATIVE_DFLASH}' \\\\" "$tmp_file"
fi
# 8. Add dtype
if [[ "$bf16" == "yes" ]]; then
sed -i "/--host {host}/a\\ --dtype bfloat16 \\\\" "$tmp_file"
fi
# 9. Add kv-cache-dtype
if [[ "$kv_dtype" != "default" ]]; then
sed -i "/--host {host}/a\\ --kv-cache-dtype ${kv_dtype} \\\\" "$tmp_file"
fi
# 10. Add comment at top showing test variant
sed -i "1s|^# Recipe:.*|# Recipe: ${suffix} test variant\\n# Test config: pc=${pc_setting} spec=${spec_type} kv=${kv_dtype} bf16=${bf16}\\n#|" "$tmp_file"
# Output the transformed content
cat "$tmp_file"
rm -f "$tmp_file"
}
#==============================================================================
# MAIN
#==============================================================================
main() {
local test_num=""
local recipe_file=""
local list_mode=false
# Parse arguments
while [[ $# -gt 0 ]]; do
case $1 in
--test)
test_num="$2"
shift 2
;;
--list)
list_mode=true
shift
;;
-h|--help)
usage
exit 0
;;
-*)
echo "Error: Unknown option: $1" >&2
usage
exit 1
;;
*)
recipe_file="$1"
shift
;;
esac
done
# Handle --list mode
if [[ "$list_mode" == "true" ]]; then
list_tests
exit 0
fi
# Validate arguments
if [[ -z "$test_num" ]]; then
echo "Error: --test <num> is required" >&2
usage
exit 1
fi
if [[ -z "$recipe_file" ]]; then
echo "Error: Recipe file is required" >&2
usage
exit 1
fi
# Validate test number
if [[ -z "${TEST_CONFIGS[$test_num]+x}" ]]; then
echo "Error: Invalid test number: $test_num" >&2
list_tests
exit 1
fi
# Check input file exists
if [[ ! -f "$recipe_file" ]]; then
echo "Error: Recipe file not found: $recipe_file" >&2
exit 1
fi
# Get output filename
local basename
basename=$(basename "$recipe_file" .yaml)
local dirname
dirname=$(dirname "$recipe_file")
local output_file="${dirname}/${basename}-current.yaml"
# Get test config for display
IFS='|' read -r suffix pc_setting spec_type kv_dtype bf16 <<< "$(get_test_config "$test_num")"
echo "Generating test variant..."
echo " Input: $recipe_file"
echo " Output: $output_file"
echo " Test: #$test_num ($suffix)"
echo " Config: prefix_cache=$pc_setting speculative=$spec_type kv_dtype=$kv_dtype bf16=$bf16"
echo ""
# Generate the transformed recipe
transform_recipe "$recipe_file" "$test_num" > "$output_file"
echo "✓ Generated $output_file"
}
main "$@"
Thanks for Ornith, great job as always. I have just one question: is this fix propagated in the current Mistral Medium PrismQuant config ? rdtand/Mistral-Medium-3.5-128B-PrismaQuant-4.75-vllm · Fix Transformers config Thanks :)
@tenari first off, thanks for the great work!
I am trying to apply prismaquant (aura) on deepseek 4 flash through your prismaquant repo. In stage 3 it failed after a few shards. Then on rerunning it continued but again OOM after 14 shards, and running again still leads to the same OOM error.
How do you tune the memory related parameters when you set up a bigger model?
Any help appreciated!
The memory levers live in three places, matching the three big residency consumers — the layer cache, the activation cache, and pack/export transients. The ones with real headroom for larger models:
Layer cache (the dominant one, probe + cost stages) — the streaming machinery in prismaquant/layer_streaming.py (LRU + pressure-shrink) sized by the budget computed in incremental_measure_quant_cost.py. Tonight's chain logged it directly: layer cache budget = 70.9 GB (free=115.9, headroom=45.0, dynamic_reserve=45.0, explicit) with prefetch auto: workers=4, cache_slots=10, memory_slots=14, est_layer=6.9 GB. The levers:
- CACHE_HEADROOM_GB (45 in the Hy3 drivers) — raising it directly shrinks the layer-cache budget; the comment on it records the failure mode it guards (shard-1 OOM when an 84.9 GB cache + backward workspace overran the pool).
- LAYERS_PER_SHARD (8 for Hy3) — fewer layers per shard shrinks per-sweep residency at the cost of more full-model sweeps; the driver comment quantifies the tradeoff (auto picked 1/shard ≈ 16 h, 8/shard ≈ 2.5 h — the pressure is the layer cache, not tracked-Linear state).
- PRISMAQUANT_PROBE_PREFETCH_LOOKAHEAD (2 for Hy3; the GGUF-lane Hy3 probe OOM was fixed by dropping 10→3) and PRISMAQUANT_PROBE_MIN_AVAILABLE_GB (40) — the prefetch depth and the free-memory floor the probe paces against. source_prefetch.py is the fail-fast residency gate on that path.
Activation cache — ACTIVATION_ROWS_LIMIT (1024 for Hy3; 11 GB on disk/resident), plus NSAMPLES/SEQLEN upstream of it. This shrinks memory but is quality-coupled: the 4B GGUF lesson was that starved activation rows (r256, 2.6% Hessian rank) degrade GPTQ toward RTN — lower it only with the rank consequence in mind.
Expert stacks and pack transients — PRISMAQUANT_EXPERT_COST_SAMPLE (16 of 192 for Hy3) bounds expert cost-measurement residency; _SLICE_MAX_ELEMS in nvfp4_cb_formats.py (64M elements) chunks the bit-pack that caused the 155 GB OOM; the cost path also slices stacks >256M elements along dim 0. On export, the streaming writer's peak is ~one source tensor + codebooks by design, self-capped at set_per_process_memory_fraction(0.75) in its main(), with _MAX_OPEN_SHARDS=4 bounding source mmaps and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True fighting fragmentation.
Two more that exist but matter less for the CB lane: PRODUCTION_CACHE_LRU_GB (64 default) sizes the ProductionWeightCache, but CB runs with PRODUCTION_CACHE=0 so it's idle here; and autoscale.py sits in the streaming subsystem — I know it's part of the huge-model path but I'd have to read it to tell you its specific knobs honestly.
For a 600B-class target on this box, the plays in order: raise CACHE_HEADROOM_GB / drop LAYERS_PER_SHARD (pure time-for-memory, no quality cost), keep prefetch lookahead low, sample experts harder — and only then touch ACTIVATION_ROWS_LIMIT, because that one trades quality, not just wall-clock.