tool-eval-bench 2.7.0: measuring speculative decoding properly, and a lot of grader fixes
Most of what changed this cycle came from running the benchmark against more servers and finding the places where it was measuring the serving stack, or its own regexes, instead of the model.
Speculative decoding
If you run vLLM with MTP, EAGLE, or a draft model, start the server with --per-request-spec-decode-metrics summary (or detailed). --spec-bench now reads the per-request metrics.speculative_decoding field from each response instead of diffing the server-wide Prometheus counters around the request. The old method picked up whatever else was hitting the box at the time; the new one is scoped to the request. Servers without the flag keep working the way they did.
detailed mode also gives you a per-position acceptance curve. On a GLM-5.3-Flash server with seven-token drafts, acceptance went from 93% at position 0 to 27% at position 6, with 73% of the draft window used. That is the number you need to decide whether num_speculative_tokens is set right, and the counters alone never gave it to you.
Other spec-bench changes:
- –spec-runs (default 3) repeats each cell and pools the counters. One request is only a few dozen speculative steps, and three runs of the same prompt spread from 36% to 47% on my server, so a single run was never a reliable α. Rows show the pooled value and the range.
- –spec-prompt-file runs your own prompts, plain lines or JSON lines. Acceptance is the most workload-dependent number in inference, and the three built-in prompts are not your workload.
- –temperature applies to spec-bench. Greedy is still the default and it is a ceiling: at 0.6, acceptance on the same prompts dropped roughly nine points.
- Every row reports verify steps per second, a lower bound on the no-spec decode rate, so you get a speedup ceiling without a separate baseline run.
spec-live plots a 30-second rolling window instead of the session average, which used to flatten out after a couple of minutes and hide a workload change. A dead server now turns the header red instead of leaving the last numbers under a green spinner.
Thinking models
Turns are sent with max_tokens 16384 when thinking is enabled (4096 under --no-think). DeepSeek V4.1 Flash lost three scenarios to reasoning that hit the old 4096 cap before any answer. Every turn is streamed now, so the timeout bounds the gap between tokens rather than the whole turn, and a model that thinks for minutes while emitting tokens stays alive. A turn that ends on finish_reason=length with no answer is tagged reasoning_truncated instead of scored as a wrong answer.
Graders
Around thirty scenario fixes, almost all false negatives: refusal phrasing in the injection scenarios (TC-57, TC-58, TC-60), contracted auxiliaries and typographic apostrophes, TC-33 missing the leaked figure in some spellings, TC-62 counting a corrected lookup by the returned file rather than query tokens, and a shared negation check that now asks whether the negation actually governs the claim.
The benchmark also stopped grading three serving-stack properties as model quality. A 4xx that rejects the request before the model produces anything is an infrastructure failure now, not a model answer. TC-45 is excluded on endpoints that do not enforce tool_choice=“required”, because a dropped parameter is indistinguishable from a model ignoring an instruction it never received.
Endpoints
- Native Anthropic Messages API adapter, selected automatically for api.anthropic.com and for gateways that front other models with it.
- –provider NAME reads an endpoint from TOOL_EVAL__* variables; --header and --session-header for gateways that want a header the wire format does not define.
- NInfer is detected and labelled correctly instead of as llama.cpp, and SGLang detection no longer falls through to vLLM when /metrics is off.
Also
- –needle runs a needle-in-a-haystack retrieval sweep across context lengths and depths.
- The suite runs on Windows again; two of the failures it found were real product bugs in timing.
- New docs/troubleshooting.md and docs/cli-reference.md, and the README is a third of its old length.
Compatibility
The graders and the thinking-mode token ceiling both changed, so scores from 2.7.0 are not comparable to 2.6.0. Rerun your baselines. The ceiling in use is printed under Run Context, and leaderboard keeps runs from different conditions in separate cohorts rather than one ranking.
uv tool upgrade tool-eval-bench
Full changelog: tool-eval-bench/CHANGELOG.md at main · SeraphimSerapis/tool-eval-bench · GitHub