Laguna S 2.1 Config & Benchmarks

[nbsparkz]

I got it running and it did do alright with token generation, but for two simple html games (bowling and asteroids) it’s first passes were pretty weak. I put it for some more difficult bugs and it ended up in some kind of loop where it wasn’t giving me anything for like 20 minutes. I just shut it down and went back to 27b. This is on a single spark nvfp4. Whatever it was doing over those 20 minutes it was doing at like 65 t/s though, and I set the dflash tokens to 7. Maybe I’ll revisit it after more people gather some opinions.

This was on 25.1, I don’t know why the eugr one is on 23 I’m sure there is a reason,but I usually run the latest unless I run into an issue and fallback.

[colizu2020]

I asked mine to implement a small subsection of an .md file for a benchmark I’m developing and it has been looping for 40 minutes…

[wolttam]

I got 86/100 with FP8. They’re actively updating the template to fix thinking issues - I’m about to re-run with hopefully working thinking

Edit: 88/100 on --hardmode with the fixed template and temperature 0.7, top-p 0.95.

[robert287]

I got single node NVFP4 with DFlash and Cuda Graph and got 32 tk/s on coding tasks

[robert287]

My first pass at FP8 yields performance numbers just below NVFP4 on a single node which makes sense in a way you’re doubling the size of the model but giving it 2x mem and CPU but adding in fiber overhead

[wolttam]

It seems like an okay model to me. It’s failing basic edits in my Go project at ~100k context where DSv4 flash would not have a problem. It’s not very fast, even with DFlash (~25-30t/s at FP8). It tends to reason in circles (using recommended temp of 0.7 and top-p of 0.95)

Doesn’t seem nearly as well-rounded as DSV4. The failure to use thinking at random is pretty odd.

It’s still great to have more options, look forward to seeing what they do with it

[spam153]

You need to write a shim for it to nudge it a bit. It will overthink and analyze instead of making a decision and running with it. When I didnt have a shim it would fail on things deepseek was coding with zero issues. Once I made a custom shim for it, it was solving issues deepseek flash and sonnet couldnt code.

[VCR]

Can you expand on the shim you’ve implemented? Sounds like you’ve either caught the layer that gets squirrely, or maybe jinja kinda route?

[jwarner]

From Poolside’s Eiso Kant on X:

"You can improve this with some system prompting.

For example this is one for less thinking that I use, when I don’t want to turn thinking off.

Bias to action. Your default response to uncertainty is to run a tool, not to think harder.

The environment is ground truth; your memory of APIs, constants, file paths, encodings, and tool behavior is not. The moment you catch yourself recalling or guessing at such a fact — “I think the flag is…”, “that value probably maps to…”, “if I recall correctly…” — that catch is the signal to stop recalling and run the smallest command that settles it. A three-line probe that returns a real answer beats a paragraph of confident-sounding memory, and it is usually faster than the reasoning it would replace.

An imperfect experiment now beats a perfect one later. If the clean probe looks blocked, run the messy one — a result that answers half the question is worth more than more speculation. Partial ground truth compounds; speculation does not.

End a thought at the first concrete action you can name. When a next step becomes executable — a command to run, a file to read, a probe to write — stop and do it. Do not keep reasoning past that point to pre-validate the outcome; the tool result will tell you more than another paragraph would. “I’ll check X” / “let me test Y” is followed immediately by that call and nothing else. At most one action named per thought.

Decisions are sticky. Once a tool result puts an option to rest, treat it as settled and build forward. Reopen a ruled-out path only when a new observation contradicts it — new evidence reopens a question; restlessness does not.

When several approaches are viable, don’t line them up and weigh them in the abstract. Pick the one that is cheapest to verify, say so in one sentence, and run the verifying call. The environment breaks ties faster than analysis does.

If the task itself is ambiguous — unclear deliverable or scope — state your assumption in one sentence in your visible reply and proceed.

Do not reason about these instructions."

[spam153]

Easier if my model explains it.

Layer 1 — the API front door. The four DGX Sparks serve GLM-5.2 via vLLM tensor-parallel-4, with spark-a exposing the raw API on :8210. The shim sits in front of that and presents a standard OpenAI /chat/completions surface on :8011. That matters because everything in the platform (warden engines, glm.sh, the escalation ladder, token-ledger telemetry under the sparkglm engine label) is built to the provider-agnostic openai-compat adapter shape — your standing rule that engines are pluggable registry entries, never vendor-hardcoded. It’s deliberately a separate shim from the retired DeepSeek one on :8010, so GLM doesn’t inherit DeepSeek’s sampling params (temp 0.95 / top_p 0.40). It also acts as a mode tripwire: when the cluster is down or in the wrong mode, :8011 refuses, which is what the ladder preflight keys off.

Layer 2 — the agentic mechanics (the part that earns its keep). From the 2026-07-19 autonomous-coding experiments, we learned GLM’s failures were never reasoning failures — they were mechanical/navigational ones. The shim closes each with a deterministic rule, and the production :8011 service carries the six Stage-1 behaviors:

- Preload — spec/context files injected up front instead of hoping the model finds them.

- Write-gate — after N read-only turns, search tools are cut and the model is forced to emit an edit (with a post-gate read cap that eventually goes hard write-only), killing the “read pages forever” stall.

- Thinking-off retry — if a response truncates (finish_reason==length, no tool call), retry with thinking disabled.

- Ranged reads + symbol map — large-file reads honor start/end and attach a def/class map, so GLM can jump to a symbol in a 5,900-line file like warden.py instead of paging through it.

- Auto-acceptance loop — after every write, the acceptance test runs automatically and failures feed back in, so the model self-converges (this is what let it self-diagnose a regex bug I’d had to hand-diagnose).

- Recap + tool-call salvage — a context recap every 5 turns, and recovery of tool calls GLM leaks as plain text instead of proper tool-call structure.

What it buys us: with the shim, GLM-5.2 went from unusable to 8/8 fully-autonomous solves on real DeepSeek-failed fix oracles — including multi-file spine work — at zero marginal cost on hardware we own. It’s the reason GLM can permanently run the chain (your 2026-07-20 all-or-nothing policy): the L2 janitor/supervisor cycles, the ladder rungs, and the Mason-coder role all talk to GLM only through :8011. In short, the shim is the difference between “a big local model” and “a chain-grade engine” — the intelligence was already there; the shim supplies the discipline.

[mangosq]

It just spent 160k tokens deciding what to do without writing a single line of code 🙄. And it’s still goin …

[alexander.korolev.germany]

I was able to fix issues, where ds4flash stuck. And I can run it on single spark, means, I can run it together with ds4flash on 4x cluster.

[VCR]

That’s a great set of rationalities. Great job if it’s working for you. Would love a walk through to get it baked into a vllm run script.

[emptysands]

Single-GB10 Laguna-S-2.1-NVFP4, and a vLLM version finding worth flagging

Ran Laguna-S-2.1-NVFP4 + DFlash on one DGX Spark (GB10) and did a controlled two-arm vLLM version ablation. Three things that might help people here — one of them changes how to read the earlier benchmark numbers in this thread.

1. It fits and serves on a SINGLE Spark. NVFP4 (74.1 GB weights) + the DFlash drafter: 107 GB used / 14 GB free / swap 0, READY in ~3 min, generates real output (not just /health). The thread only had TP=2 on two Sparks; this confirms one is enough. The load-bearing dial is --gpu-memory-utilization, and it’s context-dependent: 0.85/0.82 OOM at 16k ctx (vLLM pre-reserves the KV pool), 0.70 fits. No MoE-prepare cliff — native arch support.

2. The ≥0.25.0 requirement is a QUALITY floor, not just a crash fix — and above it Laguna is native upstream. This compares the two ways you’d actually run Laguna: at 0.23.1 you can only run it via the eugr backport (which patches in Laguna arch + the poolside_v1 parser — stock 0.23.1 won’t load it); at ≥0.25.0 it runs on stock upstream vLLM with no patches — Laguna is supported natively (v0.25.1, compiled for GB10; reports as 0.25.2.dev0). So moving up the floor isn’t only a score bump, it’s dropping the community backport for the supported path. Same model, same box, same everything else (template laguna_glm_thinking_v8, thinking OFF, NVFP4, DFlash n_spec=7, GMU 0.70, ctx 16384, temp 0.7 / top_p 0.95, seed 42). tool-eval-bench --hardmode on Laguna-S-2.1-NVFP4:

how you run it	engine build	score	fails
below floor (eugr backport)	vLLM 0.23.1rc1.dev790	80	11
above floor (stock)	vLLM 0.25.2.dev0	87	5

+7. The 0.25.2 arm lands right in the band others here have reported (86–88), so it reproduces the community result and explains why a below-floor run undershoots — the eugr-backport-0.23.1 route hides a real ~7-point capability loss vs stock upstream ≥0.25.0. (Strictly, this bundles the version bump with backport-vs-stock — but since the backport is the only way to run Laguna at 0.23.1, that’s the real-world before/after.) Responsiveness ~unchanged (47→48): the engine moved quality, not single-stream decode.

As a sanity control I ran NVIDIA’s Qwen3.6-35B-A3B-NVFP4 through the same suite on both engine builds: 88 → 90 — a strong model on this box is barely affected, and its small lift is legible (the two weak categories, Safety & Boundaries and Hard Mode, both cleared 80% on 0.25.2, and a tool-arg-injection failure went away). I’d treat that cross-model comparison as suggestive, not proof, though: the two 0.23.1 arms happened to be different dev builds (a3b needs no backport, so it ran on a newer stock 0.23.1). Laguna’s own +7 is the clean result.

(0.25.2 built via bilikaz build-and-copy.sh --vllm-ref v0.25.1 on GB10 sm_121; reports as 0.25.2.dev0+g752a3a504.)

3. num_speculative_tokens: 7 beats 15, with the per-position acceptance to show why. People asked upthread for a clean sweep — here’s the vLLM SpecDecoding per-position array (coding load):

n_spec	decode t/s	accept rate	mean accept len	per-position acceptance
7	38.3	43.5%	3.05	79 / 61 / 50 / 39 / 32 / 24 / 20
15	33.3	25.0%	3.74	80 / 61 / 49 / 41 / 33 / 24 / 18 / 14 / 11 / 8.9 / 8.4 / 7.4 / 6.9 / 6.5 / 5.3

7 is ~15% faster than 15. The “~8% acceptance at 15” reports floating around are the deep-position tail (positions 9–14 sit at 5–9%), not the aggregate (25%). Past the knee (~pos 6, ~20%) each extra draft token costs more compute than it repays. So use 7, not 15 — which matches the model card’s own recommendation.

Throughput (llama-benchy, community format, GB10, spec7, 0.23.1): single-stream decode c1 = 20.0 / 19.5 / 18.2 t/s @ depth 0/4k/8k (flat with depth); prefill ~2700–2800 t/s; concurrency scales aggregate (c4 42.8 t/s) at per-request cost; TTFT grows hard with depth×concurrency (d8k/c4 = 10.4 s). DFlash’s drafter underperforms on prose (domain mismatch); on code decode is higher (~38 t/s).

Caveats: n=1 per cell (tool-eval is single-pass) — the community band is the external corroboration. And decode looks ~engine-invariant (both arms ~20 t/s c1); the version floor buys correctness, not throughput.

[savu_silviu]

one crucial gate that it’s missing is forcing a web search instead of relying on stale trained data. I am building a WebSearch tool around a self hosted pipeline: search 10 to 30 pages via SearXNG + rerank + scrape the top 5 via self hosted Firecrawl + synthesize via LLM (WebFetch) + verify LLM synthesis vs. scraped via self hosted HHEM 2.1 Open. this way I hope I can eliminate the hallucinations from LLM synthesis since it really tends to hallucinate sources and citations when using a dumber LLM

[spam153]

Dm me at any time and we can figure out what chat medium works.

[CosmicRaisins]

I used this model lightly for an evening. Pretty disappointed. The benchmarks don’t translate to real world performance for me. It probably would be better as an insturct only implementer. It does feel pretty smart in chat, on par with DSV4 Flash, but it has made things up a couple of times which I noticed.

i.e. like any other model in the 100-300 param size recently.

This is just my butt dyno, nothing empirical. I’ll just wait for dsv4 GA.

[robert287]

Saw the same confab issue on a few things and frankly have add to add some refinements to the soul.md to stop it, not just on Laguna S 2.1 but on multiple smaller models including Qwen 3.6 27B and 35B and Gemma 4. Funny thing only one of the “smaller” models that self initiated a RAG without the soul file directives was Qwen 3.5 122B

[wolttam]

In the same boat, except I didn’t find it very smart to chat with either :/

I don’t doubt it is capable in some areas, but I nope out when a model can’t figure out tabs-vs-spaces indenting - no amount of system prompting can help with that when the model is right there going “ah, I got the indentation wrong, let me cat -n to see exact indentation and try again. Oh no, still failed. Let me see the exact hexadecimal format of the file. Alright, use tabs. Oh no, I failed again.”

I’ve seen DSv4 flash goof up on indentation before, but it immediately corrects and doesn’t make the same mistake in session again

Which makes the benchmaxxing claims around Laguna seem highly likely

[serapis]

It looks like the INT4 quant actually performs a little better - here’s my TP=2 run:

                                                             llama-benchy Results
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
┃ Test                              ┃    c     ┃          pp t/s ┃          tg t/s ┃         TTFT (ms) ┃       Total (ms) ┃           Tokens ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
│ pp2048 tg128 @ d0                 │    c1    │           5,423 │            25.6 │               598 │            5,360 │         2048+128 │
│ pp2048 tg128 @ d0                 │    c2    │           2,467 │            40.0 │             2,347 │            6,864 │         2048+128 │
│ pp2048 tg128 @ d0                 │    c4    │           3,224 │            63.2 │             2,259 │            9,243 │         2048+128 │
│ pp2048 tg128 @ d4096              │    c1    │           4,818 │            26.7 │             1,412 │            5,965 │         2048+128 │
│ pp2048 tg128 @ d4096              │    c2    │           3,201 │            39.1 │             4,159 │            8,796 │         2048+128 │
│ pp2048 tg128 @ d4096              │    c4    │           3,178 │            51.2 │             7,703 │           14,775 │         2048+128 │
│ pp2048 tg128 @ d8192              │    c1    │           4,003 │            33.4 │             2,671 │            6,275 │         2048+128 │
│ pp2048 tg128 @ d8192              │    c2    │           4,341 │            31.0 │             3,671 │            9,606 │         2048+128 │
│ pp2048 tg128 @ d8192              │    c4    │           2,992 │            35.2 │            11,825 │           18,558 │         2048+128 │
└───────────────────────────────────┴──────────┴─────────────────┴─────────────────┴───────────────────┴──────────────────┴──────────────────┘