Easy-vllm — Let a code agent build & serve any model on vLLM for your DGX Sparks

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
, ,
1

Hello everyone,

I’d like to introduce an open-source project: GitHub - tbvjvsladla/easy_vllm_simulator · GitHub

Its goal is to ease the pain that newcomers hit when bringing up a new LLM on vLLM — especially the build & serving difficulties that are specific to DGX Spark.

DGX Spark raises the entry bar compared to a workstation or data-center GPU. GB10’s unified memory behaves differently from discrete VRAM, and standing up clustered distributed inference means wrestling with NCCL / RDMA configuration. Both are real walls before you serve a single token.

On these forums, people with deep LLM-serving expertise have been lowering that barrier by open-sourcing finished Docker containers, and that genuinely helps. But I think it still has a limit for beginners: a finished container is tied to one author’s model, vLLM version, and hardware choices. The moment your model, your vLLM version, or your GPU differs, you’re back to square one — waiting for compatibility, building from source, and applying one-off patches. And it isn’t only a beginner problem; even experts pay this tax again on every new model, because a hard-won fix rarely transfers to the next one.

So I took a different route. Instead of shipping a finished container, I attack the problem with a code-agent (Claude Code) based meta-harness + loop-engineering design:

  • You clone a skeleton + generation engine, not a finished product. No node IPs or NAS paths are baked into the repo — everything environment-specific is read from a git-ignored manifest, so it ports to anyone’s machine.

  • The code agent is wrapped in a harness of deterministic scripts (version resolution, VRAM estimation, KV-clamp math) + skills + a written “constitution,” and it runs a loop: build → serve → adversarially verify → improve. Each pass tightens.

  • You just approve at each fork (HITL gate). The heavy, error-prone calculations (torch pins, NGC base-image tags, KV clamps) are done by deterministic scripts, not by probabilistic guessing.

Below is one full cycle, told as Journey 0 → 5. The numbers in it are pulled from real work logs (devlog/testlog), not invented.

(screenshot suggestion: the one-cycle diagram — clone → terraform → build → serving interview → adversarial perf → self-improvement loop)

Journey 0 — “How do I even use this?”

You finish git clone, open Claude Code inside the repo, and ask the obvious first question. The agent’s answer is deliberately conservative:

It looks at your environment first. Until terraforming is done, it only gives general advice — it will not fabricate an answer like “serve with TP=8” without knowing your GPU / OS / model location. (That exact kind of ungrounded guess caused a real deployment accident — a 1-GPU machine got told to use TP=8 — so it’s now blocked by design.)

Then it lays out the map: ① terraform → ② build → ③ serving interview → ④ adversarial performance → ⑤ self-improvement, and asks the one question that branches everything: is your setup one node, or several?

💡 A tip I use constantly on real runs: big steps look silent (loading a large model across 2 nodes can take 20–30 min; a source build longer). Just tell the agent “peek every 3–5 min and report progress” and it sets up a background watch — % loaded, per-node memory headroom, next gate. You wait with confidence, and you catch trouble (e.g. memory dropping below the safety line) the moment it happens.

Journey 1 — Terraforming: teach the system your environment

“Terraforming” = filling the empty skeleton with your environment’s real values, which land in one manifest.yaml. Only the blank manifest.template.yaml is tracked; your filled-in values are git-ignored — no private info leaks into the repo.

  • Single node — ~5 minutes. You set GPU model / CUDA and, importantly, how you acquire model weights: managed (pre-loaded on a read-only NAS mount) · ephemeral (pulled into the container on demand, gone when it stops — the usual case for a plain PC) · custom (your own persistent path). Leave nodes[] empty and it runs self-contained.

  • Multi-node — the agent scans and interviews. It runs a 5-prerequisite interview (interconnect, network, SSH, Claude Code on the sub-node, sub model access), and only after you explicitly approve does it scan. It probes the sub-node over SSH and checks hardware homogeneity — cpu_arch / gpu_model / GPU count must match exactly (a hybrid 5090-plus-GB10 cluster is intentionally unsupported, because you can’t cleanly benchmark serving across mismatched GPUs). Then it measures the interconnect with ib_write_bw.

A key design choice: if the interconnect is below the pass line, the agent stops fail-closed — it does not write the manifest. “Slow, but let’s proceed anyway” is blocked in code, so you don’t wake up at 3 a.m. to an OOM from an unvalidated distributed setup. (The post includes a full HW→SW checklist for when RDMA bandwidth is low.)

✅ Verified live on 2× DGX Spark: all 5 gate branches (single / peer-unreachable / bandwidth-too-low fail-closed / multi-ready / three-way match) passed; ib_write_bw measured 208–218 Gb/s (~90% of a 200 Gb/s link). The sub-node also receives an A2A delegation key — without it, the sub’s runtime skills refuse to serve (fail-closed): only a sub whose HW the main actually verified can do real serving work.

Optional last step — host safety stack (your sudo, never unattended). On a unified-memory box, GPU OOM = whole-host hard-down, so the agent offers (Y/N, not forced) a mem_watchdog (kills the offending container before the host dies) + earlyoom + optional kdump (post-mortem crash dump). You run the install yourself with --apply; the agent explains what/why but never sudo-installs behind your back. On a discrete GPU you can safely skip it.

(screenshot suggestion: the 2-node scan → homogeneity check → ib_write_bw result, or the 2-tier safety-defense diagram)

Journey 2 — Build: the easy wheel path vs. the hard source path

Now you pick a vLLM version, and easy-vllm’s smartest part shows: the agent decides “easy road or hard road” deterministically, not by guessing. The basis is layer coupling — a vLLM version pins a torch version, and the torch version determines the NGC PyTorch base image and the build track:

  • torch 2.10.x → easy road: prebuilt wheel. e.g. vLLM 0.18.0 → torch 2.10.0 → NGC 26.01-py3, pip install --no-deps. Mostly smooth.

  • torch 2.11+ → hard road: source build. NGC’s torch is an alpha build whose C++ ABI clashes with the prebuilt _C extension — the “hard ABI wall.” A wheel would build fine but break silently at serve time, so it goes source.

When the first base-image guess fails, the agent doesn’t guess again — it greps the candidate base’s headers to prove the missing symbol exists there first, records the evidence, and (with your approval) overrides one line. Real case: vLLM 0.23.0 failed to build on NGC 26.03 (torch::stable::Tensor has no member "layout") → evidence-based override to 26.05 → built, ABI wall cleared.

And when even that isn’t enough — a model architecture stock vLLM structurally can’t run — the patch scope climbs a ladder: deps patch → source-gate patch → pin a community fork by SHA → checkpoint swap.

🔬 DeepSeek-V4-Flash on GB10 (sm_121), after 6 failures: stock vLLM hit a double hard-wall (sparse-MLA attention allows major ∈ [9,10] only; MXFP4 MoE auto → MARLIN-repack → unified-memory OOM → host down). Fix = the jasl/vllm SM12x fork (PR#41834) pinned by SHA (c766cbc6, force-push-immune) via a VLLM_REPO/VLLM_REF build-arg, plus --moe-backend humming and an NVML patch for GB10’s clock telemetry. serve#8 succeeded — KV 386,512 tokens, reasoning intact, zero host-downs.

Journey 3 — Serving-strategy interview: “can serve” ≠ “serves well”

A built container isn’t the finish line. The same model can OOM or waste half your VRAM depending on KV cache size / context length / batch. The vllm-recipe-explorer skill narrows this by interview, on one core philosophy: measurement > formula. Per-token KV formulas assume full attention and badly overestimate KV for today’s GQA / sliding-window models (measured 8× for gemma, 1.9× for gpt-oss), so it doesn’t pin batch from a formula — it serves once, reads the log, and sizes from the real numbers.

This is where the name “simulator” comes from. Because weights, overhead, and per-token-KV are model invariants (set by the model, not the card), you can measure on the hardware you have and clamp for a target GPU you don’t have yet — using vLLM’s --kv-cache-memory-bytes (an absolute-byte clamp) so the recipe is portable:

  1. Measure on your current GPU (weights / overhead / per-token-KV),

  2. Clamp to target_VRAM × gmu − weights − overhead.

So with a single GB10 you can validate “how would this model serve on an RTX PRO 6000 (96 GB)?” with a real container. Specify a smaller card and you get genuinely smaller batch/KV numbers — it recomputes every time instead of copy-pasting. (Generated recipes carry an honesty header: if host ≠ target architecture, the margin is a cushion, not a guarantee — re-measure on the target when you can.)

✅ Verified: one physical GB10 simulated RTX PRO 6000 (96 GB) / RTX 4080 (16 GB) / RTX 4070 (12 GB) — each brought up as a real container to health-200 + a real completion. For gemma-4-12b, a GB10 budget would give batch ~52; simulating the 96 GB card, the same recipe yields batch ~36 — smaller card → genuinely smaller numbers. Also verified: gemma-4-12b converging cleanly (measured batch 39 where the formula said 3 — an 11.5× overestimate), gmu tuned to 0.90 (GB10’s OS holds ~11 GB, so the 0.92 default would OOM at startup).

(screenshot suggestion: the Phase-1 estimate/rank → Phase-2 trial-loop pipeline, or a converged recipe’s .yaml)

Journey 4 — Adversarial performance check: “serves” ≠ “serves fast”

The recipe converged and the container is up. Now: is it fast enough? This journey exists because of a real mistake:

You: DeepSeek decode is at 15 t/s — is that normal?

Agent (at the time): …looking at 2× GB10 unified-memory bandwidth, 15 seems to be the ceiling.

You: No. On the same setup, decode should be 30–45.

Agent: …you’re right. I trusted my own conclusion too fast. External references (same 2× GB10 forum threads) show 31–44 t/s. The real culprits were a missing flashinfer SM120 decode kernel + MTP (speculative) being off. Fixed → 15 → 32 t/s.

That AI self-preference incident spawned the adversarial-benchmark skill: a Devil’s-Advocate that attacks the claim “this is fast enough,” armed with a triple rubric — (a) a deterministic roofline (expected speed from hardware bandwidth; it also settles “is RDMA slow?” deterministically), (b) external references (what others get on the same GPU), (c) you, as the final backstop. Measurement is vllm bench serve (warm requests only), and PASS/REFUTE is a deterministic rule, not an LLM majority vote. If it’s below expectation, it kicks the Journey-3 recipe interview to re-search — a loop that runs until it passes.

There are two tiers: a lite benchmark that runs automatically right after serving (a 5-line speed/capacity snapshot, inform-only), and a full adversarial gate that runs on your explicit approval and drives the loop. (I intentionally keep the lite one on by default even if you say “skip it” — a stability check that costs a few seconds, so nobody ships a half-speed serving config unaware. Only a firm “never run it” suppresses it, and only for that session.)

✅ Verified: roofline expected ~15.5 vs measured 15.9 (no-MTP) and ~31 vs 34.5 (MTP) — both regimes consistent; comm term 0.06% → “is RDMA slow?” deterministically refuted. The gate PASSes 34.5 t/s and REFUTEs a degraded 15 t/s with a “turn MTP on” prescription. (It even caught its own author: dogfooding the skill’s design doc, one of five Devil’s Advocates found that my roofline formula had back-fitted the numbers by omitting MTP. Fixed on the spot.)

Journey 5 — The self-improvement loop: why it gets smarter with use

Most tools stay as they start. easy-vllm doesn’t — every session you co-work with the agent accumulates as an asset for the next one. Three meshing mechanisms:

  1. Layered adaptation. The written constitution is the single source of truth. A new lesson becomes a corollary there, and the upper layers (skills / workflow / recipe) adapt until consistent — “restoring consistency is part of the same task,” no half-fixes left behind. (The reverse is forbidden: convenience never bends the constitution.)

  2. A librarian (wiki-desk). It weaves all work history (plan / devlog / testlog / simlog) into a deterministic relationship graph and, when you start new work, recalls “what worked and what failed for this model + version” in authority order (execution-truth > plan-intent). It deliberately does not index the constitution — so the library can still challenge it (anti-confirmation-bias).

  3. Main↔sub improvement role. On multi-node, a sub-node’s insights are recovered as documents only (no code/config extraction), read by the main (HITL), and folded back via a template→render→deliver pipeline — self-improving tooling that never leaks the sub’s private info upstream.

📌 I called the librarian while writing the project’s README: at install/verify time it was 61 docs / 137 edges; now it’s 93 / 249. “It accumulates with use” is literally visible in the data.

Closing

That’s one cycle — clone → terraform → build → serving interview → adversarial perf → self-improvement. When a new model or vLLM version drops, you spin the cycle again with one line (“update to vLLM X”), a little easier each time. Three safety guards hold it all up:

  • 🔒 No “done” before a real smoke test — you must serve and get a non-empty completion. (lint passing ≠ serving.)

  • 🔒 No unattended auto-download — model acquisition mode is chosen at terraforming; a missing model is fetched only after a user-approval gate. Never silently.

  • 🔒 Deterministic scripts for version resolution, agent judgment at HITL gates — torch pins / NGC tags are computed, not guessed; you confirm at each fork.

Honest scope: validated primarily on 2× DGX Spark (GB10, aarch64, sm_121a, unified memory, CUDA 13.2). First cross-hardware proof (x86_64 / RTX 5090 sm_120 / WSL2 single-node): the full functional cycle passed, but the performance gate (Journey 4) was *not run there. Target-GPU simulation validated for RTX PRO 6000 / 4080 / 4070 from a single GB10. Non-NVLink multi-GPU and other axes are designed-for but unverified — if you run it on different hardware, that itself is the project’s next chapter.

Repo: GitHub - tbvjvsladla/easy_vllm_simulator · GitHub — issues and feedback very welcome, and ⭐ stars are always appreciated. 🔥

(There’s also a “hint” layer: verified serving recipes published as hint/<vllm>/<model>/<arch> git tags — a map, not an answer key — so a stuck agent doesn’t burn tokens re-exploring a wall someone already got through.)

2

The idea is very good on the beach, and the agent fighting with the LLM works. The OEM is the most dangerous because the node falls and nothing happens until it returns from the beach. We have to test it. Thanks for your work.