The setup (scaffolded and reviewed before buying the box - configs, scripts, runbooks; CI covers lint/static checks only, nothing has touched real hardware yet):
DGX Spark FE arriving soon. The plan: an always-on team of 8 agents (orchestrator, planner, coder, QA, security, ops…) doing real project work, with all routine inference local.
Serving is built on spark-vllm-docker:
:8001 — Qwen3.6-35B-A3B NVFP4, resident 24/7 for the high-frequency agents (Marlin path)
:8002 — GPT-OSS-120B MXFP4, started on demand for heavy coding/architecture/review work (CUTLASS path, avoiding the Marlin wrong-token issue), stopped when idle
sync + drop_caches before every 120B cold load
tmux/systemd supervision now; llama-swap + LiteLLM as a router is the planned phase-2 upgrade
Resident model stays up while the heavy one loads — both fit in 128 GB UMA by design, with headroom budgeted for KV.
MTP is configured optional-with-fallback, not load-bearing. Box is deny-by-default for egress; everything binds to localhost.
I’ve been through the FP4/NVFP4 PSA, the Qwen3.6-35B mega-thread, and the 122B v2.1 thread — the questions below are the gaps I couldn’t close from reading:
Heavy tier choice: for agentic coding on a single Spark, would you take Qwen3.5-122B-A10B with the v2.1 patches (~51 tok/s) over GPT-OSS-120B MXFP4 today? Any quality or stability trade-offs you’ve hit beyond raw tok/s?
NVFP4/MTP delta: as of the v2.1 patches and PR #39920, is Marlin still the recommended path for Qwen3.6-A3B NVFP4 — and given the recent MTP illegal-memory-access report on a 120B NVFP4 (Jun 4 thread), is MTP safe to make load-bearing yet, or should it stay an optional flag?
Two vLLM instances on one GB10: any guidance on splitting --gpu-memory-utilization between a resident 35B and an on-demand 120B on UMA — and how much KV headroom you’d reserve for long multi-turn tool loops?
Cold loads: is drop_caches still the best mitigation for slow mmap on big model loads, or is there a better current practice (or a fix coming from the Spark team side)?
24/7 duty cycle: for those running a resident model around the clock — any thermal/power stability practices you’d recommend? The shutdown-under-load and power-draw-degradation threads are mostly about load spikes; I’m trying to learn what months of continuous residency look like before I commit an agent team to it. (Planning a fieldiag + sustained-load burn-in during week one regardless.)
Will post configs and llama-benchy numbers back to this thread once the box is running.
If anything above is wrong or naive, that’s exactly what I’m hoping to find out before day one rather than after.
THIS is what I’m trying to do also. I just got mine on Thursday night and haven’t had time to even really do more than log in and install tailscale. I did the one line ollama install with gemma4-12B and pointed hermes at it from a different server and it works but that was just a sloppy excited “check out my new toy” knee jerk reaction. My goal is to fill up a backlog in paperclip and just have hermes chew through tasks all night and then I’ll be hands on with it the following day clearing roadblocks, responding to questions and reviewing the previous nights work. You know, exactly what I’ve been doing for the last decade but with just a computer instead of people
I used to run precisely this before I got a second unit:
hermes with agent (just one agent with sunbagent configured with different endpoint serve)
qwen 3.6 27b with mtp 3 Nvfp4 at 21-22 tps for main agent and yarn to 512K session (36gb per 1m q8 tokens) at 0.4 ram
Nemotron Cascade 2 30b A3b with 60 tps for sub-agent with total 2M cache (very small token, around 12gb per 1m in q8) and 4 seqs at 0.35 ram
Works really well, but the startup scrip must launch 27b first, wait for serve to be available, compress and defrag vram, launch cascade 2.
Main agent has deep model, subagents have fast model.
However in reality most tasks still executed by the main agent, so this extra complexity is barely warranted and actually not beneficial. Later one I switched to one model, first mistral 4 small 119b a6.5b once vllm 0.21 released and then finally to just qwen 3.5 122b which was a best balanced model for me until a stable recipe supporting 500k+ context with deepseek V4 flash finally landed and I switched to it.
I would say go for just qwen 3.5 122b Intel autoround int4 and don’t overcomplicate. But it’s just mo
Spinning down a model and spinning up another takes too long (10 minutes or so) if you plan to use it as something you sit and watch. If you’re ok with the delay, then it’s feasible.
I did this for a week or so, and then bought another spark, so that tells you something… :)
(and then bought a third a week after that)
I was pretty happy with Qwen 122b on a single spark as the “brain” roles, and 35b as the 8x concurrency developer. But I did use 122b as part of hermes kanban to verify everything the 35b agents did before passing it.
If you’re happy with using 35b as a verifier as well as a coder, then it’s doable. But I’d want to spin up 122b at the end of a kanban phase to check everything and send it back if not up to scratch.
FWIW I just spun up Deepseek 4 Flash on 2 sparks linked as a test to replace 122b, and what I’m seeing so far is near sonnet quality at about the same speed I was getting out a single spark with 122b.
My wallet is small but I also encountered the needs to run multiple models concurrently. Beside running Qwen3.5-122B on my DGX Spark, I dig up my old notebook with RTX 3060 and use it to run Hermes/PI agents and other smaller models.
If you want to keep it local and short on cash, you can do what I do and search for cheap second hand noteboooks with RTX 3XXX or RTX 4XXX. Those will be the cheapest way for your to add CUDA capability.
Totally feel the pain. I justified as a business expense. But running hermes on the laptop and pointing at a spark is totally viable unless you truly need always on. MoE is getting so much better at running on sub 16gb VRAM these days.
The “complexity barely warranted” warning is valid, and I want to push on it honestly. One structural difference that might save me from your outcome: my orchestrator/brain isn’t local (it’s a budget capped cloud model) so the local tiers are pure specialists. The resident 35B handles the constant light traffic (triage, planning, ops heartbeats), and the big model only wakes for queue-driven coding/review where a slow spin-up is invisible overnight. If I still end up where you did (the big one doing everything) then 122B-int4 resident is exactly my recorded fallback, and you’re now the third person to recommend it, so it’s a strong fallback. Question: what does your “compress and defrag vram” step actually run between serves? That’s my still open Q4, is it sync + drop_caches, or something better?
Yes, “if you’re ok with the delay, it’s feasible” describes my operating mode exactly, the heavy tier does overnight queue work, I review in the morning. And I’m outright taking your phase-end verification pattern idea -batching the big model’s QA pass at the end of a kanban phase instead of per-task is fewer spin-ups for the same gate. (“then I bought a second… and a third”- my wallet read that and left the room…)
I’ll be totally honest - I haven’t tried automating hot swapping models. When I did that, I did it myself. And yes - shutting vllm down, ensuring vram was totally free (which isn’t instant) was a big part of that.
Your 2nd comment and your wallet leaving the room - totally get that. I was spending +$1k a month on API credits to generate content I can now do one 3x sparks for the cost of electricity - I’ll pay hardware cost back in <12 months. If you don’t have a verfied business comparison the economics won’t work out.
But - one spark in exactly your use case - frontier API model fronting and using the spark for always on /goal type stuff while you sleep without burning API credits? Absolutely valid.
122b autoround is my recommendation - fast enough and very strong. But I’d also test 35b - insanely fast and I rely on it for all my worker roles and it barely slips even at 8x concurrency. Just needs tight instructions.
Really appreciate this one, absolutely valid from someone running three of these is the most reassuring thing I’ve read all week :)
Your payback math lines up with mine almost exactly. I modeled the whole thing against current API pricing before committing (capped cloud orchestrator, everything heavy running local for electricity), and it lands in the same zone, so “verified business comparison” is exactly the bar I used. Glad to hear it holds up in practice and not just in my spreadsheet.
And good to know 35B stays solid at 8x with tight instructions that’s precisely the bet I made for the worker roles. The “vram free isn’t instant” detail is going straight into my swap automation notes too.
Personally have 2 agents Openclaw with Hermes for backup for each with Qwen3.6-35B as the main LLM for all 4. Then I have an API to Gemini Pro as the “brain power” only when needed, which I use about 5% of the time. I tried multiple LLMs on a single Spark, but this was just so much smoother and average about 60t/s.
Plus any day now Qwen3.7-35B or higher will be out and be even more highly tuned for Agents and surpassing reasoning of the frontier models just 6 months ago.
Off the top of my head, I think it’s in the —override-generation-config and it’s “presence_penalty”: . Low values don’t penalize repeating token generation, higher values do. Useful for avoiding loops in agentic contexts.