Thanks for this recipe, Marco. It has been the model server under a week of work on our Thor, so here is what we built on it, and a few things we learned running it under a real agent workload.
We use it (your FP8 hybrid, 3 slots, 256K) as the local model behind Smithy, a Rust agent IDE (this is its Windows fork), for unattended Runs: one sentence in, then a branch of commits whose tests pass and a morning report out, with nobody at the keyboard.
Repo: GitHub - Divhanthelion/Smithy-Windows: Smithy for Windows: a native Rust IDE and coding agent on your own model, with Jev checking every step and unattended Runs that turn one sentence into tested commits. · GitHub
Notes on the recipe itself, in case they help anyone else:
- Pin the KV cache with
--kv-cache-memory-bytes. After a fresh boot the Thor counts page cache as used memory, so a fraction-based budget (--gpu-memory-utilization 0.75) came out at −0.71 GiB of KV and the server would not start. A fixed 8 GiB (~290k tokens) has worked since.
- Prefix caching is the big win for agents. A repeated 16k-token prompt went from 9.05 s to 1.0 s to first token; our Runs get about 90% of their prompt tokens from cache.
- The FP8 hybrid measured +19–27% decode and −2.7 GiB against the base checkpoint for us. Single stream with a 16k shared prefix: 42.9 / 28.7 / 47.1 tok/s on code / prose / JSON. Three streams together run about 1.4× one.
- Power mode: at MAXN, sustained inference tripped the over-current counter (oc3) hundreds of times an hour. At 90 W, three hours of generation tripped it zero times.
- A second model beside it: a 4B model loaded next to Flash-Next costs nothing while idle, and 12–62% of Flash-Next’s decode while busy (they share memory bandwidth).
The write-up is below. The full technical report, with the numbers, commands and appendices, is here: Smithy-Windows/reports/2026-09-24-unattended-runs-on-the-thor.md at main · Divhanthelion/Smithy-Windows · GitHub
Teaching a local model to work while we sleep
Smithy’s unattended Runs take one sentence, build X, and come back with a branch of commits whose tests pass and a report of what happened, with nobody at the keyboard. All of it ran on a Jetson AGX Thor with a local model, Qwen3.8-Flash-Next: nothing per token, and nothing leaves the desk.
One rule: the compiler and the tests decide when work is done, not the model. A task commits only after the runner’s own checks pass, nothing has regressed, and no existing test was weakened or deleted. The model can’t commit, reset or push. Around that rule sit three things:
- a planner that gives each task the command that proves it
- Jev, a small judge model, watching for loops, false "done"s and loosened tests
- research whose every quote is checked against a saved copy of its source
From 3½ hours to 53 minutes. Four runs of the same toy intent, an ISO 8601 duration parser with a command-line tool:
| Run |
Result |
Time |
| 1 |
1 of 6 tasks, stopped by hand |
3 h 30 m |
| 2 |
3 of 3 |
2 h 23 m |
| 3 |
3 of 3 |
1 h 49 m |
| 4 |
3 of 3 |
53 m |
What made the difference:
- Prefix caching. None of the first run’s prompt came from cache; later runs got about 90%. A repeated prompt’s first token went from 9 s to 1 s.
- Research sized to the question. Questions run three at a time and are kept in a shared library, so nothing is looked up twice.
- A task’s turn ends the moment its checks pass.
- The FP8-hybrid checkpoint, which decodes 19–27% faster.
- Thinking only where it pays. Quick lookups skip it; harder questions keep it.
- No per-reply output cap. It kept cutting off good work. The lesson: a limit should be a backstop that never fires during good work.
Every real failure (loops, a false “done”, a missing toolchain) was caught by something watching what the model actually did, the tests or Jev, and never by a blunt limit.
A real project. Then we asked it to add the molad and Daf Yomi to our Rust Hebrew calendar. It finished all three tasks, and the result is right. It disagrees with Hebcal on none of 13,991 days of Daf Yomi (1923–2040) or 351 molad announcements, and 41,030 days that worked before still do. It needed a person six times, never for the code: a laptop restart, three Smithy bugs, over-current throttling at full power (gone at 90 W), and a missing lint tool.
A local judge. Jev is the one piece that needs the internet. JevK5, a 4B model running beside Qwen on the Thor, gets 7 of our 63 test cases wrong, against 1 for the hosted Jev. None of its misses is a destructive command, a weakened test or an unfinished task. It answers in about 0.35 s while Qwen is busy, but its thresholds still need tuning.
Honestly: at 30–45 tokens a second, the Thor spends an hour on what a frontier API does in minutes. Overnight, private and free, that’s a good trade. The checks already make the results trustworthy; what’s left to fix is everything around the model.