Server default is 75 on the Mia recipe so you’ll be on the same settings as me - which I’ve mapped to xhigh for Hermes, and can set to 100 via /reasoning max. I’m currently testing my Hermes core patch for the looping, if it ever happens to you shout and I’ll share it (if I manage to get it stable).
@meanaverage - some more detail on the reasoning budget dial from my 4.1 itself:
4.1’s reasoning effort is a documented 1-100 dial that the encoder writes into the prompt as a literal line (Reasoning Effort: 75 (range 1-100, the higher the value, the more thorough the reasoning)) at index 0 in thinking mode, so it’s learned compliance rather than an enforced ceiling; DeepSeek’s table is low=50, high=75, max=100 with no xhigh (that’s SGLang’s alias), and we run 75 because 100 measured 3x the reasoning tokens for about 3 points of tool-eval score.
That’s also the only drawback I currently experience with that model. Apart from that, it is indeed a quantum leap that makes a Spark cluster appear sufficient for many tasks.
MiMo 2.6 Flash has achieved a similar leap forward, and has done so with a significantly further reduced number of parameters. However, the drop in speed during long context sessions renders it virtually unusable for larger interactive sessions.
I’ve spent time working on it with 4.1 this morning.
Mia’s TP4 recipe I use has a loop abort in, DSV41_LOOP_ABORT. It cuts off output and can break real replies without fixing the build-up - it:
Watches the model while it generates and cuts the reply off when it spots exact repetition.
It cuts anything: reasoning, visible answers, tool calls. It’s chopped off multiple real replies for me - so I turned it off
It only catches exact repeats (4 identical copies, or 8 identical lines of 16+ characters). The loops I’m seeing are short, varied lines (“Let me go.” / “Emit.” / “OK.”), so it most likely wouldn’t have fired.
Even when it does fire, the loop text already generated still gets sent back to the model on every later turn, so the build-up continues.
I’ve patched Hermes core today with a fix for the specific loop issue I’m seeing:
Acts after the reply arrives, once, and only on the hidden reasoning that gets sent back to the model. It never touches visible answers or tool calls.
It’s built specifically for the pattern I’m seeing (short, varied, repeated lines) and keeps the genuine reasoning before the loop starts.
It stops the loop feeding itself, which is the thing that causes it to get worse and worse
Tested across our fleet for false positives and doesn’t affect the prompt cache.
Will monitor over a few days of hard use and see if it does the trick.
Running the same here (basically) – it’s a modified Mia’s recipe:
knapcio/DeepSeek-V4.1-Flash-4x-DGX-Spark-TP4 at 58f23215
DSV41_LOOP_ABORT=1, DSV41_LOOP_NGRAM=32, DSV41_LOOP_REPEATS=4, DSV41_LOOP_LINE_REPEATS=8 in DEEPSEEK_ENV
Confirmed again that I haven’t seen any looping, but from my history it looks like I’ve only compacted once. The 25 longest threads so far have been 500-750K, but compaction has been rare. So, very much looks like something that must be affected by compaction (ie. very large cold pre-fill? agressive draft head + the kind of input compaction creates?)
Might be interesting to try compaction summary text as cold input and see if looping starts immediately with that same input on a fresh thread.
Interesting! I haven’t run into the “Let me go / Emit” loop on my vLLM build, though to be fair I only have ran it for a couple days worth of tokens. Since it’s the same native weights on a different engine, it could be a useful cross-check if you hit it regularly: spark3-vllm-ds41f. TP3 is the main configuration, and there’s experimental TP4 support for a four-node switchless ring, so it should fit your cluster. Whichever way it goes, it would help show whether the loop follows the model or the stack. If you can share a compaction summary that triggers it, I’m happy to replay it on my side too.
@christopher_owen I am using your latest update. Was the main change the size of the context? I’ve been using the prior version for a few days straight with no issues. So far I have put over 500M tokens through it with 4 concurrent streams at XHigh … prefix cache at 96% on switchless TP3.
I got the new version staged with the larger context parameter and will switch to it later today.
I’m just about to release a version that’s finally focused on decode performance, by making TileLang work on SM121. If you’ve been following along, I’ve been doing maintenance until now, like removing non-determinism and other things.
I’ll put a full write-up in the other thread, but here’s a sneak preview. It’s the same software on 3 and 4 Sparks, each at its own limits. Numbers are aggregate tok/s at temperature 0, 256 output tokens, with reasoning on, so every measured token is reasoning text:
I would love independent verification of these numbers. These are from the tests I use for development in this repo.
On your other questions:
1M context: yes. On 4 Sparks it runs the model’s full 1,048,576-token context with 16 sequences. The KV cache (10.5 GiB per node) holds about eight full 1M windows at once, and a 1M prompt prefills in about four minutes. On 3 Sparks I cap it at 512K and 8 sequences, because memory is the limit there.
Memory: at TP=4, every node kept at least 20 GiB free at its lowest point during the benchmark, so the server takes roughly 100 GiB per node. At TP=3 the low point is about 6 GiB free, with a 3.5 GiB KV cache. I should probably tune it more, right?
So the fourth Spark gets you:
about 20–25% more decode at 1–8 streams;
about 35% more prefill;
twice the concurrent sequences and twice the context, with plenty of memory left over.
Hey j0n and thank you! 👋
I use a mix of paid American and Chinese models (all of the ones you would expect!), plus my local setup. What are you using on your end?
I’ve tried to make the docs easy to follow for both humans and agents. I also added bin/spark doctor to help with OS tuning and getting everything ready. Maybe I have left too much documentation in the README.md and it would benefit from a Quick Start section at the top?
If you give it a go, I’d love to hear how it works with your setup! Feel free to ask questions or point out anything confusing - even a quick “I got stuck here” would help me make it better for everyone.
Thanks - looks good. Not able to take my models down during the working week when they have a lot of work to do, but I’ll look deeper into your solution this weekend if I get the time. Parting with a working stack isn’t an easy decision when so much work is being done by it - the risks are big.
I’m not switchless by the way, mine is a TP4 stack with a 100gb Mikrotik.
I’ve been using Qwen Code together with Qwen 3.8 Flash Next and made this game with the combo! I had to use Claude Fable 5.1 when Qwen got stuck on some of the tricky geometry (quaternions and 3D spatial rotation).
A Quick Start at the top of the README.md would be fabulous, as otherwise it reads more like a record of the (phenomenal) investigation work that has been done rather than telling a human user what to do. But I don’t want to take away from what you’ve achieved as it’s incredible work. What was less clear was if I had to install your dgx-spark-memory-saver and/or whether I would need to manually configure the kernel to the 64 KiB profile myself or if your repo would do that. And I suppose also what I had to do (regarding @stu.miller’s comment) if I don’t have a switch to set up the switchless RoCE routing. Some of these are quite scary steps, so clear instructions for what order they should be done in, and how, and how to revert all the changes would be reassuring. Having said that, I only have two Sparks so I’m afraid I can’t test it out yet!
Thanks for the kind words, and for taking the time to explain where things were unclear! Thanks for sharing the game, too - interesting to hear where you needed another model’s help with the geometry. It’s pretty fun!
That’s fair feedback about the README. It’s grown around the investigation, and it needs a clearer starting point for someone wanting to use it. A Quick Start at the top makes sense.
I’ll make sure it explicitly covers:
Whether dgx-spark-memory-saver is required or optional for each configuration.
Whether the 64 KiB kernel needs to be installed separately, and what the repo handles automatically.
The switchless RoCE setup, including cabling and routing.
The order of operations, checks after each stage, and how to revert the changes.
You should be able to understand what will change on your machines - and how to undo it - before starting. For example, there is a real trade-off when booting without the graphical user interface as the recipe suggests.
And no worries about only having two Sparks! This feedback is useful on its own. I’ll also make the hardware requirements and tested configurations clearer so people can tell up front what applies to their setup.
