Can someone please just help me set the DGX Spark up for optimal LLM use?

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

I’ve been wrestling with trying to just get my OEM DGX Spark equivalent (ASUS) to work even remotely optimally for a bit now. The only thing I have working right now is llama.cpp. But llama.cpp feels extremely slow at the moment. In Qwen3.6 27B, I’m getting ~10-14 Tok/s depending on the quant value of the cache/model. I tried TensorRT, but it’s somewhat unclear as to how I am supposed to actually query it.

I found some random obscure github which helped me bring image generation times down from like 2-3 minutes to 13 seconds earlier via switching to its custom ComfyUI build. What this tells me is that there is basically some obscure deployment available for this thing, for anything I might want to do, that might be several times faster than whatever I’m trying to use.

Right now I am trying this with VLLM to hopefully get some slightly faster speeds:

docker run -it --gpus all --ipc=host --network=host -v /opt/app:/opt/app -v /opt/vllm-cache:/root/.cache vllm/vllm-openai:nightly-aarch64 python -m vllm.entrypoints.openai.api_server --model /opt/app/models/Qwen3.6-27B-NVFP4 --trust-remote-code --dtype bfloat16 --speculative-config '{"method":"mtp","num_speculative_tokens":3}' --max-model-len 262144 --gpu-memory-utilization 0.85 --max-num-seqs 2 --max-num-batched-tokens 32768 --enable-prefix-caching

But I have no idea whether it’s the right way to run this. How in the world am I supposed to?

[AutoTuner]: Tuning fp4_gemm: 96%

It’s been stuck at this step for ~30 minutes, and I have no way of knowing whether it’s doing anything at this point. This process is completely killing the CPU/GPU, too, so I can’t even check in on it. It’s basically frozen.

Is there some roll-up post of the “obscure hyper optimized packages” for the Spark which are easy to execute? It’s basically a console. It’s one piece of highly unified hardware. I can’t imagine that there’s much variation once you have something that “works”. Otherwise I am considering just returning it because it’s too much hassle.

Thank you

2

A couple of pointers:

#1 You are doing it the hard way. If you want to run vllm go here:

or here:
https://github.com/eugr/spark-vllm-docker

#2 You will never get impressive Qwen 3.6 27B speeds on the DGX Spark. It’s a dense model that requires too much memory access per token. It strains the Spark’s < 300GB/s memory bandwidth. Some tricks like MTP might help a bit, but you are still in low single-digit territory either way.

#3 MoE models are the sweet spot for the Spark. Something like Gemma 26b-a4b or Qwen 3.6 35b will give you better speeds and take better advantage of the specific tradeoffs in your hardware.

Good luck.

3

Qwen3.5-122B-A10B-Int4-AutoRound was probably the best single spark recipe I came across before going to dual sparks, easy to run with the recipe in eugr’s spark-vllm-docker

4

Thank you, I will try this. My current task is some rote categorization of many words in a large context size file set. Will these MoE models still do these sorts of simple categorization requests, or will 27B do this much better? My understanding is that there are tradeoffs for large MOE models vs dense models depending on task type.

5

Ok course the devil is in the details, but I would definitely try a faster MoE model for a task like this. Denser models tend to excel at heavier problem solving tasks.

6

Search for Henry’s Qwen 3.5 122b recipe and thread. IMO, for a single Spark, the best setup period. If I had a spare machine, it would only run that.

Just note, the thread and a lot of ancillary conversation evolved the recipe and the choice of exact model variant. Worth a read before jumping in.

Ad it all pivots around Eugr setup or even Sparkrun, which incorporates it.

7

Thanks. So I read (or rather, took about 5 minutes skimming) through that long topic and there are a few takeaways for me:

This seems to be the current fastest setup.

Or maybe this one.

This seems to be a 27B setup that someone advertises to do surprisingly fast work on a DG spark.

And this seems to be “old reliable”:

Unfortunately I don’t have enough space on my rinky dink single 1TB drive for all of this stuff, so I’m going to have to start rationing.

For 122B, for my specific task, I’m finding that I need to cut off its thinking after about 10-15k tokens or so. After that, it’s not really extracting any more useful insight out of the data and starts doing completely useless garbage. But I haven’t tried the specific initialization parameters outlined in these threads; I just found one on a Reddit thread. My output speed is only 20-ish tokens/s. If I could get it to work faster via the flash flags that other people are using, that might be okay even if it takes a long time reasoning.

27B in its default state is even slower on VLLM at NVFP4 than it is on llama.cpp at q8, which was baffling. I will definitely be eager to try the prisma recipe that was posted.

I’m curious about 3.6 35B, but not sure if any specifically good setup was mentioned that beats out 3.5 122B in that thread. Supposedly some redditors have said that 3.6 35B can be quite good.

Lastly, I have a question. I’m trying to give it a soft transition out of the thinking stage, if I have to cut it off. It doesn’t seem to be working though:

    "model": model_name,
    "messages": [
        {"role": "system", "content": system_msg},
        {"role": "user", "content": user_msg}
    ],
    "temperature": 0.1,
    "max_tokens": 30048,
    "top_p": 0.95,
    "stream": True,
 "reasoning-config": {"reasoning_start_str": "<think>", "reasoning_end_str": "I have enough information to give a concrete answer set now; let me present my best answer.</think>"},
 "thinking_token_budget": 10000

My “thinking_token_budget” seems to be working, it definitely cuts it off. But the thinking end and start tags are totally not behaving as the VLLM docs say they kind of should. Any ideas?

8

35b and 122b are different tools, it really depends on what you want.

When I had 2 sparks I was running 122b on one, and 35b on the other. Latter used for high speed high concurrency workers in a Hermes kanban setup. 122b is slower and more intelligent, so I used that to co-create specs and be an adverserial reviewer to the 35bs code in a self-healing loop.

In my experience, for coding use, Qwen models with thinking on just don’t really work effectively. I have thinking switched off on mine. Chat template is another massive thing - frogeric’s templates seem to be the best to ensure tool calling.

My current set up is:

2x sparks running DS4 flash - between 40-50 t/s single user
1x spark running Qwen 3.6 35b FP8 - allows me to easily have 4-6 concurrent workers at 70+ t/s

I appreciate all the recipes on here can be confusing - personally as you’re starting out I’d stick with the tried and tested models in the spark run repo and build your knowledge from there. Once you’ve more confident in what the settings mean it gets easier.

Though this stuff is moving so quickly every day there’s a new toy to play with and it’s so hard not resist potentially breaking a working set up to try them…

9

I would suggest start with spakrrun. Most user friendly. Maybe not most optimal but easiest to get something working fast. Then eugh spark-vllm-docker. And then if you want research different images, experiment with configs etc. I did it hard way and only discovered this forum and community after I had built vllm, sglang, llamacpp from sources, watching kernel builds for 8 hours and fail in the end.. Don’t lose motivation, get it working fast, get agent going, he will help you move further. Qwen 3.5 122b is an excellent suggestion. Don’t be tempted with Nemotron 3 super, it totally works but has terrible character and inferior at tool use and coding, but has good knowledge, just terrible in executing anything. Perfect big firm consultant.

10

So far, I just have 122B running. I have had it running for half a week or so now. I took a mixture of some parameters that I found on Reddit and the “old_reliable” execution I found in that other thread. It isn’t too bad. The issue for my categorization tasks is it tends to overly reason and then reason itself into a looping corner when applied to categorization tasks. Otherwise, I am okay with its performance. Getting about 20-40 t/s, bit all over the place. When I presented my findings to the LLM itself, it suggested cutting off the 3rd speculative execution layer and reducing it to 2, since it looked like it was being accurate less than 50% of the time on average. I did that and the acceptance rate went up a lot. Speeds were mostly the same, maybe a tiny bit faster. I would like to try out those “DFlash” variants, but it’s quite a rabbit hole. One day absolutely wasted on a stupid “can’t download this HF model” even though I clearly specified the model’s file path, and it was done in the exact same way as for my current installation. It was incredibly frustrating.

Someone in another topic gave me some instructions that I can supposedly follow currently. The ones in most of those threads I linked are outdated and an absolute mess to follow because they involve building vllm at different pull requests as well as having to specify the exact commit so that there isn’t a desync in the file structure. I will try that when I get time. For now 122B isn’t exactly perfect, but it’s decent. In some ways my 5090 running 27B at q8 was a bit better, in a lot of other ways, the responses from this one are a lot more “accurate”. The thinking loops are kind of an issue though, but I need for it to think at least a little for my tasks.

11

What does one do when the sparkrun recipes fail?
The @spark-arena/f1cbb973-cde2-4d0e-9ebf-538ece1993e9 for google/gemma-4-12B-it straight from the home page?
I’ve stumbled across them but more end up not running for me than do. (single 128GB node)

12

I guess I’m going to document my little journey here a bit. I could be completely off base and posting outdated garbage here, but either way:

I wanted to try to get more performance out of 122B, and it was looping while thinking, or just thinking too much. So I thought it would at least be a reasonable tradeoff when it looped, if it at least looped faster. It was also useless when not thinking, so that wasn’t an option. As far as performance, I’ve seen up to 50 t/s (if I read the output right anyway) on a simple script, and this is what it looks like on about ~75k token document processing, with a 2k token prompt:

image
image1918×1135 292 KB

I had a lot of trouble getting this optimized version to run, so I guess I’m posting what I needed to do to get it running here, in case it helps anyone.

The repo is here, I saw it mentioned in another topic on this forum.

But it doesn’t work out of the box (big surprise, right?).

First of all, these fixes (found in the “issues” section of the repo) need to be applied to install.sh:

sed -i 's|TORCH_VERSION="2.12.0.dev${TORCH_NIGHTLY_DATE}+cu130"|TORCH_VERSION="2.12.0+cu130"|' install.sh
sed -i 's|TORCHVISION_VERSION="0.27.0.dev${TORCH_NIGHTLY_DATE}+cu130"|TORCHVISION_VERSION="0.27.0+cu130"|' install.sh
sed -i 's|TORCHAUDIO_VERSION="2.11.0.dev${TORCH_NIGHTLY_DATE}+cu130"|TORCHAUDIO_VERSION="2.11.0+cu130"|' install.sh

#On line 382 — change ONLY the replacement-side URL.
sed -i 's|(${TORCHAUDIO_VERSION} triton --index-url https://download.pytorch.org/whl/\)nightly/cu130|\1cu130|' install.sh

(the last sed doesn’t really work cleanly for me though, so just replace the line yourself in vim)

Finally, the run of the actual vllm container then proceeded to not actually run properly for me. With the help of free ChatGPT and some googling, I managed to refine the run line to be a bit more optimized (think it’s around line 588):

LAUNCH_CMD="docker run -d --name vllm-qwen35 \
    --gpus all --net=host --ipc=host \
    -v ${MODELS_PARENT}:/models \
    -v /opt/app/vllm-cache:/root/.cache/vllm \
    -v /opt/app/torch-cache:/root/.cache/torch \
    vllm-qwen35-v2 \
    -c 'pip install \"fastapi<0.137\" >/dev/null 2>&1 || true; exec vllm serve /models/${MODEL_BASENAME} \
    --served-model-name qwen \
    --port 8080 \
    --generation-config auto \
    --trust-remote-code \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --max-model-len 262144 \
    --gpu-memory-utilization 0.90 \
    --reasoning-parser qwen3 \
    --reasoning-config '\\''{\"reasoning_start_str\":\"<think>\",\"reasoning_end_str\":\"</think>\"}'\\'' \
    --max-num-batched-tokens 8192 \
    --attention-backend FLASHINFER \
    --speculative-config '\\''{\"method\":\"mtp\",\"num_speculative_tokens\":2}'\\'''"

By default, this installed with fastapip>=0.137 in its VLLM build, which completely breaks v1 completion url (still apparently an issue). Just making it install 0.136.3 fixes that.

“reasoning-config” is needed to give it a maximum reasoning length for some reason. I’m sure others are using harnesses that can detect looping around this thing, but for me I have a pretty simple task so I need it to just hard stop.

Supposedly this github also works particularly fast on Qwen3.5-35B-A3B. I will try that later, maybe it’ll be better for my purposes. I will also try the Prisma 27B soon maybe, hopefully I can get it working without absolutely everything going wrong somehow.