HOW-TO: setup-dgx-spark docker inference - A "Sane" Inference Stack for GB10 (Need Contributors!)

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10 Projects
, ,
21

You sir are a legend! I ran the FP8 but whilst i’m still waiting for my second spark the context window of 256k was too large. However, I did manage to adjust this so I could finally use AWQ with:
```
#!/bin/bash

docker run -d \

--name vllm-awq \\

--restart unless-stopped \\

--gpus all \\

--ipc host \\

--shm-size 64gb \\

--memory 110g \\

--memory-swap 120g \\

--pids-limit 4096 \\

-p 0.0.0.0:8002:8000 \\

-e HF_TOKEN="${HF_TOKEN:-}" \\

-e VLLM_LOGGING_LEVEL="INFO" \\

-e NVIDIA_TF32_OVERRIDE="1" \\

-e TORCH_ALLOW_TF32_CUBLAS_OVERRIDE="1" \\

-e VLLM_TORCH_COMPILE="1" \\

-e VLLM_FLOAT32_MATMUL_PRECISION="high" \\

-e VLLM_LOG_STATS_INTERVAL="10" \\

-e VLLM_ATTENTION_BACKEND="FLASHINFER" \\

-e VLLM_FLASHINFER_FORCE_TENSOR_CORES="1" \\

-e VLLM_FLASHINFER_MOE_BACKEND="throughput" \\

-e CUDA_VISIBLE_DEVICES="0" \\

-e PYTHONHASHSEED="0" \\

-e VLLM_USE_V2_MODEL_RUNNER="0" \\

-e VLLM_ENABLE_PREFIX_CACHING="1" \\

-e TORCH_CUDA_ARCH_LIST="12.1f" \\

-v $HOME/huggingface:/root/.cache/huggingface \\

scitrera/dgx-spark-vllm:0.15.1-t5 \\

vllm serve bullpoint/Qwen3-Coder-Next-AWQ-4bit \\

--served-model-name qwen3-coder-next \\

--load-format fastsafetensors \\

--attention-backend flashinfer \\

--port 8000 \\

--max-model-len 262144 \\

--block-size 128 \\

--max-num-seqs 16 \\

--max-num-batched-tokens 131072 \\

--gpu-memory-utilization 0.70 \\

--kv-cache-dtype auto \\

--enable-prefix-caching \\

--trust-remote-code \\

--enable-auto-tool-choice \\

--tool-call-parser qwen3_coder \\

--disable-uvicorn-access-log \\

--kv-cache-metrics \\

--cudagraph-metrics \\

--enable-mfu-metrics \\

-cc.max_cudagraph_capture_size 512 \\

--tensor-parallel-size 1

```
Works like a dream! I’ve not found anywhere else on the web that gave me a working config, much appreciated!

22

glad it helped you out man! I’ve been using a simplified version as recommended by eugr and it also works fine. I haven’t tried the AWQ quant yet! Did you run into problems running FP8 256k on a single device? I did hit OOM problems with 0.90 GPU util but with 0.80 GPU util it stands up just fine

23

I think he is asking if qwen3 next coder will be a recipe in your repo anytime soon hehe I do see that the recipe is available at Spark Arena - LLM Leaderboard though (great contrib, thank you!)

24

ahh ok, i hadn’t tried the 0.80 setup, once again utilization defeats me. Will give it a go now!
Thanks

25

I think I still don’t fully understand why everyone uses this. It is insecure and has stability consequences. What am I missing?

@eugr My mistake, it is there: spark-vllm-docker/recipes/qwen3-coder-next-fp8.yaml at main · eugr/spark-vllm-docker · GitHub
I would like to ask you eugr acuallly, it is also in the vllm-docker repo Code search results · GitHub

Why not just go withe the official way Nvidia runs containers? (Running vLLM - NVIDIA Docs - no ipc). The stack I posted runs without. This is why one defines shm-size also. Another side effect of that is that the Spark won’t crash during inference.
See Containers For Deep Learning Frameworks User Guide - NVIDIA Docs
Since we often run models with 0.8, 0.9, etc… and turn them off/on, I don’t understand why we would keep the ipc sharing.

26

It’s not required, just like priveleged, as long as you map infiniband and set shm-size, but in my testing it resulted in slight increase of performance, especially in the cluster. Probably worth revisiting at some point.

27

Running with 0.90 is risky, as it spawns some processes to compile graphs which in most “normal” setups run on CPU RAM and don’t affect VRAM, but on Spark it’s all unified, so you get OOM more easily. I limited the number of threads recently, so it should crash less often, but I normally don’t go above 0.88

28

I’m working on adding an AI on the website to help you write recipes based on the latest spark-vllm-docker and vLLM options used in the latest supported container version.

29

Added the first version of AI assisted recipe creation yesterday

30

@eugr I go up to 0.93 with no problems but:

  • I don’t mount ipc
  • I have a sufficiently big swap
31

Just a quick note for readers that I have been busy with Kubecon prep and will get to this again after that.

32

After having a serious dig into llama-swap and considering the options, I chose to stay with the pure docker-compose approach. Also, I had to stick with my custom waker, gateway, etc… not to loose auto wake-on-request, single-tenant guard, utility model and idle timeout.

A new update is in testing at the moment. The branch is Refactor model management and enhance Docker configurations + models by jdaln · Pull Request #8 · jdaln/dgx-spark-inference-stack · GitHub for people who would be interested. Although this was initially started as a refactoring, it is more like a v1-RC1 now.

33

I got a

Openwebui/openclaw → liteLLM → Lama Swap → vLLM / lama.cpp

setup

the nice thing with lama-swap is the gui.. but it is really slow on serving models
now I wonder if I shall move to lama.cpp only..

but then I need to find and download the models as gguf.

34

On my side, I will soon test with Librechat but if I had not bothered making a stack for that, I would definitely go for OpenWebUI.

Llama Swap + LiteLLM was not enough for me at least, unless I am missing some info.

35

lama.cpp Userinterface WebUI is superfast… I can load a 4B model in seconds and get an instant reply

and somehow my OpenWebUI is very slow compared to it..
takes minutes to load and thinks for ages..same model

36
37

Martin’s project proposes the non-docker native approach and adds the chat part of the stack on the DGX host. I prefer to reserve all memory to inference and run that on another server.

My stack has been improves and changes will land on main in an hour or so. Please check it out

38

That makes perfect sense. We need to get the maxiumum use out of the RAM we paid so much for.
Will check it out

39

First version I would consider stable is out GitHub - jdaln/dgx-spark-inference-stack: Serve the home! Inference stack for your Nvidia DGX Spark aka the Grace Blackwell AI supercomputer on your desk. Mostly vLLM based for now and single-spark. For the not-so-rich buddies · GitHub . I expect bug fixes requests but did my best :)

40

Working on a new version, preview in the dev branch :)