DGX Spark memory not released using llama.cpp?

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

I have been using a single node DGX Spark to serve qwen3.5 122b-a10b via llama-server. I have noticed that the memory consumption seems to be constantly going up. Closing chats does not seem to free up any space. Is this normal behavior? I have a separate RTX 6000 workstation and the VRAM seems to be cleared out instead of accumulating.

3

Not normal.

Presumably you’re using this in a nonstandard way, maybe not in a container. That model should be run at int4 using vLLM.

There are a couple of threads around here regarding llama.cpp, a recent one is for Step-3.5-Flash and has a solid and extensible containerized infrastructure - if you need llama.cpp, recommend setting that up.

4

Yeah it was bare metal deployment. I am getting great speed at about 23 t/s for Q5_K_S. I guess vLLM is the preferred inference engine for Spark? Do you happen to have a good suggestion of getting started via docker + vLLM?

Oh maybe one thing to add. I am serving the model for OpenWebUI to use. That should not be a problem right?

5

eugr’s spark-vllm-docker and sparkrun are your best friends

That said, you WILL run into situations where llama.cpp may be needed. It’s lighter, so when you must squeeze out every last possible bit of memory it is the way. Like for Step-3.5-Flash

6

Nice!! Thanks!

7

You will see single threaded 26-27 t/s with qwen3.5 122b-a10b using int4-Auroround, and about 43 t/s total with concurrency = 2 on vLLM. Along with much faster prompt processing.

8

Music to my ears lol

9

Did you compile it?

llama.cpp runs great for me with these compile settings:

cmake -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_NATIVE=ON \
  -DGGML_CUDA=ON \
  -DGGML_RPC=ON \
  -DCMAKE_CUDA_ARCHITECTURES=121a-real

cmake --build build -j$(nproc)
10

Yeah I compiled it

Compiled with
cmake -B build
-DCMAKE_BUILD_TYPE=Release
-DGGML_CUDA=ON
-DGGML_CUDA_GRAPHS=ON
-DCMAKE_CUDA_ARCHITECTURES=121

Run command
/llm/llama.cpp/build/bin/llama-server
-m /llm/llm_models/Qwen_Qwen3.5-122B-A10B-Q5_K_S-00001-of-00003.gguf
–mmproj /llm/llm_models/bartowski-mmproj-Qwen_Qwen3.5-122B-A10B-bf16.gguf
–flash-attn on --no-mmap
-c 163840 --fit on
-ngl 999 --threads 18
–cache-type-k q8_0 --cache-type-v q8_0
–kv-unified
–temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00 --presence-penalty 0.0 --repeat-penalty 1.0
–host 0.0.0.0 --port 5907
–jinja --no-webui

Speed is good. ~20 t/s as I mentioned above. But the memory usage just keeps going up after each chat, even after I closed down and deleted chats. The front end is OpenWebUI on another computer

11

not sure if this flag will make a difference but worth trying

 -DCMAKE_CUDA_ARCHITECTURES=121a-real
12

here is my config for the 122b. i think i haven’t run this recently so it might be a bit different now.

“-np 1” might be key here. ggerganov recommended i try that to fix a slowdown over time with glm 4.7 flash and i’ve been adding it to all my configs since. to my understanding it disables a kv cache that doesn’t work well. Note that I believe this is the opposite of your -kv-unified flag (which is the default i believe)

I feel like its always a struggle keeping up with what is the best current config with llama.cpp as the code is changing rapidly and the documentation is sparse.. I’m guessing thats the case for most of the inference options though.

HF_MODEL="unsloth/Qwen3.5-122B-A10B-GGUF:UD-Q4_K_XL"

#-----------------------------------------

./llama.cpp/build-cuda/bin/llama-server \
  -hf "$HF_MODEL" \
  --ctx-size 0 \
  --temp 0.6 \
  --top-p 0.8 \
  --top-k 20 \
  --min-p 0.01 \
  --presence-penalty 1.5 \
  --repeat-penalty 1.0 \
  --batch-size 4096 \
  --ubatch-size 4096 \
  -np 1 \
  --no-mmap \
  --metrics \
  --host 0.0.0.0 \
  --port 8080