@Julien D— Drop the MIC. This is one of the most comprehensive and practical posts on the DGX Spark forums. Huge thanks for sharing the repo — the on‑demand loading + idle shutdown via gateway/waker is elegant, memory‑smart, and feels production‑ready
Quick frontend question
Are you using Open WebUI and letting users pick models via the model field in the payload, or do you have a custom client / proxy in front of the gateway? I’d love to understand how model selection flows through to the end‑user experience — especially how you expose multiple backends behind a single OpenAI‑compatible endpoint.
I’m very interested in putting together stacks on DGX Spark for STEM research and coding, and I’d like to use my Spark to beat cloud LLMs on both latency and cost for my workloads.
vLLM sleep / wake on Spark
I’ve started assembling stacks and digging into vLLM sleep/wake (L1/L2) on the Spark’s 128 GB unified memory pool. The docs say L1 frees more than 90% of GPU memory with wake times 18–200× faster than cold starts, but that’s mostly discrete‑GPU data. I’m planning to test on Spark soon with pre/post nvidia-smi and torch.cuda.mem_get_info() snapshots to see how it behaves under UMA.
The memory‑usage utility you shared looks perfect — especially if I can get a CSV with actual footprints. That would be a game‑changer for planning realistic multi‑model combos and understanding how much we can truly fit into a single Spark under different KV cache and sleep policies.
Here’s an updated (early) table of combos I’m exploring — not the full set yet:
| Combo ID | Models (core stack) | Active headroom | Best reranker | Added mem (active) | Sleep impact (L1/L2 est.) | Why it fits this stack |
|---|---|---|---|---|---|---|
| A | gpt‑oss‑120B + Qwen3‑Coder‑30B + GLM‑4.7‑Flash | 8–13 GB | Qwen3‑Reranker‑8B FP8 | ~8 GB | ~30–35 GB → <5 GB | Qwen synergy + strong code/math reranking before the heavy model. |
| B | Nemotron‑3‑Nano‑30B‑A3B + Qwen3‑Coder + GLM‑4.7‑Flash | 13–18 GB | Qwen3‑Reranker‑8B FP8 | ~8 GB | ~7–10 GB → <2 GB | 1M context + Qwen reranker = long‑doc RAG; still leaves room for a VL model. |
| C | Pure Qwen3 family | 18–23 GB | Qwen3‑Reranker‑8B FP8 | ~8 GB | ~8–12 GB → <2 GB | Seamless family instructions + top‑tier code/math; can add a 0.6B embed pair. |
| D | gpt‑oss‑120B + GLM‑4.7‑Flash + Llama‑3.1‑8B | 10–18 GB | Qwen3‑Reranker‑8B FP8 | ~8 GB | ~30–35 GB → <5 GB | gpt‑oss for finance/HFT; reranker cleans chunks before the expensive model. |
Your waker already nails fast switching. Sleep mode might let me keep 2–3 extra models warm, but your container‑per‑model approach may still be more robust for production isolation and failure modes.
Why I’m excited about Nemotron‑3‑Nano and MoE
Newer MoE models like Nemotron‑3‑Nano‑30B‑A3B are changing the game: with only a few billion active parameters, they deliver reasoning, coding, agentic behavior, and long‑context performance that rivals or beats much larger dense predecessors. That’s what makes me optimistic about using vLLM sleep/wake plus MoE to realistically deploy three to five models on a single Spark without brutal tradeoffs.
Model size is just one factor, though. I’m really focused on:
- Prompt engineering tuned for STEM plus web/RAG.
- Local RAG with Qdrant.
- Careful temperature and decoding tuning.
- Citation control and grounded answers.
- High‑quality context embeddings and strong reranking.
It’s not the “sexy” part of the stack, but doing this well gets much better results out of whatever base model you’re running.
Request: stats / memory footprints
I saw in your repo TODO that you’re planning to explore more dynamic model combos and are already collecting GPU usage data in stats/. Any chance you could share the actual measured memory footprints (or CSV/output) for the models you’ve run on DGX Spark — including active usage and KV cache if you have it? Even approximate numbers per model plus typical batch size and max tokens would be incredibly useful for designing realistic multi‑model stacks on Spark’s unified memory.
That kind of data would make a huge difference for people trying to do what you’re doing in production, and I’d love to contribute back with my own Spark sleep/wake measurements once I’ve run them.
Massive thanks again — your work is moving the ecosystem forward. I’m excited to follow your TODOs and hopefully contribute (especially around sleep‑mode strategies and multi‑model orchestration).
Legendary. 🚀