# Building Local + Hybrid LLMs on DGX Spark That Outperform Top Cloud Models

**URL:** <https://forums.developer.nvidia.com/t/building-local-hybrid-llms-on-dgx-spark-that-outperform-top-cloud-models/359569>\
**Category:** DGX Spark / GB10 Projects\
**Tags:** llama3-70b-instruct, deepseek, nemotron, llama, nim, jetson\
**Created:** [February 3, 2026, 6:54pm UTC](https://forums.developer.nvidia.com/t/building-local-hybrid-llms-on-dgx-spark-that-outperform-top-cloud-models/359569 "2026-02-03T18:54:02Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![griffith.mark](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/griffith.mark/32/453455_2.png) [@griffith.mark](https://forums.developer.nvidia.com/u/griffith.mark)\
**Post date:** [February 3, 2026, 6:54pm UTC](https://forums.developer.nvidia.com/t/building-local-hybrid-llms-on-dgx-spark-that-outperform-top-cloud-models/359569/1 "2026-02-03T18:54:03Z")

</div>

## **From “Can It Run?” to Sovereign Hybrid RAG on DGX Spark – Qwen3‑80B + vLLM + LiteLLM (Live Stack)**

* * *

## 1. Why I’m Sharing This Stack

My DGX Spark is now running a full hybrid RAG stack with Qwen3‑80B at roughly 45 tok/s, under 150 ms end‑to‑end latency, and zero cloud on the critical path—and it already feels stronger than cloud assistants on privacy, cost, and customization. Most DGX Spark threads focus on “how do I get the model to load without OOM?”; I’m now at the point where the stack is stable, and I’m trying to turn it into a practical, privacy‑first assistant. This is my first serious attempt at a vLLM‑based hybrid RAG stack on DGX Spark, so I’m very open to suggestions, corrections, and better patterns from everyone here (including NVIDIA folks).

Right now I’m running Qwen3‑Next‑80B‑A3B at around 45 tok/s, with 115–120 GB of unified memory in use and roughly 95 % GPU utilization; Qwen3‑Embedding‑0.6B and Qwen3‑Reranker‑0.6B add under 1.5 GB VRAM overhead. My goal is to see how far I can go with a local + hybrid approach before I really need to lean on cloud LLMs. I don’t have any experience with LangChain or LangGraph yet—right now I’m just using LiteLLM as a simple router with per‑model settings—so any starter patterns or anti‑patterns for Mixture‑of‑Models orchestration would be very welcome.

### 1.1 Cloud vs Local Summary

| Dimension | Cloud Providers | My Local + Hybrid Stack |
| --- | --- | --- |
| Cost | Recurring per‑token spend (expensive for deep research or code) | 0.00 USD for ~75 % of queries after implementation |
| Latency | 400–2000 ms (network + queue) | \<150-300 ms end‑to‑end on typical queries |
| Privacy / sovereignty | Prompts and documents leave the machine; compliance is hard | Zero egress (fallbacks only on explicit failure) |
| Customization | Fixed models, black‑box routing | Full control of temperature, routing depth, RAG limits, corpus composition |
| Private data | Upload limits, no persistent memory | Unlimited Qdrant corpus (my PDFs, DOCX, codebases, internal notes) |
| RAG and corpus quality | Generic web only | Targeted academic/technical expansion, multi‑hop citation graph |
| Open‑book | Closed‑book benchmarks | Permanently open‑book with CUDA docs, Blackwell papers, key arXiv papers in Qdrant |
| Implementation‑grade answers | Often miss exact implementation details | Retrieves exact passages and generates implementation‑grade answers |

* * *

## 2. What I Currently Have Running

Everything sits behind `pipelines:9099/v1`, fronted by Open WebUI and orchestrated by `pipe-fixed.py` plus a few helper modules.

### 2.1 Live Containers and Roles

```plaintext
CONTAINER ID IMAGE STATUS PORTS
f220e0ce3b2c ghcr.io/berriai/litellm:main-latest Up 3 h 0.0.0.0:4000->4000/tcp
41f392f6ff70 pipeline-deps:latest Up 33 m 0.0.0.0:9099->9099/tcp
71d157f02224 ghcr.io/open-webui/open-webui:cuda Up 3 h (healthy)0.0.0.0:8080->8080/tcp
ea2f7071c12a qdrant/qdrant:v1.11.0 Up 3 h 0.0.0.0:6333->6333/tcp
cdd82e1d2238 searxng/searxng:latest Up 21 h 0.0.0.0:8888->8080/tcp
5466fb74b97f caddy:2-alpine Up 21 h
d801572ab84d valkey/valkey:8-alpine Up 21 h 6379/tcp
0423861c8044 nvcr.io/nvidia/vllm:25.12.post1-py3 Up 21 h 0.0.0.0:8025->8000/tcp
bde1af729d3a nvcr.io/nvidia/vllm:25.12.post1-py3 Up 21 h 0.0.0.0:8020->8000/tcp
5f4c61cb0d7a nvcr.io/nvidia/vllm:25.12.post1-py3 Up 2 d 0.0.0.0:8005->8000/tcp

```

What each one does (today):[1]

- `openwebui` – User‑facing interface (chat, uploads, model selection).
- `pipeline-deps` (`pipelines`) – Exposes `pipelines:9099/v1` and runs my Hybrid‑RAG inlet and routing logic (`pipe-fixed.py` + helpers).
- `litellm` – Central router for all models (local and hosted), handles temperature, `top_p`, `max_tokens`, and fallbacks.
- `qdrant` – Local vector store for static documents, notes, and code.
- `searxng` – Meta search engine used by `web.py` for live web search (JSON).
- `embedding` – vLLM service running Qwen3‑Embedding‑0.6B for dense embeddings.
- `reranker` – vLLM service running Qwen3‑Reranker‑0.6B for cross‑encoder reranking.
- `vllm-qwen-80` – vLLM service running Qwen3‑Next‑80B‑A3B for main generation.
- `valkey` – Valkey/Redis used for caching and coordination.
- `caddy` – Simple reverse proxy and TLS frontend.

Resources: memory sits around 115–120 GB out of 128 GB unified; GPU utilization is about 95 %. Embedding + reranker add under 1.5 GB VRAM on top of the 80B model.[1]

* * *

## 3. Routing and RAG Profiles

Open WebUI sends chat requests to `pipelines:9099/v1`, which calls `pipe-fixed.py`.[1]

- `pipe-fixed.py` extracts the user message, applies `MODEL_CLASSIFICATION_RULES`, and picks a RAG profile from `MODE_CONFIG`.[1]
- These limits are applied dynamically based on the selected model name in Open WebUI.
- The profile controls how deep the system goes into web, Qdrant, and academic retrieval and how many chunks are reranked.[1]

### 3.1 MODE\_CONFIG (Simplified)

| Profile | weblimit | qdrantlimit | academiclimit | reranktop |
| --- | --- | --- | --- | --- |
| Fast | 5 | 10 | 5 | 10 |
| Auto (Standard) | 10 | 15 | 8 | 15 |
| Expert | 100 | 25 | 15 | 25 |
| Heavy / Moe | 250 | 30 | 20 | 30 |
| Think | 75 | 20 | 12 | 20 |
| Code‑Fast | 5 | 10 | 5 | 10 |
| Code‑Expert | 100 | 25 | 15 | 25 |
| Code‑Heavy | 250 | 30 | 20 | 30 |

Fast and Auto act as “light RAG”; Expert, Heavy, Moe, and Code‑Heavy go to full 250 web + 30 Qdrant + 20 academic depth.[1]

(Internally this is driven by simple rules such as “if model name matches `Code-.*` → use Code‑Heavy profile”, but I’ve kept the code out of the post for brevity.)

### 3.2 Parallel Retrieval

For each query, `pipe-fixed.py` launches three asynchronous retrievals:[1]

- `web.py` → SearXNG at `http://searxng:8888` (`format=json`, category “it” for code, “general” otherwise).
- `qdrant.py` → local Qdrant, using `embed_text()` via Qwen3‑Embedding‑0.6B at `http://embedding:8025`.
- `academic.py` → arXiv, PubMed, CORE, Semantic Scholar, OpenAlex (API keys, rate‑limits, multi‑hop expansion).

All results are merged into a single list of chunks for reranking.[1]

### 3.3 Reranking and Context Injection

- `reranker.py` sends the query + merged chunks to Qwen3‑Reranker‑0.6B at `http://reranker:8020`.[1]
- The reranker returns a list sorted by relevance score; I keep the top‑k (10–30, controlled by `reranktop`) and inject those chunks into the LLM prompt context.[1]

### 3.4 Generation and Fallbacks

- LiteLLM at `http://litellm:4000/v1` routes the request to the chosen model with the appropriate temperature and `max_tokens`.[2][1]
- The primary path is Qwen3‑Next‑80B‑A3B on vLLM (port 8005), which produces grounded answers with citations.[1]
- Responses go back to Open WebUI, including source references.[1]

Fallbacks to cloud models are reserved for exceptional cases—for example, timeouts, `max_tokens` limits, or explicit “use‑cloud” profiles—and are designed to keep cloud usage to a small fraction of total queries.

* * *

## 4. Model Routing, Temperatures, and Hosted MoM Options

I treat `MODEL_CLASSIFICATION_RULES`, `MODE_CONFIG`, and `litellm_config.yaml` as a simple dispatcher for models and RAG profiles.

### 4.1 Local SuperQwen Variants

From `litellm_config.yaml`, my local SuperQwen profiles (served via `http://vllm-qwen-80:8000/v1`) look roughly like this:

- Auto / Fast / Expert / Heavy / Moe → all backed by `openai/Superqwen`, with temperatures in the 0.3–0.6 range and `max_tokens` from 512 up to 8192.
- Code‑Fast / Code‑Expert / Code‑Heavy → same backend with lower temperatures (0.2–0.3) and larger `max_tokens` for code‑heavy tasks.

Current temperature by task:[2][1]

| Profile family | Typical temperature | Max tokens (approx) | Thinking |
| --- | --- | --- | --- |
| Code‑\* | 0.20–0.30 | 2 048–8 192 | On for Expert/Heavy |
| Research / Expert | 0.45 | 6 144 | On |
| Heavy / Think / Moe | 0.50 | up to 8 192 | On |
| Fast | 0.30 | 512–2 048 | Off |

### 4.2 Hosted Models I’m Experimenting With

Through LiteLLM I also have a universe of hosted models configured, mainly for future MoM experiments:[2]

- xAI / Grok: Grok‑4, Grok‑4 Fast Reasoning, Grok‑4.1 Fast Reasoning, Code‑Grok4, Code‑Grok4‑Fast.
- Perplexity: Perplexity Sonar, Sonar Pro, Sonar Reasoning Pro, Sonar Deep Research (`return_citations: true`).
- NVIDIA hosted: `nvidia_nim/meta/llama3-70b-instruct`, `nvidia_nim/qwen/qwen3-next-80b-a3b-thinking`, GLM‑4.7‑style code models, Nemotron‑3‑Nano‑30B‑A3B (NVFP4).
- Hugging Face router: Qwen2.5‑72B‑Instruct via `router.huggingface.co`.
- DashScope (Qwen family): Qwen‑Turbo, Qwen‑Plus, Qwen‑Max, Qwen‑VL, and Qwen3 models such as Qwen3‑32B, Qwen3‑30B‑A3B, Qwen3‑235B‑A22B with `enable_thinking` flags.[2]

Right now I don’t have LangChain or LangGraph on top of this; it’s just LiteLLM plus routing rules. I’m especially interested in providers that expose temperature, `top_p`, `max_tokens`, and “thinking” flags cleanly so they can slot into a Mixture‑of‑Models graph later.[2][1]

I’m also very interested in memory usage and real‑world performance of hybrid MoM setups with Nemotron‑3‑Nano‑30B‑A3B (NVFP4), GLM‑4.x, and strong GPT‑class OSS models under a 120 GB unified‑memory budget. My hunch is that a single Spark can be very competitive locally, and that a 2–6 node Spark cluster would be exceptional for MoM, but I’d really like to see real numbers from people who have tried it.[1]

* * *

## 5. Why I Chose a Qwen3‑Aligned RAG Stack

Instead of the default Open WebUI RAG (MiniLM/SBERT + cosine), I switched to a fully Qwen3‑aligned stack: Qwen3‑Embedding‑0.6B + Qwen3‑Reranker‑0.6B + Qwen3‑Next‑80B‑A3B.[1]

### 5.1 RAG Stack Comparison

| Feature | Open WebUI Default RAG | My Qwen3 Stack (Embedding + Reranker + LLM) |
| --- | --- | --- |
| Embedding model | Generic (all‑MiniLM, Sentence‑BERT) | Qwen3‑Embedding‑0.6B – multilingual, domain‑strong, family‑aligned |
| Retrieval method | Dense cosine similarity only | Dense + cross‑encoder reranking (Qwen3‑Reranker‑0.6B) |
| Relevance lift | Baseline | Roughly 15–30 % better on technical/multilingual queries in practice |
| Model coherence | Mixed models → inconsistency | Same Qwen3 family → embedding/reranker/LLM synergy |
| Latency / memory | Higher overhead if using cloud embeddings | \<1.5 GB VRAM total for embed + reranker, sub‑100 ms local RAG |
| Customization | Limited | Full vLLM tuning, LoRA fine‑tuning possible |
| Accuracy on my data | Good for general text | Superior after domain‑specific corpus + tuning |

My working assumption is that keeping everything in the same model family reduces semantic drift between embedding, reranking, and generation; I’d love to see counter‑examples or benchmarks (e.g., BGE‑M3 vs Qwen3‑Reranker on DGX Spark) from others.

* * *

## 6. Advantages Table (Cloud vs DGX Spark Hybrid)

These rows are meant as a qualitative comparison, not precise pricing.

### 6.1 Cloud LLMs vs My DGX Spark Hybrid

| Feature | OpenAI GPT‑4‑class | Anthropic Claude‑class | Google Gemini‑class | Perplexity Sonar Pro | xAI Grok‑4 Heavy | My DGX Spark Hybrid |
| --- | --- | --- | --- | --- | --- | --- |
| Cost | Per‑token pricing | Per‑token pricing | Per‑token pricing | Per‑token or Pro tier | Roughly fixed monthly | 0.00 USD per query after hardware |
| Latency | About 800–2000 ms | About 600–1500 ms | About 500–1200 ms | About 400–900 ms | About 300–800 ms | \<500 ms end‑to‑end |
| Data privacy | Prompts go to OpenAI | Prompts go to Anthropic | Prompts go to Google | Prompts go to Perplexity | Prompts go to xAI | Zero egress by default |
| Private docs | File caps, limited memory | Limited | Varies, often small caps | No persistent private KB | None | Unlimited Qdrant corpus |
| Academic live search | No | No | Limited | No | No | arXiv, PubMed, CORE, Semantic Scholar, etc. via `academic.py` |
| Reranking | None / proprietary | None / proprietary | None / proprietary | Proprietary | None | Qwen3‑Reranker‑0.6B two‑stage RAG |
| Temperature / routing | Mostly fixed | Limited | Limited | Black‑box | Limited | Full per‑profile control |
| Rate limits | Hard limits | Hard limits | Hard / RPM caps | Hard limits | Hard limits | None beyond my hardware |

Cloud limitations in one sentence each (from my perspective):

- OpenAI / Anthropic / Google: I pay for every token, my data leaves the box, private files are constrained, and I don’t get full per‑task temperature and routing control.
- Perplexity: dynamic RAG and academic routing are excellent, but I’m locked into their models, pricing, and black‑box decisions.
- Grok‑4 Heavy: a flat monthly fee for “uncensored” reasoning that I can get close to, or match, on my own hardware for near‑zero marginal cost.

Perplexity’s “secret sauce” is dynamic RAG + intelligent model routing + academic search; my goal with this stack is to do the same thing locally, but with my own reranker, my own corpus, and my own fallback logic.

* * *

## 7. Corpus Building and Fine‑Tuning Plans

### 7.1 Network‑Effect Corpus Building

For domains like machine learning and AI, I’m trying a network‑effect corpus strategy:

1. Use `academic.py` to fetch around 20 key LLM/ML papers (arXiv, Semantic Scholar).
2. Expand by authors, co‑authors, references, and citations 5–10 hops out.
3. Chunk and embed everything with Qwen3‑Embedding‑0.6B and store it in Qdrant.

This already feels more like a domain “bible” than a generic web crawl.

### 7.2 Fine‑Tuning Qwen3‑Embedding‑0.6B (Planned)

I haven’t fine‑tuned yet; my plan is:

- Build triplets (query, positive\_chunk, negative\_chunk) from Qdrant and historical queries.
- Add domain prompts such as “Represent this GPU programming document for semantic search…”.
- Use LoRA with SWIFT/DeepSpeed or SentenceTransformers for contrastive training.
- Evaluate with MRR@10, NDCG, and RAGAS, then deploy by updating `start-embedder-reranker.yaml`.

Any practical advice on hyperparameters, batch sizes, or DGX‑specific pitfalls would be very welcome.

* * *

## 8. Prompt Flow Diagram

This diagram matches the current routing and RAG flow. Click Diagram to Enlarge:

 ![](https://global.discourse-cdn.com/nvidia/original/4X/2/0/f/20fc8f504f4c577bd89b74bc7ad0b183beb1141e.jpeg)

Key advantage: the explicit reranking step eliminates noisy chunks that dense retrieval alone would include.

**9. Mixture of Models (MoM) and Open‑Book RAG – Where I Want to Go Next**

I’m not using LangChain or LangGraph yet; right now everything is LiteLLM plus my own routing logic. My next goal is to turn this into a true Mixture of Models graph, where local and hosted models work together instead of competing.[2][1]

### 9.1 Planned MoM Flow (LangChain + LangGraph)

The flow I’d like to build (feedback welcome):[1]

- A complex research query arrives → LiteLLM routes first to a local MoE‑style Qwen3‑80B profile for an initial reasoning pass.
- If the graph detects high uncertainty or a very demanding task, it spawns parallel branches:
  - Branch A: a “Think” model such as Qwen3‑Next‑80B‑thinking via NVIDIA NIM for deeper chain‑of‑thought.
  - Branch B: Perplexity Sonar Deep Research for fresh web citations.
  - Branch C: Code‑oriented models such as Code‑Grok4, GLM‑4.x, or Qwen3‑235B when there is heavy code or math.

- Outputs from these branches are merged, reranked, and synthesized back into a single answer by the strongest local model, using my own RAG context.

The idea is that \>80 % of the compute still happens locally, and cloud models act as optional specialists used only on the hardest 5-10% of queries and typically for only a few thousand tokens. With fallback rules based on timeout, `max_tokens`, and usage‑based routing, the cost should stay low even with those specialists in the loop.[1]

I’m still evaluating whether I really need full LangGraph for this or whether smarter LiteLLM‑only routing would be enough, so any real‑world experience with stateful MoM graphs on Spark hardware would be hugely helpful.

### 9.2 Open‑Book vs Closed‑Book

Most leaderboards test models in a closed‑book setting: no internet, no private docs, no external corpus. My target setup is deliberately open‑book plus live research:[1]

- Drop entire domain “bibles” into Qdrant: CUDA docs, PyTorch and Triton source, internal specs, and relevant arXiv papers.
- On each query, retrieve the exact passages that matter and let the LLM reason with the book open.

For example, once the corpus is in place, I want to be able to ask:

> “How do I implement a custom FlashAttention kernel for Blackwell that supports NVFP4?”

Closed‑book models, even very strong ones, may hallucinate or provide outdated guidance. My goal is for the local stack to pull kernel signatures, relevant sections of the Blackwell whitepaper, correct Triton examples, and the latest FlashAttention‑3 paper, then produce implementation‑ready code.[1]

This is why I see local RAG and MoM as the next step beyond cloud‑only usage: once the corpus is right, general benchmarks become less important than actual performance on my own data and tasks.

* * *

## 10. Alternative Stacks for Other Workloads

A full hybrid RAG + reranker + 80B stack is not the ideal fit for every workload. These are other configurations I think make sense:

| Workload | Recommended stack | Why it might be better than my full hybrid |
| --- | --- | --- |
| Pure fast chat / code | Ollama + DeepSeek‑R1 or Qwen2.5‑32B | Lower latency, simpler, no reranker overhead. |
| Vision / multimodal | Local Qwen‑VL‑Max or LLaVA‑Next | Native image and multimodal understanding. |
| Heavy fine‑tuning | NeMo on 1–2 Sparks | Optimized for distributed training and experimentation. |
| Edge / low‑power | Jetson Orin + small GGUF models | Battery‑friendly, deploy at the edge. |
| Zero‑GPU setups | `llama.cpp` on Mac M‑series | No NVIDIA hardware required. |
| Multi‑node scaling | 2–6 Spark cluster + tensor‑parallel vLLM | Near‑linear throughput; 235B‑class models become realistic. |

My hope is that DGX Spark can be the “sovereign core” for heavy RAG + MoM, while lighter stacks handle edge and low‑power scenarios.

* * *

## 11. The Levers I’m Trying to Tune

 ![levers](https://global.discourse-cdn.com/nvidia/original/4X/a/7/3/a731975bb632897c532922a94cc0075c8890d53a.jpeg)

I’m still new to LangGraph/LangChain and to designing serious Mixture‑of‑Models workflows, so any feedback—from high‑level architecture to small config tweaks—would be genuinely appreciated.

I’d especially love input on:

- MoM orchestration patterns (starter LangGraph flows or alternatives).
- Embedding fine‑tuning pitfalls on Spark (for Qwen3‑Embedding‑0.6B or similar).
- Real‑world multi‑Spark scaling numbers (2–6 nodes, tensor parallel, 235B‑class models).
- Anything in this stack that looks obviously wrong, fragile, or over‑engineered.

## 12. Playbooks

I’m planning to publish a five‑part playbook series on this vLLM stack as it stabilizes. Web‑search has been stable for about two weeks, and I’ll be publishing that playbook this week.

| # | Playbook Title | Core Content |
| --- | --- | --- |
| 1 | **Open WebUI + SearXNG (Private Web Search)** | Enable private web search via SearXNG, enforce JSON output, surface citations directly in the UI. |
| 2 | **LiteLLM Smart Routing & Dynamic Temperature** | Map model names to RAG profiles, set per‑task temperatures, graceful fallbacks to cloud models only on failure. |
| 3 | **Static RAG with Qdrant (Local Document Indexing)** | Ingest 50+ PDFs/DOCX, auto‑chunk, embed with Qwen3‑Embedding‑0.6B, store in Qdrant, expose via hybrid retrieval. |
| 4 | **Dynamic RAG – Academic Sources (arXiv, PubMed, CORE, etc.)** | Build `academic.py` with API keys, rate‑limiting, multi‑hop expansion to create a deep, high‑quality academic corpus. |
| 5 | **Qwen3 Embedding + Reranker (Local RAG Engine)** | Deploy Qwen3‑Embedding‑0.6B and Qwen3‑Reranker‑0.6B on vLLM, benchmark vs BGE‑M3, optionally fine‑tune on your own corpus (LoRA, SWIFT, DeepSpeed). |

Good Luck developing your own Production LLM Stack.

Mark

---

<div class="post-metadata">

**Author:** ![aniculescu](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aniculescu/32/382649_2.png) [@aniculescu](https://forums.developer.nvidia.com/u/aniculescu)\
**Post date:** [February 3, 2026, 7:41pm UTC](https://forums.developer.nvidia.com/t/building-local-hybrid-llms-on-dgx-spark-that-outperform-top-cloud-models/359569/2 "2026-02-03T19:41:11Z")

</div>

Thanks for sharing this, I will move this to GB10 projects

---

<div class="post-metadata">

**Author:** ![eugr](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/eugr/32/449615_2.png) [@eugr](https://forums.developer.nvidia.com/u/eugr)\
**Post date:** [February 3, 2026, 7:58pm UTC](https://forums.developer.nvidia.com/t/building-local-hybrid-llms-on-dgx-spark-that-outperform-top-cloud-models/359569/3 "2026-02-03T19:58:10Z")

</div>

This seems to be a nice setup.

As for the agentic framework, my suggestion would be to go with [PydanticAI](https://ai.pydantic.dev/) instead of LangChain/LangGraph. It’s much easier to debug and is not overly bloated and over-engineered.

---

<div class="post-metadata">

**Author:** ![griffith.mark](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/griffith.mark/32/453455_2.png) [@griffith.mark](https://forums.developer.nvidia.com/u/griffith.mark)\
**Post date:** [February 3, 2026, 9:05pm UTC](https://forums.developer.nvidia.com/t/building-local-hybrid-llms-on-dgx-spark-that-outperform-top-cloud-models/359569/4 "2026-02-03T21:05:50Z")

</div>

> [@Playbook #1 – Open WebUI + SearXNG (Private Web Search) on DGX Spark](https://forums.developer.nvidia.com/t/playbook-1-open-webui-searxng-private-web-search-on-dgx-spark/359578):
>
> This is the first playbook in my series from the thread: It focuses on one thing only: Wiring SearXNG into Open WebUI on DGX Spark for private, local web search, with JSON output, Redis caching, Caddy in front, and clean integration with LiteLLM + vLLM. The goal is a reproducible, production‑style setup that any DGX Spark dev can drop in: SearXNG + Redis on an isolated searxng-net. vLLM + LiteLLM + Open WebUI on llm-net. Open WebUI talks to LiteLLM for models and to SearXNG only via it…

Here is the first playbook in the series. Hopefully everything is correct, it is 3-4 weeks since I coded this.

---

<div class="post-metadata">

**Author:** ![Keyper-AI](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/keyper-ai/32/465531_2.png) [@Keyper-AI](https://forums.developer.nvidia.com/u/Keyper-AI)\
**Post date:** [February 4, 2026, 3:00pm UTC](https://forums.developer.nvidia.com/t/building-local-hybrid-llms-on-dgx-spark-that-outperform-top-cloud-models/359569/5 "2026-02-04T15:00:08Z")

</div>

Great write up. I am in a very similar boat. My end goal is to completely get away from chatGPT or any cloud models and have everything local.

I haven’t had much luck with litellm integrated with Open WebUI it seems to bug out quite often. I am curious though how your smart routing and dynamic temperatures works with it. Do you have more details on that?

---

<div class="post-metadata">

**Author:** ![griffith.mark](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/griffith.mark/32/453455_2.png) [@griffith.mark](https://forums.developer.nvidia.com/u/griffith.mark)\
**Post date:** [February 19, 2026, 5:34am UTC](https://forums.developer.nvidia.com/t/building-local-hybrid-llms-on-dgx-spark-that-outperform-top-cloud-models/359569/7 "2026-02-19T05:34:01Z")

</div>

## Playbook 2 – Turning Spark Arena Models into a Unified Multi‑Mode Assistant with LiteLLM + Open WebUI

* * *

Since my original post, things have evolved a lot. Spark Arena and the GB10 community have gone from “here are some cool models” to a full **recipe + leaderboard ecosystem** that makes it trivial to stand up Nemotron‑3‑Nano NVFP4, GLM‑4.7‑Flash‑AWQ, Qwen3‑Coder‑Next‑FP8, Qwen‑Instruct‑80B and more on a single or dual DGX Spark. The net effect is a **quantum leap in developer velocity** : instead of burning hours on flags and container wiring, we can treat these as reliable building blocks and focus on UX, routing, and product behavior.

Huge thanks to Raphael Amorim (@raphael.amorim), @eugr, and everyone working on spark‑vllm‑docker, llama‑benchy, and Spark Arena itself — Playbook 2 is entirely about **what you can build on top of their work** , not yet another way to launch a model.

What follows is how, on a **single Spark** (usually one primary Spark Arena model up at a time), you can use **LiteLLM + Open WebUI** to turn that into a clean, multi‑mode assistant stack with natural model names like `NeMo Expert`, `GLM Expert`, `Qwen Coder Fast`, `Phi Fast`, `Grok-4`, etc. LiteLLM resolves everything; Open WebUI just shows these names in the dropdown.

* * *

## 1. Architecture: Open WebUI → LiteLLM → Spark Arena + hosted

The architecture is intentionally simple:

- **Open WebUI**

- **LiteLLM proxy/router**

- **Spark Arena vLLM backends + hosted APIs**

Open WebUI only sees **model names** from LiteLLM – e.g., `NeMo Expert`, `GLM Expert`, `Qwen Coder Fast`, `Phi Fast`, `Grok-4` – and LiteLLM resolves them to whichever backend is actually running.

* * *

## 2. LiteLLM model\_list: real model names, real modes

Your recipes from Spark Arena already defines the modes you need, using concrete model names.

### Local Nemotron‑Nano profiles

```yaml
# ── LOCAL NEMOTRON NANO (optimized for speed) ──
- model_name: "Base" # Native (thinking ON)
  litellm_params:
    model: openai/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4
    api_base: http://host.docker.internal:8000/v1
    api_key: os.environ/NEMOTRON_API_KEY
    temperature: 0.4
    top_p: 0.9
    max_tokens: 8192
    extra_body: {}
    chat_template_kwargs:
      enable_thinking: true
- model_name: "Fast"
  litellm_params:
    model: openai/microsoft/Phi-4-mini-instruct
    api_base: http://phi-judge:8000/v1
    api_key: simple-api-key
    temperature: 0.2
    top_p: 0.9
    max_tokens: 8192
    timeout: 30
    initial_messages:
      - role: system
        content: |
          You are a fast, local judge / evaluator / classifier.
    extra_body: {}
- model_name: "Expert" # Balanced reasoning
  litellm_params:
    model: openai/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4
    api_base: http://host.docker.internal:8000/v1
    api_key: os.environ/NEMOTRON_API_KEY
    temperature: 0.6
    top_p: 0.98
    max_tokens: 16384
    timeout: 300
    extra_body: {}
    chat_template_kwargs:
      enable_thinking: true
      effort: "medium"
- model_name: "Heavy"
  litellm_params:
    model: openai/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4
    api_base: http://host.docker.internal:8000/v1
    api_key: os.environ/NEMOTRON_API_KEY
    temperature: 1.0
    top_p: 1.0
    max_tokens: 32768
    timeout: 600
    stream_timeout: 240
    extra_body: {}
    chat_template_kwargs:
      enable_thinking: true
- model_name: "Code"
  litellm_params:
    model: openai/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4
    api_base: http://host.docker.internal:8000/v1
    api_key: os.environ/NEMOTRON_API_KEY
    temperature: 0.2
    top_p: 0.95
    max_tokens: 16384
    timeout: 300
    stream_timeout: 180
    extra_body: {}
    chat_template_kwargs:
      mode: code
      enable_thinking: true
      clear_thinking: false

```

### GPT‑OSS‑120B profiles

```yaml
# ===================GPT OSS 120B ==========================================
- model_name: "Fast"
  description: Ultra-fast classifier / router / moderator – short & strict
  litellm_params:
    model: openai/gpt-oss-120b
    api_base: http://host.docker.internal:8010/v1
    api_key: sk-1234
    temperature: 0.2
    top_p: 0.9
    max_tokens: 2048
    timeout: 12
    initial_messages:
      - role: system
        content: |
          You are a fast, local classifier/router/moderator.
- model_name: "Expert"
  description: Balanced reasoning – good default quality/speed trade-off
  litellm_params:
    model: openai/gpt-oss-120b
    api_base: http://host.docker.internal:8010/v1
    api_key: sk-1234
    temperature: 0.55
    top_p: 0.95
    max_tokens: 32768
    timeout: 400
    extra_body: {}
    chat_template_kwargs:
      enable_thinking: true
      effort: medium
- model_name: "Heavy"
  description: Maximum reasoning depth & longest allowed outputs
  litellm_params:
    model: openai/gpt-oss-120b
    api_base: http://host.docker.internal:8010/v1
    api_key: sk-1234
    temperature: 0.75
    top_p: 0.98
    max_tokens: 65536
    timeout: 1200
    stream_timeout: 600
    extra_body: {}
    chat_template_kwargs:
      enable_thinking: true
      effort: maximum
- model_name: "Code"
  description: Programming & technical tasks – prefers clean output
  litellm_params:
    model: openai/gpt-oss-120b
    api_base: http://host.docker.internal:8010/v1
    api_key: sk-1234
    temperature: 0.15
    top_p: 0.85
    max_tokens: 32768
    timeout: 600
    extra_body: {}
    chat_template_kwargs:
      mode: code
      enable_thinking: true
      clear_thinking: true

```

## 2.5. How “modes” map to temperature / length / effort

The different “modes” (Fast, Expert, Heavy, Code) are just **concrete per‑model profiles** with carefully chosen parameters in `litellm_config.yaml`:

- **Temperature**

- **max\_tokens / timeouts**

- **effort / thinking flags**

When you pick `Fastvs Expert` vs `Heavy` vs `Code`, you’re really choosing **a bundle of temperature + length + timeout + prompt discipline** that you’ve pre‑curated for that model’s sweet spot, from Hugging Face, UnSloth model cards etc.

* * *

1. **Routing and fallbacks (concrete names, Grok‑4 fallback)**

The final piece is the **routing\_strategy** and **router\_settings**. Using the same mode names `"Fast"` `"Expert"` for GL, Nemotron Nano etc with a simple, explicit fallback chain ending at `Grok-4` results a clean UI and model dropdown menu.

```yaml
# ── ROUTING & SETTINGS ───────────────────────────────────────────────────────
routing_strategy: usage-based-routing
router_settings:
  max_tokens_threshold: 4096
  fallback_on_max_tokens: true
  retry_on_failure: true
  num_retries: 2
  timeout: 60

# Corrected fallbacks – each fallback entry must list the model names that
# should be used when the primary model fails or when the token limit is hit.
fallbacks:
  - "Fast":
      fallback_to: ["Fast","Expert", "Heavy"]
  - "Expert":
      fallback_to: ["Expert","Heavy"]
  - "Heavy":
      fallback_to: ["Heavy","Grok-4"]

# Context‑window fallback mappings – ensure that when the primary model’s
# context window is exhausted, the system falls back to a model with a
# larger window.
context_window_fallbacks:
  "Fast": ["Expert", "Heavy"]
  "Expert": ["Heavy"]
  "Heavy": ["Heavy", "Grok-4"]

```

You keep **one version** of this block active at a time, depending on which Spark Arena recipe is your primary. Either way:

- Fast classifiers escalate into their corresponding Expert/Heavy modes and then `Grok-4`.
- The primary local assistant (`Expert` ) escalates to its Heavy profile and then `Grok-4`.
- Code‑oriented models fall back to Qwen‑Coder profiles and, if needed, `Grok-4`.

No generic labels; everything is grounded in real `model_name` strings from your config, with `Grok-4` as the single hosted backstop.

* * *

## 4. Phi Mini in the stack – the fast tier

`Phi Fast` runs Microsoft’s Phi‑4‑mini‑instruct as a dedicated **small, fast tier** , wired as its own vLLM instance on `phi-judge:8000`. Large quantized models like Nemotron‑3‑Nano NVFP4, GLM‑4.7‑Flash‑AWQ, and Qwen3‑Coder‑Next‑FP8 typically use \< 62% of DGX Spark’s VRAM in their Spark Arena recipes, which leaves comfortable headroom to keep Phi‑4‑mini loaded at the same time together with Qdrant, embedding model and reranking model etc.

This gives you:

- Sub‑second, ultra‑cheap responses for classification, routing, moderation, and quick Q&A.
- A local‑only Fast tier: `Fast` is always tried first for short/simple tasks, and only escalates to `Expert/Heavy/ Grok-4` when it hits timeouts, token limits, or context constraints via the routing rules above.

You get snappy, resilient behavior without complicating Open WebUI at all; it just exposes `Fast` as another model, and LiteLLM handles the routing.

* * *

## 5. What this gives you in practice

On a single DGX Spark:

- Spark Arena recipes give you vLLM deployments for Nemotron, GLM, Qwen‑Coder, Qwen‑Instruct.
- LiteLLM wraps those (plus small locals like Phi‑4‑mini and hosted models like Grok‑4, Qwen‑Max/Qwen3, Llama‑3‑70B) into a single OpenAI‑compatible endpoint with **per‑model routing and fallbacks** using real model names.
- Open WebUI talks only to LiteLLM and simply lists `Fast`, `Expert`, `Heavy`, `Code`, etc. — nothing abstract, no alias layer.

The result is a hosted‑grade, multi‑model assistant experience with simple, concrete model choices and robust failover, running entirely on your hardware, with Spark Arena doing the heavy lifting underneath.

**Auto** mode will be added at a future date to automatically route to these basic modes, MoE and compound cloud/local MoE modes.

@raphael.amorim, @eugr thank you for providing the community Spark Arena, recipes etc, it is great having a foundation to build on! There is still more work to do here integration of Modes and Auto Mode, lets think about developing a community front end for the model in Spark Arena Models to provide a complete solution for Spark Users. I do like the openWebUI front end it does provide a lot of flexibility for customization. I do really like Qdrant as a backend vector database for openWebUI. The backend with RAG and pipelines are very simple and work well once you work out how to code them but very hard to get going the first time when you are trying to configure everything. I will post more on RAG and building RAG corpus at a future date.

Many thanks,

Mark

---

<div class="post-metadata">

**Author:** ![kuntz.jamie](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/kuntz.jamie/32/187837_2.png) [@kuntz.jamie](https://forums.developer.nvidia.com/u/kuntz.jamie)\
**Post date:** [February 20, 2026, 2:00am UTC](https://forums.developer.nvidia.com/t/building-local-hybrid-llms-on-dgx-spark-that-outperform-top-cloud-models/359569/8 "2026-02-20T02:00:40Z")

</div>

Thank you So much for these detailed and extremely useful guides! You have greatly helped this newbie along the path to spark’s greatness. Much appreciated and many kudos.

---

<div class="post-metadata">

**Author:** ![raphael.amorim](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/raphael.amorim/32/446733_2.png) [@raphael.amorim](https://forums.developer.nvidia.com/u/raphael.amorim)\
**Post date:** [February 20, 2026, 4:24am UTC](https://forums.developer.nvidia.com/t/building-local-hybrid-llms-on-dgx-spark-that-outperform-top-cloud-models/359569/9 "2026-02-20T04:24:03Z")

</div>

Thanks for the kind words and your contributions Mark.  
It’s very rewarding see the community growing and building together with lots of creative ideas. There’s more to come, some other folks in the community are reaching out and join forces so we can build a strong, collaborative and coherent ecosystem we all benefit from without stress and burnout

---

<div class="post-metadata">

**Author:** ![eugr](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/eugr/32/449615_2.png) [@eugr](https://forums.developer.nvidia.com/u/eugr)\
**Post date:** [February 20, 2026, 9:52pm UTC](https://forums.developer.nvidia.com/t/building-local-hybrid-llms-on-dgx-spark-that-outperform-top-cloud-models/359569/10 "2026-02-20T21:52:17Z")

</div>

Nice! Now if you add llama-swap to this setup, then you’ll be able to launch/switch models on demand. I run a similar stack - OpenWebUI → LiteLLM → llama-swap → vLLM (via our community docker/recipes)/llama.cpp, although since I’m constantly testing new models on Spark, llama-swap is temporarily out of service.

I like the idea of “modes” and more sophisticated request routing.

I use fallbacks in my setup, but they are to the same model on a different machine when the primary is not available (and for some models, to OpenRouter/Anthropic/OpenAI).

---

<div class="post-metadata">

**Author:** ![griffith.mark](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/griffith.mark/32/453455_2.png) [@griffith.mark](https://forums.developer.nvidia.com/u/griffith.mark)\
**Post date:** [February 24, 2026, 11:51am UTC](https://forums.developer.nvidia.com/t/building-local-hybrid-llms-on-dgx-spark-that-outperform-top-cloud-models/359569/11 "2026-02-24T11:51:40Z")

</div>

**Optimizing Multi-Model Workflows in Open WebUI on DGX Spark for Enhanced Performance**

In developing AI pipelines, I have implemented sophisticated routing mechanisms, LiteLLM configurations, and fallback strategies. However, a subtle inefficiency persisted in my personal workflow when using Open WebUI with local models hosted on NVIDIA DGX Spark.

Previously, when evaluating responses from a local model, I would frequently compare them against outputs from external models such as Grok, Perplexity, or Claude. This required manually copying prompts, switching browser tabs, and executing queries separately—a process that disrupted focus and efficiency multiple times daily.

I have since discovered that Open WebUI natively addresses this through its multi-model capabilities, streamlining the comparison process.

### Key Feature: Multi-Model Selection

Open WebUI enables the selection of multiple models directly from the dropdown menu, supporting both local and hosted options. Look at the model selector and you will see a “+” sign (in the screenshot below, see the “+” next to the Fast mode model selected). Just click “+” and add a new model or new model mode. After you have done this, you will see multiple icons for each model.

 ![image](https://global.discourse-cdn.com/nvidia/original/4X/5/1/a/51a67130d1337c4155d2fb70f5ce8bfb9ce5308c.jpeg)

Upon submission, the prompt is dispatched in parallel to all selected models, with responses streaming back in real-time and displayed in adjacent panels or tabs for direct comparison.

 ![image](https://global.discourse-cdn.com/nvidia/original/4X/e/7/4/e74656c865672b43b76808c3b9241850c7f09f71.png)

 ![image](https://global.discourse-cdn.com/nvidia/original/4X/7/e/1/7e1cb53b9ed2e44efa6449b063befcf062a26d67.jpeg)

**Integrated Response Merging**

For tasks requiring a unified response rather than individual evaluations, Open WebUI now includes native merging functionality for multi-model executions. By enabling the merge option after selecting models, the system automatically synthesizes a cohesive output from the various responses, without the need for custom aggregation scripts.

This feature operates under the hood, delivering a single, integrated reply that combines the strengths of local model depth with cloud model versatility.

**How to See the Merged Response**

It is not obvious how you merge responses. After all models have finished their answers, a small icon (three dots connected by lines) appears on the right under the response. Just click this icon and you will get the merged response.

 ![image](https://global.discourse-cdn.com/nvidia/original/4X/e/d/c/edc68c9e362018b2f69b8f5f3a651987483d8acb.png)

 ![image](https://global.discourse-cdn.com/nvidia/original/4X/3/d/3/3d3a70e716b9715e97820934b374fa1cb00f5c42.png)

### Impact on Workflow

Formerly, I utilized Open WebUI primarily as an interface for local models, relying on separate tabs for cloud interactions. It now serves as a centralized platform, where DGX Spark-hosted models and external APIs like Grok, Perplexity, and Claude are accessible via simple checkboxes.

I select the appropriate model combination based on the task, activate merging as needed, and proceed efficiently. This approach minimizes context switching and manual integration efforts.

**If you are an Open WebUI user and have not used or were unaware of this function give me a like on this post.**

Big Shout out to the Open WebUI UI/UX developers on this function.

Happy Model Merging!

Mark

**Sources** :  
[1] Features | Open WebUI [⭐ Features | Open WebUI](https://docs.openwebui.com/features/)  
[2] Models | Open WebUI [Models | Open WebUI](https://docs.openwebui.com/features/ai-knowledge/models/)  
[3] Which model is used when using the ‘merge responses’ function? [Which model is used when using the 'merge responses' function? · open-webui/open-webui · Discussion #6640 · GitHub](https://github.com/open-webui/open-webui/discussions/6640)  
[4] Configure the model for merging responses · Issue #9490 - GitHub [Configure the model for merging responses · Issue #9490 · open-webui/open-webui · GitHub](https://github.com/open-webui/open-webui/issues/9490)  
[5] I’m the Sole Maintainer of Open WebUI — AMA! : r/OpenWebUI [Reddit - The heart of the internet](https://www.reddit.com/r/OpenWebUI/comments/1gjziqm/im_the_sole_maintainer_of_open_webui_ama/)  
[6] feat: Add Edit and Delete Buttons to Model Selection Dropdown #2869 [feat: Add Edit and Delete Buttons to Model Selection Dropdown · open-webui/open-webui · Discussion #2869 · GitHub](https://github.com/open-webui/open-webui/discussions/2869)  
[7] How to Use LLAVA Multimodal with OpenWebUI & GPT-4 to Analyze … [https://www.youtube.com/watch?v=yZkmolyV0Zk](https://www.youtube.com/watch?v=yZkmolyV0Zk)  
[8] default model settings across multiple models - ollama model file [Reddit - The heart of the internet](https://www.reddit.com/r/OpenWebUI/comments/1h13p6c/default_model_settings_across_multiple_models/)  
[9] Guide to extending OpenWebUI using Pipelines [Guide to extending OpenWebUI using Pipelines](https://zohaib.me/extending-openwebui-using-pipelines/)  
[10] Enhance Multi-Model Comparison View with Expandable Output … [feat: Enhance Multi-Model Comparison View with Expandable Output Cards and Horizontal Model Selection · Issue #13070 · open-webui/open-webui · GitHub](https://github.com/open-webui/open-webui/issues/13070)  
[11] Open-Webui Mixture of Agents part 2 - LinkedIn [Open-Webui Mixture of Agents part 2](https://www.linkedin.com/pulse/open-webui-mixture-agents-part-2-raymond-bernard-qacsc)  
[12] How To connect models to OpenWebUI [How To connect models to OpenWebUI - UbiOps Technical Documentation](https://ubiops.com/docs/howto/howto-connect-model-to-openwebui/)  
[13] Run Open Source Multimodal Models Locally Using Ollama - YouTube [https://www.youtube.com/watch?v=1VBwYM6\_xww](https://www.youtube.com/watch?v=1VBwYM6_xww)  
[14] feat: Optimize RAG Pipeline with Async, Parallelism, and … [feat: Optimize RAG Pipeline with Async, Parallelism, and Batching · open-webui/open-webui · Discussion #13966 · GitHub](https://github.com/open-webui/open-webui/discussions/13966)  
[15] Tools | Open WebUI [Tools | Open WebUI](https://docs.openwebui.com/features/extensibility/plugin/tools/)

---

<div class="post-metadata">

**Author:** ![sean.clearpixels](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@sean.clearpixels](https://forums.developer.nvidia.com/u/sean.clearpixels)\
**Post date:** [March 1, 2026, 1:15pm UTC](https://forums.developer.nvidia.com/t/building-local-hybrid-llms-on-dgx-spark-that-outperform-top-cloud-models/359569/12 "2026-03-01T13:15:35Z")

</div>

Nice setup. What do you use to measure the accuracy of your system? Any issues with hallucinations?

So far I’ve only had a play with Ragas.

---

<div class="post-metadata">

**Author:** ![griffith.mark](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/griffith.mark/32/453455_2.png) [@griffith.mark](https://forums.developer.nvidia.com/u/griffith.mark)\
**Post date:** [March 1, 2026, 2:29pm UTC](https://forums.developer.nvidia.com/t/building-local-hybrid-llms-on-dgx-spark-that-outperform-top-cloud-models/359569/13 "2026-03-01T14:29:14Z")

</div>

**see below: Edit Update Error**

---

<div class="post-metadata">

**Author:** ![griffith.mark](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/griffith.mark/32/453455_2.png) [@griffith.mark](https://forums.developer.nvidia.com/u/griffith.mark)\
**Post date:** [March 1, 2026, 2:45pm UTC](https://forums.developer.nvidia.com/t/building-local-hybrid-llms-on-dgx-spark-that-outperform-top-cloud-models/359569/14 "2026-03-01T14:45:51Z")

</div>

**Re: Hallucination testing — phased approach with academic grounding**

Thanks again for the Ragas pointer — hadn’t come across it before, so that’s now firmly on the list to dig into properly.

The initial focus has been on academic STEM research. I have self‑hosted the full arXiv dataset — approximately 3 million papers — split across 10 Qdrant collections by subject area, and are getting good retrieval results from a hybrid dense + sparse setup. On top of that I have built 15 Open WebUI tools to query external academic databases in parallel, covering Semantic Scholar, OpenAlex, CORE, Connected Papers, Europe PMC, PubMed, Perplexity Academic, Dimensions/Lens, ResearchGate/Academia, ArXiv Recent Alerts, AI2 Asta, Google Gemini Academic, a research LLM, and a consolidated academic paper and metadata fetcher. The intent is that any claim the model makes can be cross‑referenced against multiple authoritative sources simultaneously — for STEM research this gives you a verifiable ground truth that’s far more reliable than general web search for hallucination testing.

For the broader evaluation layer the approach is explicitly two‑phased. The primary judge layer is built around [free‑tier models on OpenRouter](https://openrouter.ai/models?order=pricing-low-to-high): there are some genuinely strong models available at zero cost, and the goal is to extract meaningful consistency and agreement signals at scale without touching paid APIs. Only if that proves insufficient for a given use case do we escalate to paid frontier models — Claude, DeepSeek, Grok, Perplexity — as high‑fidelity judges that can provide more nuanced grading and rationale.

Some of the Spark Arena local models running on the DGX Spark are also emerging as strong judge candidates in their own right. Nemotron Nano supports a native “thinking mode” that surfaces its full reasoning chain rather than just a verdict, which makes disagreements between judge and task model much more actionable. GLM‑4.7 Flash offers similar structured reasoning at a much lower GPU cost, making it a good lightweight local judge for higher‑volume checks. That means the judge layer can blend capable local models (with visible reasoning and zero API cost) and cloud models, giving multiple independent perspectives on the same answer.

LangChain handles the orchestration logic and routing between these judge tiers, and [LangFlow](https://www.langflow.org/)provides a visual GUI to iterate on and tune the evaluation pipelines — prompts, routing logic, thresholds, and escalation rules — without constantly rewriting code. [Checkout this excellent Nvidia Guide on using Langflow on the RTX](https://blogs.nvidia.com/blog/rtx-ai-garage-langflow-agents-remix/).

Beyond arXiv, I have already started extending Qdrant with grounded technical corpora: official documentation across Python, Docker, Debian, Linux, JavaScript, AppleScript, and Swift/Xcode stacks, alongside core textbooks and curated coding datasets from Hugging Face. The goal is that implementation and coding answers are evaluated against authoritative, versioned references rather than just whatever happens to be in the pretraining mix. If a model’s answer contradicts the official docs or a known benchmark dataset, that becomes a clear, automated signal rather than a subjective judgment call.

All of this is still very much under construction — there’s a fair bit of plumbing in progress to connect the retrieval, judging, and orchestration layers cleanly. Once it’s end‑to‑end tested and producing stable metrics, I’ll post a proper update, pipelines, tools, LangFlow etc with concrete results.

Mark

---

<div class="post-metadata">

**Author:** ![griffith.mark](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/griffith.mark/32/453455_2.png) [@griffith.mark](https://forums.developer.nvidia.com/u/griffith.mark)\
**Post date:** [March 1, 2026, 3:18pm UTC](https://forums.developer.nvidia.com/t/building-local-hybrid-llms-on-dgx-spark-that-outperform-top-cloud-models/359569/15 "2026-03-01T15:18:58Z")

</div>

> **[Wired for Action: Langflow Enables Local AI Agent Creation on NVIDIA RTX PCs](https://blogs.nvidia.com/blog/rtx-ai-garage-langflow-agents-remix/)**
>
> Simple drag-and-drop AI workflow designs are powered by open-source models and GPU acceleration, and feature support for NVIDIA RTX Remix and Project G-Assist.

---

<div class="post-metadata">

**Author:** ![Marsy](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/marsy/32/456092_2.png) [@Marsy](https://forums.developer.nvidia.com/u/Marsy)\
**Post date:** [March 1, 2026, 4:00pm UTC](https://forums.developer.nvidia.com/t/building-local-hybrid-llms-on-dgx-spark-that-outperform-top-cloud-models/359569/16 "2026-03-01T16:00:39Z")

</div>

Thank you so much for sharing!

Going to take me a few to go over moreover wanted to say thanks.

---

<div class="post-metadata">

**Author:** ![Balor](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/balor/32/482703_2.png) [@Balor](https://forums.developer.nvidia.com/u/Balor)\
**Post date:** [March 7, 2026, 10:32am UTC](https://forums.developer.nvidia.com/t/building-local-hybrid-llms-on-dgx-spark-that-outperform-top-cloud-models/359569/17 "2026-03-07T10:32:24Z")

</div>

Just bought a spark in Feb 2026. This is my first post. **This is excellent.** Thank you so much for sharing your knowledge here.

---

<div class="post-metadata">

**Author:** ![AoE](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aoe/32/489021_2.png) [@AoE](https://forums.developer.nvidia.com/u/AoE)\
**Post date:** [March 10, 2026, 2:20pm UTC](https://forums.developer.nvidia.com/t/building-local-hybrid-llms-on-dgx-spark-that-outperform-top-cloud-models/359569/18 "2026-03-10T14:20:48Z")

</div>

Thanks for sharing your experience Mark :)

I recently set up my own RAG for OWUI and went a different route (different needs), so I have a couple of questions:

- **What are you using as the pre-processor?**  
That would influence the quality of the content you send to the embedder. I went with Docling, but have to avoid OCR as it eats too much VRAM
- **How do you check the output of the document processing?**  
OWUI shows you the markdown output right after it’s down if you click on the file, so you can evaluate how it went.
- **What happens with pictures in the documents?**  
I’m using a multimodal model (Qwen3.5) as my main LLM so that I can get descriotions)
- **How do you select specific collections when asking questions?**  
In OWUI, collections or files can be invoked when needing to have them in the context

Thank you! :)

---

<div class="post-metadata">

**Author:** ![griffith.mark](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/griffith.mark/32/453455_2.png) [@griffith.mark](https://forums.developer.nvidia.com/u/griffith.mark)\
**Post date:** [March 12, 2026, 5:24pm UTC](https://forums.developer.nvidia.com/t/building-local-hybrid-llms-on-dgx-spark-that-outperform-top-cloud-models/359569/19 "2026-03-12T17:24:39Z")

</div>

@AoE, thanks again for the thoughtful questions—you’re spot on that preprocessing and collection selection are where most RAG quality lives. My stack is still evolving, but here’s the current state and my next priorities.

**1. Arxiv + domain layer structure**  
My initial focus has been on self hosting the full **librarian\_bots/arxiv\_metadata\_snapshot** (~2.96M papers, JSONL, ~4.8 GB) with IDs, titles, authors, abstracts, subjects, DOIs, and journal refs.[1][2]  
Instead of a single huge collection, I split into **14 subject-area collections in Qdrant** following Arxiv categories roughly:

- `arxiv-cs-ml-ai` — ML/AI/neural nets
- `arxiv-cs-systems-theory` — systems, complexity, algorithms
- `arxiv-math-pure`, `arxiv-math-applied`, `arxiv-math-phys`
- `arxiv-stat-eess` — stats, signal/electronic systems
- `arxiv-quantph-grqc` — quantum + GR
- `arxiv-hep` — high-energy physics
- `arxiv-condmat` — condensed matter
- `arxiv-astro` — astrophysics
- `arxiv-nucl-nlin-physother` — nuclear, nonlinear, misc physics
- `arxiv-qbio-qfin-econ` — q-bio, q-fin, econ
- `arxiv-misc` — catch-all

Plus curated domain collections (Attention\_Mechanisms, Dropout\_Techniques, Batch\_Normalization, Residual\_Networks, Transformer\_Architectures, Optimization\_Methods, etc.) for targeted retrieval over broad search.[3][1]

Scope note: this layer is **metadata + abstracts + citation structure only** —no full PDFs yet. Arxiv metadata is surprisingly powerful for idea tracing and citation graphs; full-text PDFs are a parallel (trickier) workstream.[2][1]

**2. Preprocessing: Docling vs current stack**  
I’ve reviewed Docling (including VLM-enriched variants like SmolDocling). On DGX Spark, full OCR + VLM can be heavy for mostly clean academic PDFs.[4][5]  
Current pipeline:

- Primary: **PyMuPDF (fitz)** — fast, reliable on standard layouts
- Fallback: **unstructured** for tricky multi-column/tables

Big limitation: figures/diagrams are dropped or become garbage—a huge loss for STEM (architecture diagrams, plots).

Fix in progress: multimodal ingestion pass

- Render page → image
- Quantized VLM (likely Qwen2-VL or Pixtral) → generate descriptions/captions
- Store descriptions in Qdrant payload alongside text chunks (searchable/embeddable)

All offline at ingestion → zero query-time cost. Not live yet.

**3. Validating processing**  
I ignore Open WebUI file preview. Ingestion writes cleaned text + metadata (pages, future figure captions) to a **processing-audit JSONL log**. I sample-check for encoding breaks, layout concatenation errors, etc. Once multimodal is on, VLM captions go into Qdrant payloads for direct querying (e.g., “find the Transformer diagram mention”).

**4. Pictures: current vs roadmap**  
Current: ignored. Retrieval falls back to caption text at best; visual content is lost.  
Roadmap: persistent VLM captions embedded/stored per figure/page. That enables queries like “Transformer architecture diagram from Vaswani et al.” to retrieve the description and map to the PDF/asset.

Your suggestion of **Qwen2.5-VL on-the-fly at inference** for ad-hoc uploads is excellent (especially for interactive reading). For my persistent corpus I want descriptions as first-class retrieval primitives, but the approaches complement each other—I’ll likely run both.[1]

**5. Why Qdrant + hybrid search wiring**  
Qdrant chosen for:[6][7][8]

- Native hybrid (dense + sparse named vectors, built-in RRF via Universal Query API—no custom fusion)
- HNSW in RAM + on-disk payload (`on_disk_payload: true`) → millions of abstracts without RAM explosion
- Collection isolation (independent re-index/WAL per subject area)

HNSW is tuned around `m=16`, `ef_construct=100`; larger collections get higher `memmap_threshold` for disk mapping.[6]

**6. Embeddings: BGE-M3 across the board**  
Three containerized BGE-M3 roles:[9][10]

- dense: 1024-dim (cosine)
- sparse: SPLADE-style token weights (exact matches: authors, arXiv IDs like 2305.14314, math notation)
- reranker: cross-encoder on fused candidates

RRF merges: strong in both → boosted; strong in one only → penalized. In practice it’s a big win on technical retrieval.[7][11][6]

Surprise: all three are lightweight on Spark—tiny GPU footprint vs the main model (Qwen3-80B etc.).[1]

**NeMo Retriever comparison**  
I did review NVIDIA’s **NeMo Retriever** docs and benchmarks while building this stack.[12][13][9] The Llama-3.2-NeMo-Retriever-1B-VLM-Embed model delivers impressive multimodal accuracy—up to 84.5% Recall@5 on Digital Corpora datasets and strong performance on financial documents like earnings reports.[12] Their full extraction pipeline (nv-ingest + nemoretriever-parse) handles charts, tables, and infographics, with NVIDIA citing up to **15× faster** multimodal PDF extraction than open-source alternatives.[12][13]

The trade-off on Spark is resource footprint. Running the full NeMo Retriever stack locally (embedding + rerank + multimodal extraction NIMs) assumes **dedicated GPU slices** —nemoretriever-parse really wants its own GPU, with documentation noting you need ~24 GB VRAM minimum for optimal throughput.[13][9] On Spark, that footprint directly competes with the main event: the 80B parameter generation model. BGE-M3 containers sip resources in comparison, coexisting comfortably alongside vLLM/Qwen3-80B on one node. For enterprise deployments with dedicated infrastructure, the full NeMo stack is absolutely the right choice.[12] On an edge box where generation is the priority and VRAM is the hard bottleneck, “lighter with good enough performance on academic text” wins for now.[1][13]

**7. Query-time collection routing**  
Users see none of this. (OpenAI-compatible `/v1/chat/completions`) handles:[1]

1. Cache check (Valkey/Redis)
2. Small router (e.g. GLM-4-9B-Chat) classifies query
3. Routes to 1+ collections (e.g. “dropout vs batch normalization” → `arxiv-cs-ml-ai` + `Dropout_Techniques`; “residual connections in transformers” → `Transformer_Architectures` + `Residual_Networks`; “Adam optimizer convergence” → `Optimization_Methods` + `arxiv-cs-ml-ai`)
4. Parallel dense + sparse embed
5. Hybrid RRF search
6. Optional SearXNG for web/time-sensitive → merge
7. BGE-M3 rerank
8. Prompt → vLLM (Qwen3-80B / Phi-4-mini)
9. Citations, confidence, cache write, Langfuse trace

LangGraph orchestrates this (parallel branches, conditionals, fallbacks) while exposing a clean API to Open WebUI/LiteLLM.[1]

**8. vs Open WebUI built-ins**  
Even without figures, subject routing + hybrid RRF + rerank + cache/web fusion beats flat-file blobs (no hybrid, no routing, no rerank). Multimodal captions should widen the gap significantly for visual STEM content once they’re live.[6][1]

Happy to share stripped-down code/config—Qdrant hybrid schema (named vectors + RRF), embedding service setup, or LangGraph wiring. Which would you want first?

Sources  
[1] Building Local + Hybrid LLMs on DGX Spark That Outperform Top … [Building Local + Hybrid LLMs on DGX Spark That Outperform Top Cloud Models](https://forums.developer.nvidia.com/t/building-local-hybrid-llms-on-dgx-spark-that-outperform-top-cloud-models/359569)  
[2] arXiv Dataset [arXiv Dataset | Kaggle](https://www.kaggle.com/datasets/Cornell-University/arxiv/versions/152/data)  
[3] arxiv-search by WILLOSCAR - Agent Skills [arxiv-search by WILLOSCAR | Agent Skills](https://www.agentskills.in/marketplace/@WILLOSCAR/arxiv-search)  
[4] Understanding Docling for Structured Document Processing [Understanding Docling for Structured Document Processing](https://www.llamaindex.ai/glossary/what-is-docling)  
[5] Docling: The Document Alchemist | Towards Data Science [Docling: The Document Alchemist | Towards Data Science](https://towardsdatascience.com/docling-the-document-alchemist/)  
[6] Hybrid Search Revamped - Building with Qdrant’s Query API [Hybrid Search Revamped - Building with Qdrant's Query API](https://qdrant.tech/articles/hybrid-search/)  
[7] Demo: Implementing a Hybrid Search System - Qdrant [Demo: Implementing a Hybrid Search System - Qdrant](https://qdrant.tech/course/essentials/day-3/hybrid-search-demo/)  
[8] Hybrid Search and the Universal Query API - Qdrant [Hybrid Search and the Universal Query API - Qdrant](https://qdrant.tech/course/essentials/day-3/hybrid-search/)  
[9] BGE-M3 — BGE documentation [BGE-M3 — BGE documentation](https://bge-model.com/bge/bge_m3.html)  
[10] BGE-M3 — BGE documentation [BGE-M3 — BGE documentation](https://bge-model.com/tutorial/1_Embedding/1.2.4.html)  
[11] BAAI/bge-m3 · Reranker - Hugging Face [BAAI/bge-m3 · Reranker](https://huggingface.co/BAAI/bge-m3/discussions/30)  
[12] NVIDIA NeMo Retriever Delivers Accurate Multimodal PDF Data Extraction 15x Faster [https://developer.nvidia.com/blog/nvidia-nemo-retriever-delivers-accurate-multimodal-pdf-data-extraction-15x-faster/](https://developer.nvidia.com/blog/nvidia-nemo-retriever-delivers-accurate-multimodal-pdf-data-extraction-15x-faster/)  
[13] Run Multimodal Extraction for More Efficient AI Pipelines Using One GPU [Run Multimodal Extraction for More Efficient AI Pipelines Using One GPU | NVIDIA Technical Blog](https://developer.nvidia.com/blog/run-multimodal-extraction-for-more-efficient-ai-pipelines-using-one-gpu/)

---

<div class="post-metadata">

**Author:** ![AoE](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aoe/32/489021_2.png) [@AoE](https://forums.developer.nvidia.com/u/AoE)\
**Post date:** [March 15, 2026, 10:07am UTC](https://forums.developer.nvidia.com/t/building-local-hybrid-llms-on-dgx-spark-that-outperform-top-cloud-models/359569/21 "2026-03-15T10:07:03Z")

</div>

I’m using the cuda130 image and it should be using the GPU, but I haven’t run any tests to specifically check if it was.

Here is the compose file. I’m not compiling anything, but based on the recipe for docling-serve, I’m sure you can get it to perform better with a custom image.

```auto
  docling:
    image: ghcr.io/docling-project/docling-serve-cu130:main
    container_name: docling
    restart: unless-stopped
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    environment:
      DOCLING_SERVE_ENABLE_UI: false
      DOCLING_SERVE_ENABLE_REMOTE_SERVICES: true
      DOCLING_SERVE_ALLOW_CUSTOM_VLM_CONFIG: true
      DOCLING_SERVE_ALLOW_CUSTOM_PICTURE_DESCRIPTION_CONFIG: true
      DOCLING_SERVE_ALLOW_CUSTOM_CODE_FORMULA_CONFIG: true
      DOCLING_SERVE_MAX_SYNC_WAIT: 600
      DOCLING_SERVE_ENG_LOC_NUM_WORKERS: 4
      DOCLING_NUM_THREADS: 4
      UVICORN_WORKERS: 1
      DOCLING_DEVICE: cuda
      DOCLING_PERF_PAGE_BATCH_SIZE: 16
      DOCLING_SERVE_OCR_BATCH_SIZE: 2
      DOCLING_SERVE_LAYOUT_BATCH_SIZE: 16
      DOCLING_SERVE_TABLE_BATCH_SIZE: 4
    volumes:
      - /opt/share/models/docling:/opt/app-root/src/.cache/docling/models
      - /opt/share/models/docling/torch_kernels:/opt/app-root/src/.cache/torch/kernels
      - /opt/share/uploads:/uploads:ro
      - /opt/share/docling/chunks:/data/chunks:rw
    command: >
      bash -c "docling-serve run"

```

And thanks for the previous reply. I still need to dig into the detailed information that you provided :)

---

<div class="post-metadata">

**Author:** ![griffith.mark](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/griffith.mark/32/453455_2.png) [@griffith.mark](https://forums.developer.nvidia.com/u/griffith.mark)\
**Post date:** [June 18, 2026, 10:40pm UTC](https://forums.developer.nvidia.com/t/building-local-hybrid-llms-on-dgx-spark-that-outperform-top-cloud-models/359569/22 "2026-06-18T22:40:14Z")

</div>

> **[Surpassing Frontier Performance with Fusion — OpenRouter Blog](https://openrouter.ai/blog/announcements/fusion-beats-frontier/)**
>
> A panel of budget models, fused through OpenRouter, outscored GPT-5.5 and Claude Opus 4.8 on 100 complex research tasks.

[Next page](https://forums.developer.nvidia.com/t/building-local-hybrid-llms-on-dgx-spark-that-outperform-top-cloud-models/359569.md?page=2)