Building Local + Hybrid LLMs on DGX Spark That Outperform Top Cloud Models

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10 Projects
, , , , ,
7

Playbook 2 – Turning Spark Arena Models into a Unified Multi‑Mode Assistant with LiteLLM + Open WebUI

Since my original post, things have evolved a lot. Spark Arena and the GB10 community have gone from “here are some cool models” to a full recipe + leaderboard ecosystem that makes it trivial to stand up Nemotron‑3‑Nano NVFP4, GLM‑4.7‑Flash‑AWQ, Qwen3‑Coder‑Next‑FP8, Qwen‑Instruct‑80B and more on a single or dual DGX Spark. The net effect is a quantum leap in developer velocity: instead of burning hours on flags and container wiring, we can treat these as reliable building blocks and focus on UX, routing, and product behavior.

Huge thanks to Raphael Amorim (@raphael.amorim), @eugr, and everyone working on spark‑vllm‑docker, llama‑benchy, and Spark Arena itself — Playbook 2 is entirely about what you can build on top of their work, not yet another way to launch a model.

What follows is how, on a single Spark (usually one primary Spark Arena model up at a time), you can use LiteLLM + Open WebUI to turn that into a clean, multi‑mode assistant stack with natural model names like NeMo Expert, GLM Expert, Qwen Coder Fast, Phi Fast, Grok-4, etc. LiteLLM resolves everything; Open WebUI just shows these names in the dropdown.

1. Architecture: Open WebUI → LiteLLM → Spark Arena + hosted

The architecture is intentionally simple:

  • Open WebUI

    • Talks only to LiteLLM via an OpenAI‑compatible endpoint.

    • In start_core_services.yml:

      environment:
  OPENAI_API_BASE_URL: http://litellm:4000/v1
  OPENAI_API_KEY: simple-api-key

  • LiteLLM proxy/router

    • Runs as a small service next to Open WebUI:

      services:
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    container_name: litellm
    restart: always
    command: --config /app/config.yaml --num-workers 4
    ports:
      - "4000:4000"
    volumes:
      - ./litellm_config.yaml:/app/config.yaml:ro
    env_file:
      - /home/mark/vllm/model-configs/env.env
    networks: [llm-net]

    • Reads a single litellm_config.yaml that defines all local and hosted models plus routing rules.

  • Spark Arena vLLM backends + hosted APIs

    • One primary big chat model via Spark Arena (Nemotron‑3‑Nano or GLM‑4.7 or Qwen‑Instruct) at a time, plus Qwen‑Coder and a small router like Phi‑4‑mini.
    • Hosted backups: Grok‑3/4, Qwen‑Max/Qwen3 cloud, GLM‑4.7 cloud, NVIDIA Llama‑3‑70B, etc.

Open WebUI only sees model names from LiteLLM – e.g., NeMo Expert, GLM Expert, Qwen Coder Fast, Phi Fast, Grok-4 – and LiteLLM resolves them to whichever backend is actually running.

2. LiteLLM model_list: real model names, real modes

Your recipes from Spark Arena already defines the modes you need, using concrete model names.

Local Nemotron‑Nano profiles

# ── LOCAL NEMOTRON NANO (optimized for speed) ──
- model_name: "Base" # Native (thinking ON)
  litellm_params:
    model: openai/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4
    api_base: http://host.docker.internal:8000/v1
    api_key: os.environ/NEMOTRON_API_KEY
    temperature: 0.4
    top_p: 0.9
    max_tokens: 8192
    extra_body: {}
    chat_template_kwargs:
      enable_thinking: true
- model_name: "Fast"
  litellm_params:
    model: openai/microsoft/Phi-4-mini-instruct
    api_base: http://phi-judge:8000/v1
    api_key: simple-api-key
    temperature: 0.2
    top_p: 0.9
    max_tokens: 8192
    timeout: 30
    initial_messages:
      - role: system
        content: |
          You are a fast, local judge / evaluator / classifier.
    extra_body: {}
- model_name: "Expert" # Balanced reasoning
  litellm_params:
    model: openai/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4
    api_base: http://host.docker.internal:8000/v1
    api_key: os.environ/NEMOTRON_API_KEY
    temperature: 0.6
    top_p: 0.98
    max_tokens: 16384
    timeout: 300
    extra_body: {}
    chat_template_kwargs:
      enable_thinking: true
      effort: "medium"
- model_name: "Heavy"
  litellm_params:
    model: openai/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4
    api_base: http://host.docker.internal:8000/v1
    api_key: os.environ/NEMOTRON_API_KEY
    temperature: 1.0
    top_p: 1.0
    max_tokens: 32768
    timeout: 600
    stream_timeout: 240
    extra_body: {}
    chat_template_kwargs:
      enable_thinking: true
- model_name: "Code"
  litellm_params:
    model: openai/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4
    api_base: http://host.docker.internal:8000/v1
    api_key: os.environ/NEMOTRON_API_KEY
    temperature: 0.2
    top_p: 0.95
    max_tokens: 16384
    timeout: 300
    stream_timeout: 180
    extra_body: {}
    chat_template_kwargs:
      mode: code
      enable_thinking: true
      clear_thinking: false

GPT‑OSS‑120B profiles

# ===================GPT OSS 120B ==========================================
- model_name: "Fast"
  description: Ultra-fast classifier / router / moderator – short & strict
  litellm_params:
    model: openai/gpt-oss-120b
    api_base: http://host.docker.internal:8010/v1
    api_key: sk-1234
    temperature: 0.2
    top_p: 0.9
    max_tokens: 2048
    timeout: 12
    initial_messages:
      - role: system
        content: |
          You are a fast, local classifier/router/moderator.
- model_name: "Expert"
  description: Balanced reasoning – good default quality/speed trade-off
  litellm_params:
    model: openai/gpt-oss-120b
    api_base: http://host.docker.internal:8010/v1
    api_key: sk-1234
    temperature: 0.55
    top_p: 0.95
    max_tokens: 32768
    timeout: 400
    extra_body: {}
    chat_template_kwargs:
      enable_thinking: true
      effort: medium
- model_name: "Heavy"
  description: Maximum reasoning depth & longest allowed outputs
  litellm_params:
    model: openai/gpt-oss-120b
    api_base: http://host.docker.internal:8010/v1
    api_key: sk-1234
    temperature: 0.75
    top_p: 0.98
    max_tokens: 65536
    timeout: 1200
    stream_timeout: 600
    extra_body: {}
    chat_template_kwargs:
      enable_thinking: true
      effort: maximum
- model_name: "Code"
  description: Programming & technical tasks – prefers clean output
  litellm_params:
    model: openai/gpt-oss-120b
    api_base: http://host.docker.internal:8010/v1
    api_key: sk-1234
    temperature: 0.15
    top_p: 0.85
    max_tokens: 32768
    timeout: 600
    extra_body: {}
    chat_template_kwargs:
      mode: code
      enable_thinking: true
      clear_thinking: true

2.5. How “modes” map to temperature / length / effort

The different “modes” (Fast, Expert, Heavy, Code) are just concrete per‑model profiles with carefully chosen parameters in litellm_config.yaml:

  • Temperature

    • Fast / classifier modes (Fast) use low temperature (around 0.15–0.2) and tighter top_p for short, stable, deterministic outputs (routing, judging, quick edits).
    • Expert modes (Expert) sit in the 0.4–0.6 range with slightly higher top_p for balanced reasoning and creativity.
    • Heavy modes (Heavy) use higher temperature and relaxed top_p so they can explore more options when you explicitly want deep thinking or long, open‑ended answers.

  • max_tokens / timeouts

    • Fast modes cap max_tokens low and use short timeouts to enforce quick responses and trigger fallbacks when a request clearly needs more room.
    • Expert modes increase max_tokens and timeouts to comfortably handle most daily chat and analysis.
    • Heavy / Max modes increase max_tokens aggressively and extend timeout / stream_timeout to cover long documents, full files, and multi‑step reasoning.

  • effort / thinking flags

    • Nemotron and GPT‑OSS profiles use enable_thinking: true and effort (medium, maximum) where supported to hint how hard to “think” internally before answering.
    • Code profiles set mode: code (and sometimes clear_thinking) to bias toward clean code blocks and minimal chatter.

When you pick Fastvs Expert vs Heavy vs Code, you’re really choosing a bundle of temperature + length + timeout + prompt discipline that you’ve pre‑curated for that model’s sweet spot, from Hugging Face, UnSloth model cards etc.

  1. Routing and fallbacks (concrete names, Grok‑4 fallback)

The final piece is the routing_strategy and router_settings. Using the same mode names "Fast" "Expert" for GL, Nemotron Nano etc with a simple, explicit fallback chain ending at Grok-4 results a clean UI and model dropdown menu.

# ── ROUTING & SETTINGS ───────────────────────────────────────────────────────
routing_strategy: usage-based-routing
router_settings:
  max_tokens_threshold: 4096
  fallback_on_max_tokens: true
  retry_on_failure: true
  num_retries: 2
  timeout: 60

# Corrected fallbacks – each fallback entry must list the model names that
# should be used when the primary model fails or when the token limit is hit.
fallbacks:
  - "Fast":
      fallback_to: ["Fast","Expert", "Heavy"]
  - "Expert":
      fallback_to: ["Expert","Heavy"]
  - "Heavy":
      fallback_to: ["Heavy","Grok-4"]

# Context‑window fallback mappings – ensure that when the primary model’s
# context window is exhausted, the system falls back to a model with a
# larger window.
context_window_fallbacks:
  "Fast":   ["Expert", "Heavy"]
  "Expert": ["Heavy"]
  "Heavy":  ["Heavy", "Grok-4"]

You keep one version of this block active at a time, depending on which Spark Arena recipe is your primary. Either way:

  • Fast classifiers escalate into their corresponding Expert/Heavy modes and then Grok-4.
  • The primary local assistant (Expert ) escalates to its Heavy profile and then Grok-4.
  • Code‑oriented models fall back to Qwen‑Coder profiles and, if needed, Grok-4.

No generic labels; everything is grounded in real model_name strings from your config, with Grok-4 as the single hosted backstop.

4. Phi Mini in the stack – the fast tier

Phi Fast runs Microsoft’s Phi‑4‑mini‑instruct as a dedicated small, fast tier, wired as its own vLLM instance on phi-judge:8000. Large quantized models like Nemotron‑3‑Nano NVFP4, GLM‑4.7‑Flash‑AWQ, and Qwen3‑Coder‑Next‑FP8 typically use < 62% of DGX Spark’s VRAM in their Spark Arena recipes, which leaves comfortable headroom to keep Phi‑4‑mini loaded at the same time together with Qdrant, embedding model and reranking model etc.

This gives you:

  • Sub‑second, ultra‑cheap responses for classification, routing, moderation, and quick Q&A.
  • A local‑only Fast tier: Fast is always tried first for short/simple tasks, and only escalates to Expert/Heavy/ Grok-4 when it hits timeouts, token limits, or context constraints via the routing rules above.

You get snappy, resilient behavior without complicating Open WebUI at all; it just exposes Fast as another model, and LiteLLM handles the routing.

5. What this gives you in practice

On a single DGX Spark:

  • Spark Arena recipes give you vLLM deployments for Nemotron, GLM, Qwen‑Coder, Qwen‑Instruct.
  • LiteLLM wraps those (plus small locals like Phi‑4‑mini and hosted models like Grok‑4, Qwen‑Max/Qwen3, Llama‑3‑70B) into a single OpenAI‑compatible endpoint with per‑model routing and fallbacks using real model names.
  • Open WebUI talks only to LiteLLM and simply lists Fast, Expert, Heavy, Code, etc. — nothing abstract, no alias layer.

The result is a hosted‑grade, multi‑model assistant experience with simple, concrete model choices and robust failover, running entirely on your hardware, with Spark Arena doing the heavy lifting underneath.

Auto mode will be added at a future date to automatically route to these basic modes, MoE and compound cloud/local MoE modes.

@raphael.amorim, @eugr thank you for providing the community Spark Arena, recipes etc, it is great having a foundation to build on! There is still more work to do here integration of Modes and Auto Mode, lets think about developing a community front end for the model in Spark Arena Models to provide a complete solution for Spark Users. I do like the openWebUI front end it does provide a lot of flexibility for customization. I do really like Qdrant as a backend vector database for openWebUI. The backend with RAG and pipelines are very simple and work well once you work out how to code them but very hard to get going the first time when you are trying to configure everything. I will post more on RAG and building RAG corpus at a future date.

Many thanks,

Mark