My Dual Sparks setup plan

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
, ,
1

# Sovereign Stack: Local Inference Buildout v2.2

**Vintage Macro Studios — April 5, 2026**

Background

I’m building a local inference cluster with two DGX Sparks (ASUS Ascent GX10) connected through a MikroTik CRS804-4DDQ 400G switch, with a Puget Systems x86 box (RTX 5090) as the cockpit/orchestration machine.

I spent a full day cross-referencing Gemini, ChatGPT, and Claude (Opus 4.6) to build a sequenced buildout plan. All three models initially converged on a conservative approach: start on the Puget (x86), use Ollama as a bridge on the Sparks, treat vLLM on ARM64/sm_121 as a fragile source-build exercise, and defer multi-node clustering as an advanced topic.

**That consensus was wrong.** None of the three models searched current sources before advising. When I pushed Claude to verify against actual documentation, it found that the ecosystem has moved substantially:

  1. Pre-built vLLM wheels for DGX Spark (ARM64/sm_121/CUDA 13) now ship nightly via community repos — no source builds required.
  2. NVIDIA officially supports 4-node DGX Spark clustering with RoCE as of March 2026.
  3. Community Docker tooling (spark-vllm-docker by @eugr) explicitly supports RoCE switch topologies, including my exact configuration.
  4. Multiple DGX OS updates (November 2025 through March 2026) addressed unified memory reporting, kernel stability, and multi-node performance regressions.
  5. NVIDIA claims up to 2.5x performance improvement since launch via software-only optimizations.

After Claude surfaced the current sources, ChatGPT confirmed: “Yeah, we both got it wrong.”

**Lesson learned: Don’t trust multi-model consensus without current source verification.** Three frontier models all missed publicly indexed information that materially changed the plan.

The Revised Plan

Step 1 — Keep Claude Code

This project adds a second lane, not a replacement. Claude Code remains the best option for Opus-level reasoning, 1M token context, and zero-setup agentic workflows. The local stack is the privacy, control, and fixed-cost lane. Hybrid is the strongest position.

Step 2 — First proof-of-life: NVIDIA’s official vLLM container on Spark 1

Skip the Puget detour. Skip the Ollama bridge. NVIDIA publishes an official vLLM guide for DGX Spark (link 1 below) with a pre-built Docker container path. Use this as the first smoke test — it’s the known-good NVIDIA baseline.

```bash
vllm serve Qwen/Qwen3-Coder-Next \
–port 8000 \
–tensor-parallel-size 1 \
–enable-auto-tool-choice \
–tool-call-parser qwen3_coder \
–max-model-len 32768
```

Why skip the Puget? The Sparks are the inference hardware. Using the Puget as a temporary inference box validates vLLM on the wrong architecture.

Why skip Ollama? With NVIDIA’s own container path documented and working, vLLM on Spark is the direct path. Ollama won’t give you batching, PagedAttention, or OpenAI-compatible tool calling.

Step 2b — Move to spark-vllm-docker for newer vLLM and cluster features

Once the official container serves cleanly, switch to the community spark-vllm-docker project (link 2 below). It provides pre-built vLLM wheels (currently v0.17.2rc1, CUDA 13.1), pre-built FlashInfer wheels, Docker containers that build in 2-3 minutes, and explicit support for single-node, dual-node direct connect, dual-node via RoCE switch, and 3-node mesh topologies.

```bash
git clone GitHub - eugr/spark-vllm-docker: Docker configuration for running VLLM on dual DGX Sparks · GitHub
cd spark-vllm-docker
./build-and-copy.sh
./run-recipe.sh qwen3-coder-next-int4-autoround --solo
```

Step 3 — Start with 32K context, not 256K

Qwen’s own model card says: reduce context to 32,768 if memory pressure occurs. Do that from the start. Validate the endpoint works before chasing max context.

Step 4 — Point Aider at the Spark 1 endpoint

```bash
aider --model openai/Qwen3-Coder-Next \
–openai-api-base http://:8000/v1 \
–openai-api-key not-needed
```

Qwen3-Coder-Next is an 80B MoE with only 3B active parameters, 256K native context, Apache 2.0. It scores 70.6% on SWE-Bench Verified and was explicitly designed to work with Aider, Claude Code, Cline, and other CLI/IDE harnesses.

Step 5 — Add Spark 2 to the cluster via RoCE through the MikroTik

NVIDIA officially supports this topology. The DGX Spark playbooks (link 3 below) include a Connect Two Sparks guide covering static IP assignment, passwordless SSH, and discover-sparks scripts.

```bash

Assign static IPs on the CX-7 interfaces

sudo ip addr add 192.168.100.10/24 dev enP2p1s0f1np1 # Spark 1
sudo ip addr add 192.168.100.11/24 dev enP2p1s0f1np1 # Spark 2

From Spark 1, build and distribute the container

cd spark-vllm-docker
./build-and-copy.sh --copy-to

Launch the cluster

./run-recipe.sh qwen3-coder-next-int4-autoround --setup
```

Target RoCE bandwidth: ~111 Gb/sec per logical interface.

Step 6 — Confirm Sparks are current on DGX OS

Check the release notes (link 4 below). Key updates since launch include the November 2025 release (Ubuntu 6.14 HWE kernel, CUDA 13.0.2, unified memory fix), January 2026 release (ConnectX-7 hot-plug power management), February 2026 rolling update, and March 2026 hotfix (fixed multi-node performance regressions from the February update).

Step 7 — Performance tuning

Once the cluster is stable, explore NVFP4 quantization (up to 2.5x gains), gpu-memory-utilization tuning, fastsafetensors loader, prefix caching, and removing --enforce-eager if CUDA graph capture works on your firmware version (20-30% throughput recovery).

Community benchmarks: Qwen3.5-397B-INT4-Autoround on a 4-node cluster hits 37 tok/s single-user, 103 tok/s with 4 concurrent users.

Step 8-10 — Future

  • OpenClaw/NemoClaw agent orchestration (only after model serving is stable)
  • Qwen Code as a second CLI front end
  • Scale to four Sparks (512GB unified, 700B-class models) — already supported by NVIDIA and by my MikroTik switch

What NOT to do first

  • Don’t start on the Puget — start on Spark 1 directly
  • Don’t use Ollama as a bridge — go straight to NVIDIA’s official vLLM container, then community docker
  • Don’t start with 256K context — start with 32K
  • Don’t start with OpenClaw/NemoClaw — start with Aider
  • Don’t start with 405B-class models — start with Qwen3-Coder-Next (80B/3B active)
  • Don’t rip out Claude Code — keep both lanes
  • Don’t trust multi-model consensus without current source verification

Key Links

*Built by the Sovereign Wizard at Vintage Macro Studios. Plan developed collaboratively with Claude (Opus 4.6) and ChatGPT (GPT-4o), then verified against current NVIDIA documentation. The AI models were helpful — after being caught giving stale advice.*

4

Thanks for sharing! I moved it to the DGX Spark topic for more visibility!

5

thanks

6

Just curious why are you using a switch to connect the two sparks to each other? Plan to expand to move of them later?

7

Good question. Yes, expansion is the plan - targeting 4 Sparks eventually (512GB unified memory for 700B-class models). The MikroTik CRS804 supports that topology natively.

I also wanted the Puget Systems x86 box (RTX 5090) connected to the same fabric via its ConnectX-7 NIC - same approach Alex Ziskind used with his 8-Spark cluster. Having the orchestration machine on the same high-speed network makes sense for my workflow.

The switch gives me flexibility: start with 2 nodes, scale to 4 without rewiring, and keep the Puget integrated for hybrid workloads.

Thanks for the question.

8

I mean this is very cool but you may learn an expensive lesson that Alex does this to create content for his channel, not because its practical. You do you though. I’ll keep up on your updates if you continue sharing them.

9

Fair point. I’m not assuming Alex’s path is automatically the practical one for everyone. For me this is partly a learning project and partly a test of what’s actually useful in a small local setup. I’ll keep sharing the wins and the mistakes.

10

Is this unified memory actually true? I have 2 Sparks and all the resources I found say connecting two DGX Sparks with the QSFP cable gives you access to roughly 256 GB of total model capacity, but it does not fuse the two boxes into one coherent 256 GB memory space like a single Spark’s internal unified memory.

11

You’re correct, and that’s sloppy phrasing on my part — thanks for the catch.
Clustering Sparks does not fuse them into one coherent unified memory space. Each
GB10 has 128 GB of unified memory internally (CPU and GPU sharing the same
LPDDR5X), but across nodes you get aggregate capacity, not unified memory. The
model gets sharded — tensor-parallel or pipeline-parallel — with each box holding
its slice of the weights and activations crossing the ConnectX-7 link via NCCL.
No cache coherence, no single address space.

So “512 GB unified” was wrong wording on my end — it should read “aggregate
capacity for sharded serving.” And to be precise about my own setup: I’m running
two Sparks (256 GB aggregate) today, not four; more nodes are a maybe-later, not
a committed plan.

Two practical notes from actually running this on my pair, since the distinction
bites in real ways:

  1. Effective capacity is a good bit less than nominal. On my dual-Spark
    MiniMax-M2.7 deployment (TP=2), weights are ~61 GB per node, but KV cache,
    activation buffers, and OS page cache eat the rest fast — page cache during
    safetensors loading can consume nearly all of a node’s ~121 GB usable if you’re
    not careful. Budget for roughly 75–80% of nominal, not 100%.
  2. The interconnect is the tax you pay for the aggregate. Every forward pass
    moves data between boxes, which is why the fabric config (RoCE, MTU, and
    critically NCCL_NET_GDR_LEVEL=0 on GB10 — GPUDirect RDMA will hard-lock the SoC)
    matters as much as the memory math.

Appreciate you keeping it precise.

— Leathery Tendons

12

Thanks!

I would be 100% more confident in your reply if it weren’t clearly LLM generated ;)