Multi-Node DGX Spark Cluster (4×) — K3s, SGLang/vLLM, ConnectX-7 SR-IOV, Full Benchmark Matrix

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10 Projects
1

I’ve been running a 4-node DGX Spark cluster for distributed LLM inference and wanted to share the setup, findings, and benchmark results with the community. The entire infrastructure is open-source as an Ansible
repository.

GitHub: GitHub - vroomfondel/dgxarley: Ansible playbooks for a 3-node K3s cluster with NVIDIA DGX Spark nodes for distributed LLM inference · GitHub

Hardware

  • 4× DGX Spark (ASUS Ascent GX10, GB10 / SM121, 128 GB each)
  • 1× HP EliteDesk 800 G4 (x86_64) as K3s master + control plane
  • ConnectX-7 QSFP28 100GbE mesh (MTU 9000) via MikroTik CRS812-8DS-2DQ-2DDQ-RM
  • MikroTik CRS310-8G+2S+ for GbE management + VLAN translation

Software Stack

  • K3s (lightweight Kubernetes) across all 5 nodes
  • SGLang (0.5.9-dev2 and 0.5.10rc0) and vLLM (0.17.0) for multi-node inference
  • Multus CNI with host-device plugin for QSFP network injection into pods
  • NCCL over Socket (100GbE QSFP mesh) for tensor/expert/pipeline parallelism
  • Ansible for full lifecycle management (networking → OS prep → K3s → K8s workloads)

ConnectX-7 SR-IOV (Optional)

One of the more interesting pieces: the repo includes full SR-IOV support for the ConnectX-7 QSFP NICs, allowing multiple pods to share the 100GbE link via Virtual Functions. Some gotchas I documented:

  • VF interface naming drops the npN suffix (PF enp1s0f0np0 → VF enp1s0f0v0)
  • VF MACs must be set on the VF interface directly (ip link set address ), not via the PF’s vf N mac command — the NIC’s internal switch can’t route unicast otherwise
  • VFs don’t inherit PF MTU — each needs explicit MTU 9000 in Netplan
  • tcpdump on the PF doesn’t see VF traffic in legacy eSwitch mode
  • Multus host-device with “ipam”: {“type”: “static”} (empty block) is required for annotation-based IP assignment on VFs

Benchmark Results (MiniMax-M2.5-NVFP4)

I’ve been systematically testing nvidia/MiniMax-M2.5-NVFP4 across different parallelism strategies, backends, and node counts. Full test logs are in the repo (TESTLOG_*.md files). Summary of key results:

┌───────┬─────────────┬────────────┬───────────────┬───────────────┬──────────────────────────────────────────┐
│ Nodes │ Parallelism │ SGLang │ Best 1‖ tok/s │ Best 8‖ tok/s │ Config │
├───────┼─────────────┼────────────┼───────────────┼───────────────┼──────────────────────────────────────────┤
│ 3 │ PP=3 │ 0.5.9-dev2 │ 16.1 │ ~52 │ triton MoE, flashinfer attn, CUDA graphs │
├───────┼─────────────┼────────────┼───────────────┼───────────────┼──────────────────────────────────────────┤
│ 3 │ PP=3 │ 0.5.10rc0 │ 16.0 │ 77.4 │ triton MoE, triton attn, CUDA graphs │
├───────┼─────────────┼────────────┼───────────────┼───────────────┼──────────────────────────────────────────┤
│ 4 │ TP=4, EP=4 │ 0.5.9-dev2 │ 15.6 │ 70.7 │ fi_cutlass MoE, triton attn, eager │
└───────┴─────────────┴────────────┴───────────────┴───────────────┴──────────────────────────────────────────┘

Notable Findings

  1. Xid 13 (Illegal Instruction) on SM121 — flashinfer_cutlass MoE dispatch kernel produces invalid instructions on GB10. Fix: moe_runner_backend=triton. A separate Xid 13 comes from fp4_gemm_backend=auto selecting
    flashinfer_cudnn JIT — fix: force flashinfer_cutlass.
  2. triton MoE is broken under EP > 1 (4-node TP=4 EP=4) — consistent “EP inference bug → worker crash.” Only fi_cutlass MoE works for EP=4 on these images.
  3. RoCE is 2× slower than TCP/Socket in our setup despite GPU Direct RDMA reporting “enabled.” Socket transport over the 100GbE QSFP mesh is the recommended path until RoCE is properly debugged (likely PFC/ECN
    misconfiguration or driver fallback path).
  4. CUDA graphs give +67% throughput at concurrency (4‖: 18.9 → 31.5 tok/s on 3-node PP=3). Single-request throughput is nearly unaffected.
  5. SGLang EADDRINUSE bug (multi-node head) — the Scheduler subprocess binds :, so uvicorn on 0.0.0.0 clashes. Workaround: omit --host (uvicorn defaults to 127.0.0.1) + HAProxy sidecar to forward external
    traffic. Still present in 0.5.10rc0.
  6. Piecewise CUDA graphs always OOM — too many variable-length chunks × 256 experts. disable_piecewise_cuda_graph=true is permanently required.

What’s in the Repo

  • Complete Ansible automation (MikroTik switch config, OS hardening, K3s bootstrap, K8s workloads)
  • SGLang and vLLM Deployment manifests with HAProxy sidecar, NCCL tuning, startup/liveness probes
  • Multus + host-device CNI setup for QSFP injection
  • ConnectX-7 SR-IOV automation (VF creation, Netplan, per-deployment VF assignment)
  • Full test log matrix (TESTLOG_*.md) with 50+ test configurations
  • HuggingFace model preload Jobs with cross-node rsync
  • Prometheus/Grafana monitoring stack, NUT UPS monitoring
  • Python package (dgxarley on PyPI) with integration test CLIs

Happy to answer questions or discuss findings. The repo is actively maintained as I continue testing new SGLang/vLLM releases and model profiles.

2 Likes