Im guessing Im not the only one working on a multi unit dashboard but is my humble attempt at it. It open source if anyone likes it: GitHub - LackOfSkillz/sparkDash: sparkDash ⚡ — Multi-DGX Spark Monitoring Dashboard · GitHub
Improvements in my fork over upstream
1. Cluster-aware overview (new capability, not in upstream)
Upstream’s overview is per-node cards. Yours adds a whole cluster layer:
- Cluster summary header — model, backend, topology (TP=2), max context, request state, API endpoint, plus aggregate VRAM/RAM/storage, total GPU power, mean temp/util (
clusterModel.ts,ClusterSummary.tsx) - Shared cluster inference panel measured at the head endpoint (
ClusterLlmPanel.tsx) - Aggregates distinguish “not reported” from zero — means divide by nodes that actually answered, partial totals self-label rather than presenting a half-cluster figure as cluster-wide
- HEAD/STANDALONE role display derived rather than persisted, so nothing on disk is rewritten
- Richer node cards: CPU, RAM, LAN + cluster addresses, uptime, primary compute process
- Readability pass: labels 10→12px, values 13→15px, tightened leading so the page ended up 40px shorter and still fits one 1680×1050 viewport
2. RoCE / InfiniBand telemetry (rdma.js — upstream has none)
Reads HCA counters under /sys/class/infiniband/*/ports/*/counters/ in the SSH round trip the network probe already makes. Netdev bytes read ~0 while a 200 Gb/s RDMA link is saturated, so upstream’s view was blind to the link that actually carries tensor traffic. Multiplies IB 4-byte words out to bytes, reports unknown (not zero) on first sample and counter reset, counts head-side only to avoid double-counting a point-to-point link.
3. Training visibility (TrainingProbe.js — upstream reports serving only)
A node running a fine-tune looked idle/DEGRADED with an empty MODEL field while the GPU sat at 96%. Now reports HF Trainer runs: script, step vs max_steps, live loss and its drop, sec/step, ETA, checkpoint count. Reconciles trainer_state.json (authoritative loss, stale between saves) with the tqdm bar (live step, no loss), and matches bars against max_steps so an eval bar isn’t misread as training progress.
4. Reliability of the SSH/collector path — the biggest body of work
- Windows OpenSSH support (
ssh.js) — the env whitelist was POSIX-shaped; missingProgramDatamade ssh.exe exit 255 with empty stdout and stderr, andHOMEfell back to/rootso keys were never found. Collector failures fell ~5.8/sec → ~0.075/sec. - Liveness state machine (
nodeLiveness.js) — upstream declared a node dead on one 10s probe; over a wireless management path that produced nine false-offlines during a benchmark on nodes with four days’ uptime. Now needs 3 consecutive failures AND no success in a 30s grace; one good probe restores. - SSH batching (
sshBatch.js) — ControlMaster doesn’t work on Windows OpenSSH 9.9p2, so commands to a host in the same 30ms window concatenate into one remote invocation with per-batch markers. Measured 3,929 commands over 1,303 connections instead of up to 16 simultaneous handshakes per cycle. - Metrics no longer zeroed on a failed poll — last good reading kept, age reported as a freshness band.
- Layout no longer flickers on a transient LLM probe failure (head-ness is a config property, not contingent on an HTTP probe answering).
- WebSocket fix (pre-existing upstream bug) — every new connection broadcast to all clients, so one reconnecting client made every other dashboard flash. Now only the joining client is served.
5. Tailscale fallback addressing
Optional tailscaleIp per Spark with a sticky LAN-first resolver (hostResolve.js). Deliberately not try-both-every-poll (would cost 10s against a 5s cadence). Refined twice from live measurement: dropped the 60s LAN re-probe (~700 doomed 5s attempts/node/day for something that changes twice), added a 3-consecutive-failure threshold to stop resolver thrash, then a single same-address retry that took an observed 8% storage-poll failure rate down to ~0.6%. Also: batch key derives from the resolver (or LAN and Tailscale commands would merge into one invocation), LlmProbe.baseUrl became a getter, WoL still uses lanIp (L2 broadcast has no meaning on Tailscale), and classifyHostScope learned 100.64.0.0/10 so a tailnet endpoint isn’t flagged “danger”.
6. LLM metric correctness
- Prefill rate fixed — upstream computed prompt-token delta ÷ poll interval; vLLM credits prompt tokens at admission, so an 80K prompt read 39,584 tok/s against a real ~1,600, and the tile showed “—” in 97% of samples. Now the ratio of two lifetime aggregates, verified against live metrics.
- Added
prefillTpsLive— a ~20s rolling window so the IN figure can return to zero when idle, with the drift-proof lifetime average kept as the number of record. - tok/s sparkline is now a real time series — it appended only when the value changed, so it froze showing a finished run indefinitely. Now sampled on a 1s timer, last 60 samples.
- Overview header shows both IN and OUT rates with fixed-width figures so sparklines don’t slide sideways each poll; dual-unit °C/°F everywhere.
7. Showcase: real prompts, safely (SHOWCASE-REAL-PROMPTS.md)
Upstream capped prompts at 4,000 JS characters. Yours runs genuine LineWright packets:
- Storage split first — bodies to
config/showcase-runs/<sparkId>/<sessionId>.json, index keeps summaries/hashes/byte counts. Without it, 32 prompts × 4 MiB × 20 runs exceeds Node’s max string length andJSON.stringifywould throw into a silent catch. - Two independent budgets (what may be run vs what may be kept), measured in UTF-8 bytes, never truncating a body and presenting it as complete
- Raw mode (off by default) — sends the prompt byte-exact, no fill suffix /
min_tokens/ignore_eos/ forced stop - Submitted + effective SHA-256 hashes so a run is reproducible from its own record
- Scoped body parser (
showcaseBodyParser.js) mounted before the global 100kbexpress.json()— the first attempt registered it at the route 650 lines later and never ran; caught by an HTTP integration test over a real socket, not a unit test - Raw mode + prompt identity exposed on the live polling snapshot, not just the archive
8. Branding + operator conveniences
- AetosDash branding with the real Aetos eagle mark (alpha-keyed, progressively downscaled to 96px), header pill / favicon / Showcase. The
sparkdash-themestorage key, package, Docker and API names deliberately left alone. - SSH launch buttons (
sshShell.js) — opens Windows Terminal per node. The client sends only a Spark id; host, user and options are derived server-side from config the browser can’t reach, arg-array spawn with no shell, non-Windows returns 501. 16 tests, and four deliberate mutations were all caught. - Cluster named “AIpocalypse Cluster”
- Vite no longer watches
baseline-audit/or its own log — that loop was reloading the page ~1×/sec during telemetry capture (10 full reloads in 18s), which looked like data loss.
Test coverage added
Upstream’s suite was 80 tests at fork point. Yours adds 15 new test files covering ssh child env, ssh batching, ssh fallback, host resolution, node liveness, snapshot stability, cluster model arithmetic, RDMA parsing, prefill rate, sshShell boundary, and four Showcase areas including a real-socket HTTP body test.
