I’ve been working on sparkview, a Python TUI monitor that accounts for coherent unified memory behavior on GB10 / DGX Spark systems.
Problem:
On GB10, nvmlDeviceGetMemoryInfo can return total ≈ MemTotal (~121 GB). That value does not reflect allocatable memory. In practice, usable capacity tracks MemAvailable, which accounts for kernel reservations and page cache pressure.
sparkview detects this condition at runtime and switches to MemAvailable for memory display.
Signals:
-
Memory display adjusted for UMA behavior (MemAvailable vs MemTotal)
-
PSI memory pressure (/proc/pressure/memory) — LOW / MOD / HIGH / CRITICAL
-
Load-gated clock states — IDLE / PASS / LOCKED / THROTTLED
-
ConnectX-7 throughput via sysfs — surfaces degraded links
-
Process list sorted by GPU memory
-
Automatic runtime detection — no configuration
Repo: https://github.com/parallelArchitect/sparkview
Requires validation on GB10 / DGX Spark hardware. If you run it on Spark, feedback is useful.
Can you include a Screenshot to showcase how it looks? Thanks
Not bad, I will give it a run.
Install (fixed)
git clone https://github.com/parallelArchitect/sparkview.git
cd sparkview
# create a virtual environment (recommended on DGX Spark)
python3 -m venv sparkview-venv
# activate it
source ~/sparkview/sparkview-venv/bin/activate
# install dependencies
pip install nvitop psutil rich textual
Run
~/sparkview.sh
#!/bin/bash
cd ~/sparkview
source sparkview-venv/bin/activate
python3 main.py
deactivate
Nice tool.
How I run it is you have uv installed:
uvx nvitop
@whpthomas appreciated — README updated with venv steps based on your feedback.
Quick note: sparkview runs as python3 main.py, not nvitop directly. The venv activates dependencies but the entry point is the script:
source ~/sparkview/sparkview-venv/bin/activate
python3 ~/sparkview/main.py
For persistent one-command launch, add the alias once:
echo "alias sparkview='source ~/sparkview/sparkview-venv/bin/activate && python3 ~/sparkview/main.py'" >> ~/.bashrc
source ~/.bashrc
Then just type sparkview from terminal.
Terminal height scaling — pushed in latest commit.
Thanks to @elsaco for the GB10 screenshot.
@whpthomas @bernardlbmi3 one thing to flag — the nvitop screenshots appear to be using MemTotal as the denominator on GB10 rather than allocatable memory.
On coherent UMA systems, MemTotal does not reflect usable capacity. MemAvailable is a closer approximation since it accounts for kernel reservations and page cache.
That would explain the ~92.7% utilization shown — it may be overstating actual pressure rather than indicating a real memory constraint.
There’s an open PR addressing this behavior: Fix incorrect memory reporting on coherent UMA platforms (GB10 / DGX … by parallelArchitect · Pull Request #208 · XuehaiPan/nvitop · GitHub
Thank you - that makes a lot more sense now. I am a low level C systems programmer since the 80’s who migrated to rust more recently. Python ¯_(ツ)_/¯
A couple of wish list items that would help this be an ‘at a glance’ utility.
Color – turn from green / orange / red to highlight high loads intelegently. Obviously high memory use isn’t a problem unless we are swapping a lot, then it is.
Temp – have a single graph that shows the highest temp out of GPU or CPU (which would be in text right) max at100C , use colors to warn
If you added these, this would be really useful.
The feedback here has directly shaped the tool — venv setup, terminal scaling, and memory denominator clarification all came from this thread. Real-world Spark feedback is invaluable when building against hardware that isn’t directly accessible.
@whpthomas the Python is just the display layer — the underlying signals come from NVML and kernel-exposed UMA state. Open to a C/Rust perspective on the collection path if you want to dig in.
From your screenshot — CLOCK IDLE, P0 state, GPU 0%, CPU 0.0% — the tool correctly did not flag THROTTLED. VLLM::EngineCore had 91.9 GiB resident in unified memory, but the system was idle. The load gate prevented a false alarm — memory was occupied, not under pressure.
Both wishlist items are in v0.2.0, just pushed:
-
TEMP row — current and session peak for GPU and CPU, color-coded green / yellow / red at 60°C and 80°C thresholds
-
Smart color — ⚡UMA turns red when PSI HIGH, clock THROTTLED, or temp > 80°C
-
Anomaly auto-logger — when an issue is detected, sparkview writes a timestamped log to ~/sparkview_logs/ with a summary.json (trigger reason, peak temps, driver, CUDA, kernel). Logs are compressed on exit. Leaves a trace for post-incident inspection — no user action required.
Thanks for all the support and the real GB10 data — it directly improved the tool.
So how do I interpret this? I am thinking maybe I need to lower my vllm memory use?
GPU ░░░░░░░░░░░░░░░░░░░░ 1% 44°C 12.6W Mem 114.0Gi/121.7Gi ⚡UMA
MEM ██████████████████░░ 93.7% Used 114.0Gi / 121.7Gi
SWAP ░░░░░░░░░░░░░░░░░░░░ 3.6% Used 1.2Gi / 32.0Gi
CPU ░░░░░░░░░░░░░░░░░░░░ 1.5% Max 10% Active 3/20
CLOCK ░░░░░░░░░░░░░░░░░░░░ IDLE 2411MHz / 3003MHz P0
UMA ████████████████████ CRITICAL some 0.81 full 0.81
TEMP █████████░░░░░░░░░░░ GPU 44°C↑44°C CPU 46°C↑46°C
INFO 05:08:01 PM | Driver 580.142 | CUDA 13.0 | Kernel 6.17.0-1014-nvidia | Up 4d 22h
3m
────────────────────────────────────────────────────────────
PROC PID USER GPU-MEM CPU% CMD
3177363 root 100.4Gi 0% VLLM::EngineCore
4393 whpthomas 0.1Gi 0% gnome-shell
4242 whpthomas 0.1Gi 0% Xorg
2756690 whpthomas 0.0Gi 0% gnome-system-monitor
Update
I reduced the gpu_memory_utilization from 0.82 to 0.8 and that seem to have sorted it out. Very Handy! - nothing else was picking that up
Glad it was useful — this is exactly the signal the tool is designed to surface.
No active GPU compute was occurring. The system was stalled in memory management — page reclaim and swap activity competing against ~114 GiB of resident model data. PSI shows ~81% of wall time spent in stalled states.
This aligns with the pre-failure condition described here:
https://forums.developer.nvidia.com/t/dgx-spark-becomes-unresponsive-zombie-instead-of-throwing-cuda-oom/353752
Those reports do not include pre-crash telemetry; PSI provides visibility into the transition.
v0.2.1
- Adds IO PSI alongside memory PSI
On GB10-class systems running VLLM-style workloads, IO pressure is a second important signal — model shard loads and checkpoint writes directly compete with the unified memory pool and can accelerate stall conditions.
I extended this a little with the extra data available from here: https://github.com/antheas/spark_hwmon
Looks like I’m actively experiencing the PD-Bug…
@twaggs88 nice work — that’s solid capture of the failure state.
611 MHz at P0 with ~96% GPU load, PROCHOT ACTIVE, and PL_LEVEL 1 indicates the GPU is operating under a constrained power condition. In this state, clocks are reduced to stay within the available power budget rather than scaling with load.
DC input at ~38.8 W under sustained GPU load is notably low. For context, GB10 is a 140 W-class part, and NVIDIA specifies a 240 W supply for normal operation:
https://docs.nvidia.com/dgx/dgx-spark/hardware.html
This suggests the system is power-limited, though it does not isolate whether the constraint is due to supply, cable, or negotiation.
Worth confirming the original PSU and cable are in use. If the supply is correct, a full power cycle may help — disconnect the brick from both the wall and the system, wait ~60 seconds, then reconnect to force a fresh PD negotiation.
The anomaly logger should have captured the full timeline under ~/sparkview_logs/.
If possible, share the summary.json from that run — it will show the trigger condition, clock behavior, power draw, and throttle state leading into the event.
Also interested in the spark_hwmon extension — exposing rail-level power data adds useful visibility for this class of issue.
Yeah, I was using the OEM supply. This particular one is the ASUS GX10, but it’s my understanding this PD bug is present in all variants. It would probably be worth running a monitoring utility as a daemon that can alert the user if it detects this condition.
Love this tool! Love monitoring like this.. now Question :)
Is that really critical? (IO) it goes high and critical but I don’t know what it translates to ;-) Is there anything I should change to lower that? am I loosing performance?
All input is appreciated! Thanks!
-Aldo
nice tool! Can you also show the max value of power consumption? Thank you!
From your screenshot — ● LOGGING was active while THROTTLED fired, so the anomaly logger captured the full event automatically.
Check ~/sparkview_logs/ — the summary.json will show the CLOCK_THROTTLED trigger, start time, duration, and peak temps. The anomaly.log.gz has snapshots every 2 seconds showing clock at 611MHz, GPU load, memory, and PROC list throughout the throttle event — a complete timeline of how long the system was in that state.
Can you share the summary.json from that session? Were the spark_hwmon power rows wired into the logger or just the display? That would help determine whether the logger captured PROCHOT state and DC input — and whether those signals should be pulled into the main tool for future runs.
IO CRITICAL with CLOCK PASS means the GPU is healthy but the system is spending 52% of elapsed time stalled on disk IO. With MEM at 95.5% and SWAP active that’s real pressure — the kernel is moving pages between memory and swap while trying to serve IO requests at the same time.
On GB10 this is worth paying attention to. The NVLink-C2C interconnect runs at 900 GB/s — KV cache that overflows GPU memory should spill into the shared LPDDR5X unified memory pool, not disk. If IO PSI is hitting CRITICAL it suggests the unified pool is full and the system is reaching the SSD. On discrete GPU that’s expected. On GB10 it means the model is overcommitting the entire 128GB pool.
https://developer.nvidia.com/blog/accelerate-large-scale-llm-inference-and-kv-cache-offload-with-cpu-gpu-memory-sharing/
Reducing gpu_memory_utilization in VLLM gives the unified memory pool more headroom — same fix whpthomas applied above, dropped from 0.82 to 0.8 and it cleared.
For larger scale KV cache management: https://developer.nvidia.com/blog/how-to-reduce-kv-cache-bottlenecks-with-nvidia-dynamo/
The anomaly logger in v0.2.1 captured this automatically — check ~/sparkview_logs/ for the summary.json from that session.
max power is on the roadmap — the spark_hwmon driver exposes full power rail telemetry including peak draw, limits, and PROCHOT state. Once wired into the main tool, that data will be captured and reported alongside the existing signals.
For now the GPU row shows instantaneous power (W) next to temperature.
Hardware access comes from the community — screenshots and field data from GB10 users continue to inform and validate the tool.
