Interesting, thanks for the reply and the education. I’m still learning the ins-and-out of this so I really appreciate it.
Last night after some googling I supposed that the memory pressure had something to do so I’ve reduced a few points on my --gpu-memory… parameter. I left a few jobs running last night and this morning while idling the results are the following:
With again IO hovering from high to critical. But SWAP now is not being utilized and overall memory utilization is a bit lower which I like. I don’t see almost any disk utilization… I wonder if I’m looking at the wrong thing but btop seems to confirm that there’s no IO, much less on idle
42 │ --- 2026-04-19 12:26:46 ---
43 │ GPU: 0% 35°C (peak 35°C) 4.2W Mem 2.3Gi/121.7Gi
44 │ MEM: 1.9% Used 2.3Gi / 121.7Gi
45 │ SWAP: 0.0% Used 0.0Gi / 16.0Gi
46 │ CPU: 0.0% Active 0/20 37°C (peak 38°C)
47 │ CLOCK: IDLE 208MHz / 3003MHz P8
48 │ PSI: LOW some 0.00 full 0.00
49 │ IO: HIGH some 0.06 full 0.06
50 │
51 │ --- 2026-04-19 12:26:48 ---
52 │ GPU: 0% 35°C (peak 35°C) 4.3W Mem 2.3Gi/121.7Gi
53 │ MEM: 1.9% Used 2.3Gi / 121.7Gi
54 │ SWAP: 0.0% Used 0.0Gi / 16.0Gi
55 │ CPU: 0.0% Active 0/20 37°C (peak 38°C)
56 │ CLOCK: IDLE 208MHz / 3003MHz P8
57 │ PSI: LOW some 0.00 full 0.00
58 │ IO: CRITICAL some 0.23 full 0.23
59 │
Notice the IO readings!
In layers/pressure.py you define:
25 │ s = result["some_avg10"]
26 │ f = result["full_avg10"]
27 │ if f > 0.10 or s > 0.30:
28 │ result["level"] = "CRITICAL"
29 │ elif f > 0.05 or s > 0.15:
30 │ result["level"] = "HIGH"
31 │ elif s > 0.05:
32 │ result["level"] = "MOD"
33 │ else:
34 │ result["level"] = "LOW"
You’re using only avg10 reading and ignore avg60 and avg300, so only the short-term spikes are noticed. However, more time might be a better reading to evaluate if the system is struggling.
Good progress — SWAP clearing confirms the gpu_memory_utilization
reduction helped with the memory pressure.
The IO CRITICAL signal at idle is a separate condition. With GPU at
0%, this points to background I/O activity rather than active compute.
Try expanding the terminal or scrolling down to the PROC section —
something may still be running from an earlier workload.
Can you share a screenshot with the PROC section visible?
Also worth checking:
mount | grep -E “nfs|cifs|smb”
If model weights or datasets are on a network mount,
/proc/pressure/io will reflect those stalls as well.
v0.2.2 adds IO anomaly logging — IO CRITICAL events are captured
automatically to ~/sparkview_logs/ with a summary.json showing
trigger and full snapshot timeline. That should help isolate the
source of the stall.
Those thresholds were tuned on discrete GPU systems — not GB10. The IO HIGH/CRITICAL signal at idle with ~1.9% memory is exactly the kind of data needed to recalibrate for coherent UMA.
To establish a baseline, a small collector tool is now in the repo:
The htop screenshot confirms it — no process-level disk IO, but
IO PSI elevated. The pressure is not coming from user-space reads
or writes.
The difference is what each tool measures:
htop I/O tab → per-process bytes to block devices
sparkview (/proc/pressure/io) → kernel time stalled on IO,
including memory management
With 96.6Gi resident in unified memory and no active process IO,
this signal is coming from below the process layer.
The current thresholds flag this as CRITICAL, but with your data
it’s clear this is not a fault condition — it’s baseline GB10
behavior with VLLM loaded.
Since you’re on v0.2.2, open a new terminal in your sparkview
folder and run two collections — first with VLLM stopped:
Each runs for 2 minutes and prints a summary at the end. Paste
both summaries here — that gives the GB10 floor and the loaded
baseline, and the thresholds in pressure.py can be recalibrated
from real hardware data.
On GB10 with spark_hwmon installed, the PWR row now shows:
PWR GPU 11W peak 47W DC 38.8W Cap 30W ● PROCHOT PL1 Tj+4.8°C
(example values)
Instantaneous GPU draw, session peak, DC input, power cap,
PROCHOT state, active PL level, and thermal rise — all on
one line, updated every refresh cycle.
spark_hwmon install instructions are in the README.
Thanks for the baseline data — that’s exactly what was needed.
If you’re set up, one additional measurement would help. A companion tool measures UMA fault latency using PTX clock instrumentation (no CUPTI dependency).
mudo@gx10-a1db:~/nvidia-uma-fault-probe$ nvcc -O2 -std=c++17 probe_launcher.cu -o uma_probe -lcudart -lcuda -lpthread probe_launcher.cu(189): error: no suitable constructor exists to convert from “int” to “cudaMemLocation”
do { cudaError_t e = (cudaMemPrefetchAsync(data, n*sizeof(float), device, 0)); if (e != cudaSuccess) { fprintf(
^
probe_launcher.cu(195): error: no suitable constructor exists to convert from “int” to “cudaMemLocation”
do { cudaError_t e = (cudaMemPrefetchAsync(data + n/2, (n/2)*sizeof(float), device, 0)); if (e != cudaSuccess) { fprintf(
^
2 errors detected in the compilation of “probe_launcher.cu”.
Sorry, I DMd you to avoid clutter here, you might have missed it. mudo@gx10-a1db:~/nvidia-uma-fault-probe$ nvcc -O2 -std=
c++17 probe_launcher.cu -o uma_probe -lcudart -lcuda -l
pthread probe_launcher.cu(189): error: no suitable constructor
exists to convert from “int” to “cudaMemLocation”
do { cudaError_t e = (cudaMemPrefetchAsync(da
ta, n*sizeof(float), device, 0x00)); if (e != cudaSucce
ss) { fprintf(
^
probe_launcher.cu(195): error: no suitable constructor
exists to convert from “int” to “cudaMemLocation”
do { cudaError_t e = (cudaMemPrefetchAsync(da
ta + n/2, (n/2)*sizeof(float), device, 0x00)); if (e !=
cudaSuccess) { fprintf(
^
2 errors detected in the compilation of “probe_launcher
.cu”.
v0.2.3 pushed — fixes the IO CRITICAL false alarm reported
in GitHub issue #3.
On GB10 with VLLM loaded but idle, the IO row always showed
CRITICAL because VLLM worker processes block waiting for
inference requests, which inflates the PSI IO metric. IO row
removed entirely.
Full writeup and GB10 baseline data now published:
Credit to @azampatti for running every iteration — patient and
committed from start to finish. Without his persistence on real
hardware none of this data would exist.
For tool documentation, build instructions, and example output:
Thanks to the entire community for the support, sosreports,
field data, bug logs and feedback. It would not be possible without it.
