Good approach for pre-launch detection. For continuous in-flight monitoring — a community member hit the PD failure mode yesterday mid-inference and captured it live: CLOCK THROTTLED at 611 MHz with 96% GPU load.
sparkview is a GB10-aware TUI monitor that watches these signals continuously — clock state, PSI memory pressure, IO pressure, and thermals — and logs anomalies automatically. When CLOCK_THROTTLED triggers it writes a timestamped summary.json with the trigger, duration, clock behavior, and system state to ~/sparkview_logs/.
https://github.com/parallelArchitect/sparkview
Community discussion and field captures including the throttle event: https://forums.developer.nvidia.com/t/sparkview-gpu-monitor-tool-with-gb10-aware-unified-memory-handling/366877
mashie’s SNMP PDU idea is exactly the right automation layer. Your smoke test + sparkview + SNMP PDU closes the full loop:
-
Pre-launch check — validates power delivery before workload
-
Runtime monitor — detects throttle conditions during inference
-
Anomaly logger — records
summary.jsonwith trigger and context -
Watch script — monitors
~/sparkview_logs/and triggers action -
SNMP PDU — cycles power, forces fresh PD negotiation
-
Recovery — system returns at expected clocks
No manual intervention required.