I am experiencing a severe, sudden performance drop on my NVIDIA DGX Spark system where the integrated Blackwell GPU gets trapped in an emergency low-power state. Even under 99% compute utilization workloads, the card refuses to scale past 15W and locks its graphics frequencies to 650 MHz.
The issue is accompanied by a massive accumulation of microseconds in the SW Power Capping log counters and an artificially lowered GPU T.Limit Temp threshold pinned exactly at 50°C (despite the chip running cool at 44°C).
I have attached a Grafana telemetry snapshot showing the sudden power cliff profile, followed by the relevant nvidia-smi diagnostic logs below.
System Environment:
-
Hardware: NVIDIA DGX Spark (Integrated GB10 Grace Blackwell SoC)
-
Driver Version: 580.142
-
CUDA Version: 13.0
Diagnostic Logs (nvidia-smi Output):
1. Clock Throttling Evidence (Locked at 650 MHz instead of 2418 MHz default):
Timestamp : Sat May 16 02:28:57 2026
Driver Version : 580.142
CUDA Version : 13.0
Attached GPUs : 1
GPU 0000000F:01:00.0
Clocks
Graphics : 650 MHz
SM : 650 MHz
Memory : N/A
Video : 598 MHz
Applications Clocks
Graphics : 2418 MHz
Memory : N/A
Default Applications Clocks
Graphics : 2418 MHz
Memory : N/A
Deferred Clocks
Memory : N/A
Max Clocks
Graphics : 3003 MHz
SM : 3003 MHz
Memory : N/A
Video : 3003 MHz
Max Customer Boost Clocks
Graphics : N/A
SM Clock Samples
Duration : N/A
Number of Samples : N/A
Max : N/A
Min : N/A
Avg : N/A
Memory Clock Samples
Duration : N/A
Number of Samples : N/A
Max : N/A
Min : N/A
Avg : N/A
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
2. Temperature Lock (Artificial 50°C T.Limit):
Timestamp : Sat May 16 02:29:48 2026
Driver Version : 580.142
CUDA Version : 13.0
Attached GPUs : 1
GPU 0000000F:01:00.0
Temperature
GPU Current Temp : 44 C
GPU T.Limit Temp : 50 C
GPU Shutdown T.Limit Temp : N/A
GPU Slowdown T.Limit Temp : N/A
GPU Max Operating T.Limit Temp : 0 C
GPU Target Temperature : N/A
Memory Current Temp : N/A
Memory Max Operating T.Limit Temp : N/A
3. Performance Counters (1,369+ Seconds of Active SW Power Capping Loop):
Timestamp : Sat May 16 02:31:03 2026
Driver Version : 580.142
CUDA Version : 13.0
Attached GPUs : 1
GPU 0000000F:01:00.0
Performance State : P0
Clocks Event Reasons
Idle : Not Active
Applications Clocks Setting : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
HW Thermal Slowdown : Not Active
HW Power Brake Slowdown : Not Active
Sync Boost : Not Active
SW Thermal Slowdown : Not Active
Display Clock Setting : Not Active
Clocks Event Reasons Counters
SW Power Capping : 1369386233 us
Sync Boost : 0 us
SW Thermal Slowdown : 0 us
HW Thermal Slowdown : 0 us
HW Power Braking : 0 us
Sparse Operation Mode : N/A
4. Power Readings (Hard-capped at ~15W):
Timestamp : Sat May 16 02:28:39 2026
Driver Version : 580.142
CUDA Version : 13.0
Attached GPUs : 1
GPU 0000000F:01:00.0
GPU Power Readings
Average Power Draw : 14.11 W
Instantaneous Power Draw : 15.36 W
Current Power Limit : N/A
Requested Power Limit : N/A
Default Power Limit : N/A
Min Power Limit : N/A
Max Power Limit : N/A
Power Samples
Duration : Not Found
Number of Samples : Not Found
Max : Not Found
Min : Not Found
Avg : Not Found
GPU Memory Power Readings
Average Power Draw : N/A
Instantaneous Power Draw : N/A
Module Power Readings
Average Power Draw : N/A
Instantaneous Power Draw : N/A
Current Power Limit : N/A
Requested Power Limit : N/A
Default Power Limit : N/A
Min Power Limit : N/A
Max Power Limit : N/A
What I Have Tried:
-
Verified that the device is connected strictly via the official heavy-duty NVIDIA power brick directly to a reliable wall source to rule out power rail voltage sags. Validated socket.
-
nvidia-powerd.serviceis not present/loaded on this specific server distribution setup.
Is this an emerging bug in Driver 580.142/CUDA 13 for the Blackwell PMIC architecture, or is there a low-level hardware firmware lock/register reset mechanism I need to trigger via the DGX container stack to release this safety state?
Any guidance from the engineering team would be highly appreciated.