I’m running out of ideas. Recently I noted that the GB10 clock goes down from over 2000MHz to 513MHz under load. I’m not sure if that is the culprit for performance degradation, measured 3x. All installed versions seemed to be correct. Range of tests to set the GPU under load should exclude any software issue. After my last factory reset on 1/1/26, I only used DGX dashboard for updating. I cannot think of anything messing this up by myself, but who knows. The only pointer I can find is from dmesg: “Detected insufficient power on the PCIe slot (27W)”.
I would love to hear thoughts or additional checks I can apply.
Here some data from what I have checked already:
Kernel: 6.17.0-1008-nvidia
Driver: 580.126.09
CUDA Toolkit: 13.0.2
nvcc --version: CUDA 13.0 (V13.0.88)
cudaDriverGetVersion = 13000
cudaRuntimeGetVersion = 13000
Confirmed that there is no other CUDA version installed
Confirm that there is only one driver installed
No thermal throttling observed, 513MHz is immediately with load
Tested with cudaTensorCoreGemm, gpu-burn, llama-bench.
Interestingly when I run SentenceTransformer(args.model, device=“cpu”) it goes down to 890 MHz, 4.59 W, it will release almost immediately (2132 MHz, 7.00 W) when processing is done. Command: watch -n 0.5 ‘nvidia-smi --query-gpu=clocks.gr,power.draw --format=csv,noheader’. GPU under load ends up with 513 MHz, 13.35 W.
Running dmesg for longer during tests I can see that it is not always at 513MHz
Command: sudo dmesg -T | grep -i -E “power|brake|throttle|clock|thermal|pcie|mlx” | tail -n 200
timestamp, utilization.gpu [%], clocks.current.graphics [MHz], clocks.current.sm [MHz], power.draw [W], temperature.gpu, pstate
2026/02/21 02:04:14.950, 0 %, 2106 MHz, 2106 MHz, 7.12 W, 39, P0
2026/02/21 02:04:17.756, 1 %, 689 MHz, 689 MHz, 4.41 W, 39, P0
2026/02/21 02:04:18.959, 19 %, 513 MHz, 513 MHz, 4.39 W, 39, P0
2026/02/21 02:04:19.159, 0 %, 513 MHz, 513 MHz, 4.24 W, 39, P0
2026/02/21 02:04:21.162, 0 %, 1287 MHz, 1287 MHz, 4.65 W, 39, P0
SMBIOS 3.3.0
BIOS Revision: 5.36
No Clock Events Reasons from NVSMI log, no active event during clock cap
GPU 0000000F:01:00.0
Performance State : P0
Clocks Event Reasons
Idle : Not Active
Applications Clocks Setting : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
HW Thermal Slowdown : Not Active
HW Power Brake Slowdown : Not Active
Sync Boost : Not Active
SW Thermal Slowdown : Not Active
Display Clock Setting : Not Active
Clocks Event Reasons Counters
SW Power Capping : 0 us
Sync Boost : 0 us
SW Thermal Slowdown : 0 us
HW Thermal Slowdown : 0 us
HW Power Braking : 0 us
Sparse Operation Mode : N/A
Hi @cho, The test with SentenceTransformer for CPU only was just to isolate if the issue is dedicated to GPU only. It seems not, but cannot say for sure. All the other mentioned tools are using GPU and the models I have tested are bringing the GPU to 96%, but then clock and wattage goes down.
@christian.pappert the insufficient power messages are from the Mellanox network driver. The log entries look like this:
mlx5_core 0000:01:00.0: mlx5_pcie_event:326:(pid 12): Detected insufficient power on the PCIe slot (27W)
mlx5_core 0000:01:00.1: mlx5_pcie_event:326:(pid 359): Detected insufficient power on the PCIe slot (27W)
So the GPU is not related to the 27W power warning and safe to ignore them.
Inside the DGX Spark the CPU and GPU are connected via a NVLink C2C path and not a PCIe bus.
I had almost exactly the same thing, the GPU dropped to around 507-513 MHz right when I put it under load, the power remained very low (~13-14W), and everything ran way slower (like 3x worse).It was driving me nuts, but here’s what finally got me back to normal clocks and full speed:First, watch what’s happening live with 1-second updates instead of snapshots (they miss the drop): Just run this in a terminal while your test is going:
while true; do
date '+%F %T' | tr -d '\n'; echo -n ',';
nvidia-smi --query-gpu=temperature.gpu,utilization.gpu,power.draw,clocks.current.graphics,pstate,clocks_throttle_reasons.active --format=csv,noheader,nounits;
sleep 1;
done
This image captured the exact moment it failed and showed no signs of overclocking. The temperature was low (approximately 40-50°C) when it happened, so it definitely wasn’t due to heat.
Tried stopping/restarting NVIDIA services and reloading drivers, but not fully. What actually fixed it for me: full power cycle. Turn everything off, unplug the power cord for a few minutes, plug it back in, turn it on, and run the test again. The clocks returned to normal and power consumption returned to healthy levels.
@christian.pappert a149952’s power cycle fix is the right call — this is the USB PD negotiation issue that hoesing documented in the GPU PD Throttle Check Tool thread. The all-clear throttle bitmask while clocks are capped at 513 MHz is the signature — the platform is limiting power delivery below the level where NVML would report a throttle reason.
I’ve been working on an enhanced fork that adds NVML direct telemetry, throttle reason decoding, clock ramp-up timing, stability scoring, and baseline comparison:
# Capture the throttled state before power cycle
python3 spark-gpu-throttle-check.py --timeline -n 40 --save-baseline
# Power cycle the brick (unplug from wall, wait 60s, reconnect)
# Capture the recovered state and compare
python3 spark-gpu-throttle-check.py --timeline -n 40 --compare
The --compare output will show you the clock, power, and thermal delta between the two runs.
One thing I’ve started to notice is that the most consistent way for me to get this on my FE Spark is to leave it off for a while, but plugged in. When I power it on after a day or so turned off, it turns on in a degraded state. Could be a fluke, but it doesn’t seem load dependent for me. But this is also really hard to nail down since it’s so inconsistent.
Same signature here, 507 MHz, DGX Spark reference unit. Adding register numbers, since I don’t see them in this thread.
Under a saturating bf16 matmul: 507 MHz, board power flat at 35 W, 9.4 TFLOPS. nvidia-smi showed no throttle reason at all and utilization.gpu read 96% the whole time.
The cap is readable. In the spbm hwmon node, pl1 was 20 W and syspl1 30 W, against a 250/300 W hardware max. After a reboot: 140 W and 231 W, same burn 98.6 TFLOPS. 10.5x.
H=$(for f in /sys/class/hwmon/hwmon*/; do [ "$(cat $f/name)" = spbm ] && echo $f; done)cat $H/power11_cap $H/power13_cap # want ~140000000 / ~231000000
syspl1 = 30 W looks like the same state as the “30 W safety mode” in the ASUS GX10 thread, with a value you can read instead of infer.
One thing does not fit: GX10_PD_Throttle_Fix says a warm reboot cannot clear this, because the brick keeps the PD MCU powered. A plain sudo reboot cleared it here. Has anyone else seen that?
(spbm is reverse-engineered from antheas/spark_hwmon, so the labels are the author’s reading, not spec)