Another Asus GX10 Problem

Its been two weeks of non-stop problems with the Asus GX10 and now I am just probably sending it back to the store. One last attempt… I am seeing serious periodic slow downs every 30 minutes or so that last for 5-10 minutes and then performance returns. I have the latest patches and firmware updates. I do not have anything else plugged into the system. With AI assistance, I created a short monitoring process and it found the below. Is there any known solution to this problem (seems somewhat related to the 30w cap bug, but its intermittent).


Issue summary: ASUS Ascent GX10 / NVIDIA GB10 intermittent GPU low-power state during inference

System: ASUS Ascent GX10, NVIDIA GB10 platform
Workload: Local LLM inference / token generation
Symptom: Approximately every ~30 minutes, token generation slows dramatically, down to roughly 0.1 tokens/sec. The slow period lasts from tens of seconds to several minutes, then performance recovers without reboot.

This does not appear to be normal thermal throttling or a user-space inference/runtime issue. During the slow windows, the GPU remains heavily utilized, but GPU power and graphics clocks collapse to a low fixed range.

Primary observed failure signature

During the slow windows, repeated nvidia-smi telemetry shows:

GPU utilization:             ~94-96%
GPU power draw:              ~15-18 W
GPU graphics clock:          ~630-650 MHz
GPU temperature:             ~44-52 C
Performance state:           P0
SW Power Cap:                Not Active
HW Slowdown:                 Not Active
HW Thermal Slowdown:         Not Active
HW Power Brake Slowdown:     Not Active
SW Thermal Slowdown:         Not Active
SW Power Capping:            0 us
HW Thermal Slowdown:         0 us

Representative log sample from 2026-05-24T17:18:34-05:00:

power.draw:                  17.69 W
clocks.current.graphics:     650 MHz
utilization.gpu:             96%
temperature.gpu:             44 C
pstate:                      P0
SW Power Cap:                Not Active
HW Slowdown:                 Not Active
HW Thermal Slowdown:         Not Active
HW Power Brake Slowdown:     Not Active
SW Thermal Slowdown:         Not Active
Average Power Draw:          17.57 W
Graphics current clock:      643 MHz
Max/available graphics refs: 2418 MHz / 3003 MHz

The same sample shows no memory, CPU, I/O, or kernel fault pressure at the time: memory available was ~29 GiB, swap was essentially idle, CPU pressure was 0.00, I/O pressure was 0.00, and recent kernel signals were empty.

Representative repeated slow-window samples

The condition persists across multiple samples in the same window:

2026-05-24 17:18:41
power.draw:              17.56 W
graphics clock:          637 MHz
GPU utilization:         96%
temperature:             44 C
pstate:                  P0
slowdown flags:          Not Active / 0 us

2026-05-24 17:18:47
power.draw:              17.11 W
graphics clock:          630 MHz
GPU utilization:         96%
temperature:             44 C
pstate:                  P0
slowdown flags:          Not Active / 0 us

2026-05-24 17:18:59
power.draw:              17.30 W
graphics clock:          630 MHz
GPU utilization:         96%
temperature:             46 C
pstate:                  P0
slowdown flags:          Not Active / 0 us

Across these samples, system memory remained healthy at ~29 GiB available, swap did not move, CPU pressure stayed 0.00, I/O pressure stayed 0.00, and the log showed no recent kernel signals.

Transition from normal/idle state to low-power trap

Before the slow condition began, the GPU was idle and reporting clocks around 1.8 GHz at low power:

2026-05-24 16:28:43
power.draw:              7.59 W
graphics clock:          1800 MHz
GPU utilization:         0%
temperature:             43 C
pstate:                  P0
slowdown flags:          Not Active

Similar idle samples show ~1.7-1.8 GHz graphics clocks with no throttling flags. Then, when inference load begins, the GPU enters a high-utilization low-power state instead of boosting normally:

2026-05-24 16:29:44
power.draw:              17.33 W
graphics clock:          637 MHz
GPU utilization:         96%
temperature:             44 C
pstate:                  P0
slowdown flags:          Not Active

This suggests the GPU is not idle or CPU-starved. It is busy but constrained to a very low power/clock envelope.

Suspicious thermal-limit telemetry

Some samples show a suspicious GPU T.Limit Temp below the current GPU temperature while all slowdown flags remain inactive. For example:

GPU Current Temp:        52 C
GPU T.Limit Temp:        45 C
SW Power Cap:            Not Active
HW Slowdown:             Not Active
HW Thermal Slowdown:     Not Active
HW Power Brake Slowdown: Not Active
SW Thermal Slowdown:     Not Active
SW Power Capping:        0 us
HW Thermal Slowdown:     0 us
Average Power Draw:      ~17 W

This looks inconsistent: if the thermal limit is truly 45 C and the GPU is at 52 C, a normal thermal slowdown flag would be expected, but none is reported.

What appears ruled out

Based on the captured data, this does not look like:

  1. Normal GPU thermal throttling
    GPU temperature during the slow state is only ~44-52 C, and NVIDIA reports all thermal slowdown counters as inactive / 0 us.

  2. NVIDIA-visible software power cap
    SW Power Cap is reported as Not Active, and SW Power Capping remains 0 us.

  3. Hardware power-brake slowdown
    HW Power Brake Slowdown is reported as Not Active.

  4. System memory pressure / swap pressure
    Representative slow-window sample shows ~29 GiB available memory, minimal swap use, and memory pressure avg10=0.00.

  5. CPU or I/O starvation
    CPU and I/O pressure both show avg10=0.00 in representative slow-window samples, and recent kernel signal logs are empty.

Observed behavior appears to be an intermittent GB10/GX10 platform-level low-power or low-clock latch:

High GPU load + P0 state + low GPU power + low graphics clock + no NVIDIA-reported slowdown reason

This seems related to, but not identical to, the known GX10 “30 W cap” reports. In this case the issue is transient and the GPU appears to be constrained around 15-18 W / ~630-650 MHz, then later recovers. The logs suggest the limiter may be outside normal nvidia-smi throttle accounting, possibly in platform firmware, EC/PD power policy, or a GB10 driver/firmware reporting path.

GX10 / GB10 platform firmware, EC, PD controller, or NVIDIA firmware can enter a transient low-power state (?) where:

GPU utilization remains ~95-96%
GPU stays in P0
GPU power is limited to ~15-18 W
graphics clock is limited to ~630-650 MHz
all NVIDIA-visible throttle / slowdown flags remain inactive

Meaning of GPU T.Limit Temp on this platform, especially cases where GPU Current Temp exceeds GPU T.Limit Temp while no thermal slowdown flag is active?

I also own an Asus GX10 and have experienced a similar issue before. First, please use the tool at the link below to check your current status:

Under normal operating conditions, high-load tasks—such as extended inference or training—should result in very hot air being exhausted from the rear. The exhaust should feel intense enough that you wouldn’t want to keep your hand near it for long. Additionally, you should notice a significant and clearly audible increase in fan noise.

Since updating to NVIDIA 580.159.03, I’ve been experiencing the same problem from time to time:

After unplugging the power supply for 3 minutes, everything is working fine again. Let’s see how long it lasts.

I feel your pain, something similar but not same on my GX10 : clock pinned low, power floor, throttle flags clean. We use a small preflight tool before every training run (similar to what @Mkei88 pointed out ) runs a short bf16 GEMM load and checks clock/power against thresholds) so we don’t kick off a multi-hour run on a wedged GPU. We just expanded our tool with a --watchdog mode that passively samples during the run, going to start testing soon. Your data has one extra fingerprint mine didn’t: T.Limit 45°C below actual die temp. That’s a platform-side ceiling drop, not a GPU-side throttle maybe why standard flags stay clean.
from my very limited understanding:

1) GPU-side throttle -> the GPU itself decides "I'm too hot
2) The board (not the GPU) lowers the cap the GPU is allowed to run at

couple things I would try, both cheap and might get you the answer without going deep:

1. Plug-through watt meter (~$15 on Amazon). Wall → meter → GX10 brick → GX10. Watch the wall watts at the moment the GPU clock drops.

  • Watts drop too-> power-supply side (brick or wall).
  • Watts stay flat → board side (something on the GX10 is capping the GPU).

2. nvpmodel -q during a slow episode. GB10 inherits Jetson’s power policy stack, and some Jetson modes have a built-in sustained-load cap that releases on a timer potentially, looks identical to a fault but is possibly firmware-intentional. If the snapshot shows a mode with a sustained cap matching your ~30 min cycle, the fix is maybe a config change, not hardware.


Going to run our --watchdog against some long inference sessions on our side over the next few days and see if we can reproduce the episodic version. Will report back here either way… And guess what–> One pre-flight just caught a throttle… have to cold flush …
( initial was stuck on 16W, cold reset back to 52w )

cold fix: probably not as relevant but might help.

Ill post results on the longer tests when done…

Before returning it, can you share sudo dmesg -T and journalctl -b as zip files from a session where the slowdown occurred? The nvidia-smi data points to a platform-level power state issue but the root cause will be in the kernel logs, not the driver API surface.

@parallelArchitect

The problem has come back. I ran dmesg -T and journalctl -b, and I’ve attached the results in a file named diag.zip.

diag.zip (87.5 KB)

Thanks for the logs. Both files reviewed. @Lutti

The journal shows a clean boot on 580.159.03 with no NVRM errors and no kernel-visible throttle events during this session. The 930MHz / 17W state isn’t generating any signal in the nvidia driver or kernel layer — the limiter is below the nvidia-smi throttle accounting surface.

The BERT table at boot contains a corrected hardware error record from the previous session — vendor MTKID, unknown section type, with PD controller event bytes at offset 0x20. The platform EC entered a reduced power envelope following a PD event in the prior session and did not restore full power policy on reboot.

Full cold power cycle — cable out, 30 seconds, reconnect — should clear the EC state and restore normal clocks. Soft reboot will not. Try it and let me know if that restores normal clocks.

@Lutti did you force the nvidia driver upgrade? It’s on hold, and the current driver is 580.142.x

It does show in the apt list –upgradable, however when running apt ugrade you get The following packages have kept back warning.

I think the kept-back warning is more a staged deployment, Im on 159.03 and have not performed any risky business to get on it.

@elsaco this was included in this regular update :

Start-Date: 2026-05-22  10:45:01
Commandline: apt dist-upgrade
Requested-By: admin (1000)
Install: nvidia-firmware-580-580.159.03:arm64 (580.159.03-0ubuntu0.24.04.1, automatic)
Upgrade: openvpn:arm64 (2.6.19-0ubuntu0.24.04.1, 2.6.19-0ubuntu0.24.04.2), libnvidia-common-580:arm64 (580.142-0ubuntu0.24.04.1, 580.159.03-0ubuntu0.24.04.1), libnvidia-fbc1-580:arm64 (580.142-0ubuntu0.24.04.1, 580.159.03-0ubuntu0.24.04.1), bind9-host:arm64 (1:9.18.39-0ubuntu0.24.04.3, 1:9.18.39-0ubuntu0.24.04.5), libnvidia-gl-580:arm64 (580.142-0ubuntu0.24.04.1, 580.159.03-0ubuntu0.24.04.1), libnvidia-extra-580:arm64 (580.142-0ubuntu0.24.04.1, 580.159.03-0ubuntu0.24.04.1), nvidia-compute-utils-580:arm64 (580.142-0ubuntu0.24.04.1, 580.159.03-0ubuntu0.24.04.1), linux-modules-nvidia-580-open-6.17.0-1018-nvidia:arm64 (6.17.0-1018.18, 6.17.0-1018.18+1), linux-modules-nvidia-580-open-nvidia-hwe-24.04:arm64 (6.17.0-1018.18, 6.17.0-1018.18+1), libnvidia-container1:arm64 (1.19.0-1, 1.19.1-1), libnvidia-encode-580:arm64 (580.142-0ubuntu0.24.04.1, 580.159.03-0ubuntu0.24.04.1), nvidia-utils-580:arm64 (580.142-0ubuntu0.24.04.1, 580.159.03-0ubuntu0.24.04.1), xserver-xorg-video-nvidia-580:arm64 (580.142-0ubuntu0.24.04.1, 580.159.03-0ubuntu0.24.04.1), libnvidia-decode-580:arm64 (580.142-0ubuntu0.24.04.1, 580.159.03-0ubuntu0.24.04.1), libarchive13t64:arm64 (3.7.2-2ubuntu0.6, 3.7.2-2ubuntu0.7), nvidia-kernel-common-580:arm64 (580.142-0ubuntu0.24.04.1, 580.159.03-0ubuntu0.24.04.1), nvidia-kernel-source-580-open:arm64 (580.142-0ubuntu0.24.04.1, 580.159.03-0ubuntu0.24.04.1), bind9-dnsutils:arm64 (1:9.18.39-0ubuntu0.24.04.3, 1:9.18.39-0ubuntu0.24.04.5), snapd:arm64 (2.74.1+ubuntu24.04.4, 2.75.2+ubuntu24.04), rsync:arm64 (3.2.7-1ubuntu1.2, 3.2.7-1ubuntu1.4), libnvidia-container-tools:arm64 (1.19.0-1, 1.19.1-1), bind9-libs:arm64 (1:9.18.39-0ubuntu0.24.04.3, 1:9.18.39-0ubuntu0.24.04.5), nvidia-driver-580-open:arm64 (580.142-0ubuntu0.24.04.1, 580.159.03-0ubuntu0.24.04.1), nvidia-container-toolkit:arm64 (1.19.0-1, 1.19.1-1), libgnutls30t64:arm64 (3.8.3-1.1ubuntu3.5, 3.8.3-1.1ubuntu3.6), libnvidia-cfg1-580:arm64 (580.142-0ubuntu0.24.04.1, 580.159.03-0ubuntu0.24.04.1), nvidia-container-toolkit-base:arm64 (1.19.0-1, 1.19.1-1), libnvidia-compute-580:arm64 (580.142-0ubuntu0.24.04.1, 580.159.03-0ubuntu0.24.04.1), libgnutls-openssl27t64:arm64 (3.8.3-1.1ubuntu3.5, 3.8.3-1.1ubuntu3.6), xdg-desktop-portal:arm64 (1.18.4-1ubuntu2.24.04.1, 1.18.4-1ubuntu2.24.04.2)
Remove: linux-modules-nvidia-580-open-6.17.0-1014-nvidia:arm64 (6.17.0-1014.14+1000)
End-Date: 2026-05-22  10:45:41

@parallelArchitect Everything seems to be back to normal, though I can’t say for sure—but after the last update, I noticed a drop in the response token from about 62 to 57 in qwen3.6:35b.

Start-Date: 2026-05-27  14:15:30
Commandline: apt dist-upgrade
Requested-By: mluttmer (1001)
Upgrade: libxnvctrl0:arm64 (595.71.05-1ubuntu1, 610.43.02-1ubuntu1), python3-samba:arm64 (2:4.19.5+dfsg-4ubuntu9.4, 2:4.19.5+dfsg-4ubuntu9.6), libldb2:arm64 (2:2.8.0+samba4.19.5+dfsg-4ubuntu9.4, 2:2.8.0+samba4.19.5+dfsg-4ubuntu9.6), vim:arm64 (2:9.1.0016-1ubuntu7.13, 2:9.1.0016-1ubuntu7.14), xxd:arm64 (2:9.1.0016-1ubuntu7.13, 2:9.1.0016-1ubuntu7.14), vim-common:arm64 (2:9.1.0016-1ubuntu7.13, 2:9.1.0016-1ubuntu7.14), cuda-toolkit-config-common:arm64 (13.2.75-1, 13.3.29-1), samba:arm64 (2:4.19.5+dfsg-4ubuntu9.4, 2:4.19.5+dfsg-4ubuntu9.6), libwbclient0:arm64 (2:4.19.5+dfsg-4ubuntu9.4, 2:4.19.5+dfsg-4ubuntu9.6), samba-common-bin:arm64 (2:4.19.5+dfsg-4ubuntu9.4, 2:4.19.5+dfsg-4ubuntu9.6), libsmbclient0:arm64 (2:4.19.5+dfsg-4ubuntu9.4, 2:4.19.5+dfsg-4ubuntu9.6), vim-tiny:arm64 (2:9.1.0016-1ubuntu7.13, 2:9.1.0016-1ubuntu7.14), nvidia-settings:arm64 (595.71.05-1ubuntu1, 610.43.02-1ubuntu1), python3-ldb:arm64 (2:2.8.0+samba4.19.5+dfsg-4ubuntu9.4, 2:2.8.0+samba4.19.5+dfsg-4ubuntu9.6), vim-runtime:arm64 (2:9.1.0016-1ubuntu7.13, 2:9.1.0016-1ubuntu7.14), samba-libs:arm64 (2:4.19.5+dfsg-4ubuntu9.4, 2:4.19.5+dfsg-4ubuntu9.6), samba-common:arm64 (2:4.19.5+dfsg-4ubuntu9.4, 2:4.19.5+dfsg-4ubuntu9.6), nvidia-modprobe:arm64 (595.71.05-1ubuntu1, 610.43.02-1ubuntu1), cuda-toolkit-13-config-common:arm64 (13.2.75-1, 13.3.29-1)
End-Date: 2026-05-27  14:15:35

@Lutti

Glad it’s back. The token drop is likely related to the dist-upgrade — cuda-toolkit-config-common, nvidia-modprobe, and nvidia-settings all updated in the same run. A clean reboot should be enough to pick up the software changes if you have not already done one. Hard to isolate which package affected performance without a controlled before/after comparison, but it is worth monitoring over several runs to see whether throughput stabilizes.

While the GPU is healthy, run this now and save a baseline. This is an enhanced fork of hoesing’s spark-gpu-throttle-check with additional diagnostics:

  • Runs a sustained GPU compute workload and measures clock behavior under load

  • Uses direct NVML access instead of nvidia-smi parsing for lower overhead and richer telemetry

  • Identifies active throttle conditions and reports the likely limiting factor: power delivery, thermal limits, or hardware slowdown states

  • Automatically marks results as INCONCLUSIVE if GPU utilization stays below 80% to avoid false-positive throttle reports

  • Tracks thermal trajectory over time to detect developing thermal throttle conditions

  • Reports clock stability across the full runtime window

  • Includes 100ms timeline sampling for transient throttle-event capture

  • Exports full JSON diagnostic reports suitable for regression tracking and bug reports

python3 spark-gpu-throttle-check.py --save-baseline

See the README for additional details:
https://github.com/parallelArchitect/spark-gpu-throttle-check


@parallelArchitect
I found the problem—I can hardly believe it!

I had turned off the graphical interface because I only access the system via SSH.

sudo systemctl disable --now gnome-remote-desktop
sudo systemctl disable --now gdm
sudo systemctl set-default multi-user.target

When I put a load on the graphics card and then checked it with nvtop, I saw a sustained load, which explains the performance drop.

Load:

Idle:

After I turned the graphical interface back on, everything was back to normal.

Load:

Idle:

@Lutti

The GDM change resolving the issue is interesting. The persistent SW_POWER_CAP state at idle without the display manager active is worth documenting.

On GB10, the display stack appears tied into GPU power-state management, so disabling GDM does more than remove the login session — it can leave the GPU operating in a capped power state.

Hi, another GX10 user with energy cap problems. After trying all fixes (firmware update, unplug x minutes…) the fix that worked for me was to unplug the hdmi cable (i dont care, only use ssh…). Since then, tops to ~90W again, and continues doing so for more than a week now.

Add the usb c plugs to your list , hopefully it won’t happen again.( I’m getting used to the protocol lol.)

Unfortunately, the problem has occurred again.

dmesg -T Log :

mlx5_core 0000:01:00.0: mlx5_pcie_event:326:(pid 12): Detected insufficient power on the PCIe slot (27W).

cx7-pcie-hotplug MTKP0001:00: Cable removal
pcieport 0000:00:00.0: PCIe Bus Error: severity=Correctable, type=Physical Layer

pci 000f:01:00.0: DOE: [2c8] ABORT timed out
pci 000f:01:00.0: DOE: [2c8] failed to create mailbox: -5

After disconnecting the power supply for about 60 seconds, everything is working fine again.

Hi, guys! Bought Asus GX10 several days ago and got this power issue two times. The turning off 1-2 minutes save me from it.

Also, sometimes I have an issue with Networking after restart - all network interfaces not available. So I need to restart spark again. Not a big deal, but for $4000 it is too much soft bugs.

USB-C PD chargers can report overheating/other problems to the device they’re powering on. Did you check temperature of the power supply, may be this causes brownout (power supply overheats, reports this through usb data lines and the spark switches to very limited power profile and/or to lower voltage). Also, probably good idea to try another power supply (like Ugreen 500W power brick, it has 240W mode).

Not checked, but thanks for idea. Will try next time