Suddenly much lower GPU performance in inference

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
,
1

ASUS Ascent GX10

Running LLMs via llama.cpp server in a Docker container.

Before, the GPU clock was constantly around 2.4 GHz. I used to get about 45 tokens / second from an LLM in inference. GPU power usage would spike up to 80 W during inference. It was okay.

Yesterday I rebooted the machine after some APT package updates.

Now the GPU clock is around 2 GHz while idle. When doing inference, the clock may drop as low as 669 MHz. GPU power usage while doing inference is around 11 … 13 W. I only get 35 tokens / second now, a massive drop.

I’ve tried to power off the device, unplug it from the power supply, and unplugged the supply from AC, waited 30 minutes, plugged it back in, tried again - and nothing has changed. Performance is still abysmal.

What can I do to get performance back up to where it was?

florin@spark:~$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=24.04
DISTRIB_CODENAME=noble
DISTRIB_DESCRIPTION="Ubuntu 24.04.4 LTS"
florin@spark:~$ uname -r
6.17.0-1021-nvidia
florin@spark:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Wed_Aug_20_01:57:39_PM_PDT_2025
Cuda compilation tools, release 13.0, V13.0.88
Build cuda_13.0.r13.0/compiler.36424714_0
florin@spark:~$
2

Power off. Unplug power from wall. Wait a minute. Plug back in and power back on.

3

Did you actually read my message?

4

All but 1 line by the look of things.

You are obviously suffering from the power delivery bug so maybe try once more?

5

Yeah, trying again did it. I unplugged it while eating lunch, so the machine sat there for more than 30 minutes, completely disconnected. The power brick was also unplugged from AC.

Plugged it back in, tried inference again, and now I’m back to normal:

  • 2.4 GHz GPU clock while idle
  • more like 2.5 GHz during inference
  • 10 W GPU power usage while idle
  • 40 W during a simple inference job (cannot run a bigger job now, but this is normal for a small inference query, and it may go over 80 W for heavy jobs)
  • 48 tokens / second, which is normal for the LLM I use

Thank you for pushing me to try again.

This bug is very annoying. Hopefully it’s something that could be fixed in the driver, or at least in firmware.

I feel like the behavior of the clock is closer to normal while idle when the bug occurs. The clock does seem like it should dip when not in use. But when inference starts with the bug activated, the GPU clock actually drops very low, which is the complete opposite of how it should work.

6

Glad it worked.

The trick is indeed to have the power brick unplugged on both ends. There was a firmware upgrade recently that is supposed to improve this behaviour.

7

Is this something more recent than the ones we were discussing previously?

8

Interesting, my second node shut down today, so I thought it’s a good opportunity to update the system and the ds4 image and since then I’m getting about half the speed.

That being said, somehow the the network changed and I haven’t tried the unplug method yet.

9

Depends when you were discussing it, I think this one was released early June for the FE models.

10

I’m pretty sure I’m up to date with the firmware. I apply it every time I see the notification. I log into the system daily.

Is there a different firmware release cadence for the ASUS clones, compared to the NVIDIA original systems?

11

Same bug has hit me once as well. The unplug and wait 30 mins trick fixed it.

12

That update isn’t available for the GX10 yet if it was released in June. It’s a shame Nvidia handles communication poorly.