RTX 3090 throttles down underload temps looks fine

I can’t find the reason why one of my 6 RTX 3090’s are downclocking

nvidia-smi --query-gpu=gpu_name,clocks.sm,clocks.mem,temperature.gpu,power.draw,clocks_throttle_reasons.sw_thermal_slowdown,persistence_mode,pstate,clocks_throttle_reasons.hw_slowdown,clocks_throttle_reasons.hw_thermal_slowdown,clocks_throttle_reasons.hw_power_brake_slowdown,clocks_throttle_reasons.applications_clocks_setting,clocks_throttle_reasons.sw_power_cap --format=csv
name, clocks.current.sm [MHz], clocks.current.memory [MHz], temperature.gpu, power.draw [W], clocks_throttle_reasons.sw_thermal_slowdown, persistence_mode, pstate, clocks_throttle_reasons.hw_slowdown, clocks_throttle_reasons.hw_thermal_slowdown, clocks_throttle_reasons.hw_power_brake_slowdown, clocks_throttle_reasons.applications_clocks_setting, clocks_throttle_reasons.sw_power_cap
NVIDIA GeForce RTX 3090, 1725 MHz, 9501 MHz, 59, 343.47 W, Not Active, Enabled, P2, Not Active, Not Active, Not Active, Not Active, Active
NVIDIA GeForce RTX 3090, 825 MHz, 9501 MHz, 47, 173.30 W, Not Active, Enabled, P2, Not Active, Not Active, Not Active, Not Active, Active
NVIDIA GeForce RTX 3090, 1710 MHz, 9501 MHz, 56, 326.26 W, Not Active, Enabled, P2, Not Active, Not Active, Not Active, Not Active, Active
NVIDIA GeForce RTX 3090, 1695 MHz, 9501 MHz, 56, 313.63 W, Not Active, Enabled, P2, Not Active, Not Active, Not Active, Not Active, Active
NVIDIA GeForce RTX 3090, 1710 MHz, 9501 MHz, 58, 326.47 W, Not Active, Enabled, P2, Not Active, Not Active, Not Active, Not Active, Active
NVIDIA GeForce RTX 3090, 1695 MHz, 9501 MHz, 59, 314.73 W, Not Active, Enabled, P2, Not Active, Not Active, Not Active, Not Active, Active

Driver Version: 470.94
Ubuntu 18.04.6 LTS

the GPU is not hot to the touch and there is a lot of airflow

any suggestions what I can do to fault find would be appreciated

It’s not really readable but it seems there’s a power limit been set on all gpus, maybe that one has a wrong power limit.
Please run nvidia-bug-report.sh as root and attach the resulting nvidia-bug-report.log.gz file to your post.

It does look like that but I assure you I have reset the power limits and I have reset the clock lock using nvidia-smi -rgc and nvidia-smi -pl 350

https://ibb.co/6r003GH
https://ibb.co/yBKYLj0

I have tested this on windows. It does the same. I can’t see why its down clocking. WHat am I missing?