We have a Linux cluster with numerous A100 GPUs. According to nvidia-smi, the GPUs operate in Perf mode P0 and are drawing ~50 W/GPU while idle. If it is known the GPUs will be left unused for a long period of time, is there a way through the command line to reduce the idle power consumption either through the Perf mode or another setting, preferably without having to reboot to toggle the power/performance settings? I have reviewed nvidia-smi and nvidia-settings and there is nothing obvious—the best I could find was a way to hide the GPU as an available device, but nvidia-smi continues to report the same idle power consumption.
Thanks,
G