2080 Ti on Ubuntu 18.04 / Permanently in Power Mode P0

We have assembled a handful of headless rigs for deep learning here. Each machine consists of a Threadripper 1900, a x399 board, and 2x 2080 Ti’s (Gigabyte Windforce). The operating system is Ubuntu 18.04. The

We tried 410 (those that come with the CUDA toolkit), 415 and 418 drivers already. But regardless the GPUs stay in power mode P0 all the time. Power consumption - despite being idle - about 65W. Fans spinning at ~50% according to nvidia-smi. Is this normal? In our previous TitanX builds the power consumption when idle is ~10W and the fans switch off.

Note that there is sufficient space between the cards. So that shouldn’t be an issue. Temperature is in idle is 40-45 degC. Please advise if I missed something. Or is this power consumption normal? is there anyway to manually toggle the cards to switch to power mode P8?
nvidia-bug-report.log.gz (1.33 MB)

Please run nvidia-bug-report.sh as root and attach the resulting .gz file to your post. Hovering the mouse over an existing post of yours will reveal a paperclip icon.
[url]https://devtalk.nvidia.com/default/topic/1043347/announcements/attaching-files-to-forum-topics-posts/[/url]

Bug report attached. Sorry for the delay.

The nvidia-persistenced is always started and stopped just shortly afterwards. Can you change it to continously running? Are you using some docker container?

Thanks generix, that did the trick. Everything looks as expected now.