High Idle performance mode in 410.xx, 415.xx and 418.xx drivers compared to 390.xx

Has anyone encountered this problem and can offer a solution -

I have a headless machine (no x server) running Ubuntu 16.04 with multiple GPUs (1080 and 2080). I recently upgraded the Nvidia drivers to accommodate the new RTX 2080 GPUs from 390.48 to 410.78. Since the upgrade the GPUs run at P0 performance mode, consuming high power resulting in high Idle temperatures. Reverting the driver back to 390.48 results in proper power management (P8 on Idle). The problem also occurs with the 410.93, 415.27 and 418.43 drivers.
Problem also occurred on a machine running Ubuntu 12.04 with 1080 and 980 GPUs.

nvidia-smi with 410.78 driver on Idle (with one 1080 GPU) -
±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
±----------------------------------------------------------------------------+
Thu Feb 21 09:51:18 2019
±----------------------------------------------------------------------------+
| NVIDIA-SMI 410.78 Driver Version: 410.78 CUDA Version: 10.0 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 1080 Off | 00000000:02:00.0 Off | N/A |
| 38% 54C P0 44W / 180W | 10MiB / 8119MiB | 0% Default |
±------------------------------±---------------------±---------------------+

Please check if starting the nvidia-persistenced is returning the gpus to normal clocks.

Turning the persistence mode on initially reduces the performance mode to P8, but after i start a service which runs a process on the GPUs the performance mode is stuck at P0 even when no process is running on them. Only when i kill all the services that run the GPU processes, the performance mode goes down back to P8.
This was tested with the 418.43 driver.

Please check if the situation changes if you run nvidia-persistenced with --no-persistence-mode option.

Without persistence mode the performance mode is stuck at P0 at all time.

Hello

I am no longer able to reach P0 mode with persistence mode running.

P2 is maximum we see.

±----------------------------------------------------------------------------+
| NVIDIA-SMI 418.56 Driver Version: 418.56 CUDA Version: 10.1 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 1080 On | 00000000:02:00.0 Off | N/A |
| 0% 52C P2 79W / 180W | 7002MiB / 8119MiB | 54% Default |
±------------------------------±---------------------±---------------------+
| 1 GeForce GTX 1080 On | 00000000:03:00.0 Off | N/A |
| 25% 48C P2 76W / 180W | 7004MiB / 8119MiB | 56% Default |
±------------------------------±---------------------±---------------------+
| 2 GeForce GTX 1080 On | 00000000:81:00.0 Off | N/A |
| 28% 57C P2 94W / 180W | 7004MiB / 8119MiB | 57% Default |
±------------------------------±---------------------±---------------------+
| 3 GeForce GTX 1080 On | 00000000:82:00.0 Off | N/A |
| 24% 45C P2 72W / 180W | 7004MiB / 8119MiB | 59% Default |
±------------------------------±---------------------±---------------------+

Running nvidia-persistenced --user nvidia --persistence-mode

Anyone else seen this on Centos 7.6 with Kernel 3.10.0-957.21.3.el7.x86_64?