Quadro p state ubuntu headless

I am Having issues with my Quadro p4000 power state

My system is running 20.04 ubuntu server headless with 440.64 driver and cuda 10.2 with Nvidia docker

With nothing running on the gpu the p state always sits a p0 and will not clock down so always at about 30w

Please make sure nvidia-persistenced is started on boot an is continuously running.

1 Like

Check that and it’s running and from start up

Please run nvidia-bug-report.sh as root and attach the resulting nvidia-bug-report.log.gz file to your post. You will have to rename the file ending to something else since the forum software doesn’t accept .gz files (nifty!).

nvidia-bug-report.log.c (284.3 KB) renamed the end .c

Nothing really obvious. Does the system have any other pcie slots you can mount the card? it’s currently sitting behind a pcie switch together with an SAS raid controller, don’t know if this has any influence on the card’s power management. Another thing worth a shot would be to enable the (depreciated) persistence mode using nvidia-smi.

ok thanks that’s the only 16x sized sort I have i will give persistence a goo might just be motherboard lay out

interestingly if I try and do gpu rest I get this

nvidia-smi --gpu-reset 
GPU 00000000:05:00.0 is currently in use by another process.

1 device is currently being used by one or more other processes (e.g., Fabric Manager, CUDA application, graphics application such as an X server, or a monitoring application such as another instance of nvidia-smi). Please first kill all processes using this device and all compute applications running in the system.
root@hills-server:~# nvidia-smi
Tue Jun 23 09:58:00 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64       Driver Version: 440.64       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro P4000        On   | 00000000:05:00.0 Off |                  N/A |
| 55%   60C    P0    30W / 105W |     12MiB /  8119MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Most likely nvidia-persistenced, it gets filtered from the process list.

Ok how could I stop fix or test that ?

sudo systemctl stop nvidia-persistenced

I have move to a different motherboard with bios set to default a clean ssd with clean ubuntu server 20.04 install then did

sudo apt update 
sudo apt upgrade
sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt install nvidia-headless-440 nvidia-utils-440

if I then do Nvidia-smi is still get p0

but now got it working by doing

Nvidia-smi -pm -1
sudo systemctl restart nvidia-persistenced

so went back to my old install and did the same but on this new motherboard and it works spot on so must be a motherboard issue thanks
for your help