Setting up nvidia-persistenced

Accelerated Computing CUDA CUDA Setup and Installation
1

Hi,

I am running the latest CUDA on Ubuntu 16.04. I would like the persistence mode to be enabled at startup.

I see that the boot process is starting the persistence daemon with the persistence mode OFF:

nvidia-+ 1867 0.0 0.0 17100 1704 ? Ss 09:08 0:00 /usr/bin/nvidia-persistenced --user nvidia-persistenced --no-persistence-mode --verbose

Where is the configuration for this? How do I enable persistence mode at startup?

Thanks,
M.

2

use the nvidia-smi tool.

run

nvidia-smi --help

3

Hi,

thank you for the quick answer. Unfortunately, it doesn’t answer my question.

I am sorry I haven’t been clear enough, so I’ll reformulate.

I want to use nvidia-persistenced (not nvidia-smi) to set up my persistence mode.

If I try to do that from bash, it complains that there’s already a nvidia-persistenced daemon running with the persistence mode switched off. Indeed, if check with ps I get the following:

nvidia-+ 1867 0.0 0.0 17100 1704 ? Ss 09:08 0:00 /usr/bin/nvidia-persistenced --user nvidia-persistenced --no-persistence-mode --verbose

Now, I suppose that this daemon is started by Nvidia at boot time. I want the daemon to be started in the same way, but with the persistence mode on, so my questions are:

  • Is there a Nvidia configuration file/script that I must edit to control the startup of the daemon?
  • What is the syntax?

Thanks,
M.

4

[url]Driver Persistence :: GPU Deployment and Management Documentation

5

Ok, I may have found a solution, I did the following:

  • Download the nvidia-persistenced archive from ftp://download.nvidia.com/XFree86/nvidia-persistenced
  • Uncompress the archive
    *Edit the nvidia-persistenced.conf.template in the subdirectory corresponding to your startup service (can be upstart, systemd or sysv)
  • add “–persistence-mode --verbose” to the line where the service is started
  • run install.sh to kill the currently running daemon and install the config with the new parameters
6

Thanks for this.

I think adding “–persistence-mode --verbose” is unneeded. The daemon appears to enable PM by default.

On my system, the archive was here: /usr/share/doc/NVIDIA_GLX-1.0/sample/nvidia-persistenced-init.tar.bz2. I didn’t need to download it. See http://docs.nvidia.com/deploy/driver-persistence/index.html#installation for more details.

My system: Ubuntu 16.04, NVIDIA 384.111.

7

Is there some kind of step-by-step guide to installing this?

$ sudo nvidia-persistenced
nvidia-persistenced failed to initialize. Check syslog for more details.

8

https://docs.nvidia.com/deploy/driver-persistence/index.html#usage

9

No instructions there on how to do it; just a description of what it’s supposed to do. Running nvidia-persistenced as superuser spits an error (see above).

Or do I have to use like:

$ sudo nvidia-persistenced --user username

where username is my login?

1 Like
10

That did not work, however,

$ sudo -i

# nvidia-smi -pm 1

Persistence mode is already Enabled for GPU 00000000:21:00.0.
Persistence mode is already Enabled for GPU 00000000:31:00.0.
All done.

# exit

This did work in Ubuntu 18.04. Thanks!

11

The setup of nvidia.persistenced is formulated as follows.

1. check the status of nvidia-persistenced

$ sudo systemctl status nvidia-persistenced

2. Enable nvidia-persistenced

$ sudo systemctl enable nvidia-persistenced

3. Reboot for execution

$ sudo reboot

If there are issues in the above-mentioned second step, developers need to do the following.

4. Open nvidia-persistenced.service

$ sudo gedit /lib/systemd/system/nvidia-persistenced.service

5. Modify and add the content of the file

1). Modify the line in section of [Service]

[Service]

Change

ExecStart=/usr/bin/nvidia-persistenced --user nvidia-persistenced --no-persistence-mode --verbose

To:

ExecStart=/usr/bin/nvidia-persistenced --user nvidia-persistenced --persistence-mode --verbose

2). Add the following lines into the file:

[Install] 
WantedBy=multi-user.target 
RequiredBy=nvidia.service

Notes:

The temporal method to set nvidia.persistenced is to use the following command:

$ sudo nvidia-smi -pm 1

Cheers

4 Likes
12

My question is nvidia.persistenced brings totally different speed effects on the training oxford_flowers102.

For instance, I use two machines to train 100 epochs AlexNet for TensorFlow.

Machine One: Nvidia RTX 2070 Super

1. Power & GPU Memory:

Initial Power: 3W/215W
Power in Training: 187W/215W

Initial GPU Memory: 300MiB
GPU Memory during Training: 5500MiB

Effect: 26 minutes for completing the training.

It increase form 3Ｗ to 187w in the power consumption. In addition, the GPU Ｍemory consumption increases quite fast. Therefore, its training speed is much more fast.

2. Processes:

Beside of GID, it shows GI ID and CI ID with the command of nvidia-smi.

Machine Two: Nvidia RTX 2060

1. Power & GPU Memory

Initial Power: 9W/160W
Power in Training: 16W/160W

Initial GPU Memory: 300MiB
GPU Memory during Training: 500MiB

Its power constantly fixes on 16W and GPU memory grows quite slow. And its training speed is about 4 hours, much more slower than RTX 2070.

2. Processes:

It shows GID but no GI ID and CI ID.

My question is what is GI ID and CI ID. Do they influence the training speed?

Note:

I enable GPU Fan Settings from 45% to 74%. But is has no effect on the training speed except the a little big noise.

13

I myself has solved the issue. The RTX series can greatly speed up the training by installing Driver v450.57, CUDA 11.0 and cuDNN 8.0.1.