nvidia-persistenced failed to initialize. Check syslog for more details.

Accelerated Computing CUDA CUDA Setup and Installation
1

Following post installation steps for CUDA 10.0 https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#recommended-post

I get the following error when trying to run

$ sudo /usr/bin/nvidia-persistenced --verbose

Ubuntu 18.04

lspci | grep -i nvidia
68:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1)
68:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)

uname -m && cat /etc/*release
x86_64
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

gcc --version
gcc (Ubuntu 7.3.0-27ubuntu1~18.04) 7.3.0

uname -r
4.15.0-47-generic

linux-headers-4.15.0-47-generic is already the newest version (4.15.0-47.50).
The following package was automatically installed and is no longer required:
  libnvidia-common-418

Installed using package manager https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#ubuntu-installation

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.104      Driver Version: 410.104      CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 00000000:68:00.0  On |                  N/A |
| 45%   36C    P5    21W / 250W |    552MiB / 11175MiB |      2%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      2145      G   /usr/lib/xorg/Xorg                            40MiB |
|    0      2193      G   /usr/bin/gnome-shell                          51MiB |
|    0      2862      G   /usr/lib/xorg/Xorg                           320MiB |
|    0      2979      G   /usr/bin/gnome-shell                         134MiB |
|    0      3011      G   /opt/teamviewer/tv_bin/TeamViewer              2MiB |
+-----------------------------------------------------------------------------+

cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module  410.104  Tue Feb  5 22:58:30 CST 2019
GCC version:  gcc version 7.3.0 (Ubuntu 7.3.0-27ubuntu1~18.04)

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

syslog output (I ran with sudo):

May  2 12:30:25 ASUS-sage nvidia-persistenced: Verbose syslog connection opened
May  2 12:30:25 ASUS-sage nvidia-persistenced: Directory /var/run/nvidia-persistenced will not be removed on exit
May  2 12:30:25 ASUS-sage nvidia-persistenced: Failed to lock PID file: Resource temporarily unavailable
May  2 12:30:25 ASUS-sage nvidia-persistenced: Shutdown (6933)

Any hints what I can do to debug/fix this?

1 Like
2

Just noticed this, bumping the thread since it will look old.

3

I am also getting this error - did you ever solve it?

4

I have the same error.

5

Yes, I solved it. For me (on Ubuntu) I had to use nvidia-smi as root (not sudo, but as root). then type nvidia-smi --help and the command for establishing persistence will be there <-pm 1>.

$ sudo -i

# nvidia-smi -pm 1

# exit

$
17 Likes
6

That works for me. More details under “Additional low level commands used for clocks and power”
on this link: Useful nvidia-smi Queries | NVIDIA

7

nvidia-smi -pm 1

Tested and working with Ubuntu 22.04.2 LTS … thanks andorjkiss

As per this Doc … ?
https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#post-installation-actions

sudo -i

Failed

/usr/bin/nvidia-persistenced --verbose
nvidia-persistenced failed to initialize. Check syslog for more details.

cat /var/log/syslog
…
Jun 18 19:13:54 usr-pc nvidia-persistenced: Verbose syslog connection opened
Jun 18 19:13:54 usr-pc nvidia-persistenced: Directory /var/run/nvidia-persistenced will not be removed on exit
Jun 18 19:13:54 usr-pc nvidia-persistenced: Failed to lock PID file: Resource temporarily unavailable
Jun 18 19:13:54 usr-pc nvidia-persistenced: Shutdown (12070)

root@usr-pc:~# nvidia-smi -pm 1
Persistence mode is already Enabled for GPU 00000000:01:00.0.
All done.

8

I don’t quite understand what andorjkiss meant by “solved it”. The post installation document says the command “/usr/bin/nvidia-persistenced --verbose” should be run on boot as the root user. So, what sense does it make to run it after boot? Also, “nvidia-smi -pm 1” outputs the same error on my machine. Furthermore, this command does not change anything – It’s still persistence mode, not persistence daemon.

However, the booting message shows that the Persistence Daemon has been started during boot:

image
image849Ă—77 5.38 KB

I don’t know how this daemon was installed or whether we should worry about it at all. But I think this is the correct way of initializing persistence daemon, instead of running on command line after booting.

9

Similar issues here nvidia-persistenced results in nvidia-persistenced failed to initialize. Check syslog for more details..
I think this problem is related to another problem I got, wich is that I cannot run docker run -it --rm --gpus all ubuntu nvidia-smiI get the error

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown.

I try to follow official docs, but I get this error.

10

~$ sudo -i
root@:~# nvidia-smi -pm 1
Unable to set persistence mode for GPU 00000000:01:00.0: Unknown Error
Terminating early due to previous errors.