nvidia-persistenced - not running.

A few questions .

  1. What is the impact of errors like the following on cuda code (e.g. Tensorflow etc.) ? Running with driver 430 on Ubuntu.

nvidia-persistenced[38948]: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced

  1. Why the following uses a non-root user - which of course would not be able to access the /var/run directory.

[Unit]
Description=NVIDIA Persistence Daemon
Wants=syslog.target

[Service]
Type=forking
ExecStart=/usr/bin/nvidia-persistenced --user nvidia-persistenced --no-persistence-mode --verbose
ExecStopPost=/bin/rm -rf /var/run/nvidia-persistenced

[url]Driver Persistence :: GPU Deployment and Management Documentation

I see that the problem is a follows.

Failed to open libnvidia-cfg.so.1: libnvidia-cfg.so.1: cannot open shared object file: No such file or directory

Looks like libnvidia-cfg is a OpenGL/GLX library -see following.

libnvidia-cfg1-430/bionic 430.09-0ubuntu0~gpu18.04.1 amd64
NVIDIA binary OpenGL/GLX configuration library

Using NVIDIA GPUs exclusively for compute (e.g. headless driver) . There is a separate display driver in the system. Would installing libnvidia-cfg mess up that display driver ?

I found in order to fix the missing libnvidia-cfg.so.1 file issue, you need to install either libnvidia-cfg1 or libnvidia-legacy-390xx-cfg1
So the following command fix the issue.
sudo apt install libnvidia-legacy-390xx-cfg1

Update November 2023: Note that there is no single ‘libnvidia-cfg1’ available in the modern stack - instead, you must specify the version that matches your driver version, as “libnvidia-cfg1-xxx” format, where “xxx” is your driver version (for example, libnvidia-cfg1-545).

It would be helpful to show how to get the driver version:
find /usr/lib/modules -name nvidia.ko
/usr/lib/modules/5.19.0-1010-nvidia-lowlatency/kernel/nvidia-530/nvidia.ko

modinfo /usr/lib/modules/5.19.0-1010-nvidia-lowlatency/kernel/nvidia-530/nvidia.ko | grep ^version
version: 530.41.03

sudo apt install libnvidia-cfg1-530
Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
The following NEW packages will be installed:
libnvidia-cfg1-530
0 upgraded, 1 newly installed, 0 to remove and 13 not upgraded.