We have the same issue here with our systems and it’s my understanding that it’s a general problem with NVIDIA. What our IT guys have told me is that the “/dev/nvidiaN” directories need to be created before a user can run a job on the NVIDIA devices. These directories either need to manually created or are automatically created when the first job is run. However, since only root has permissions to create these directories, root must be the first to run a job.
What we do is create a “/etc/init.d/nvidia” file with the following (also add “/etc/init.d/nvidia” to boot.local).
/etc/init.d% cat nvidia
mknod /dev/nvidia0 c 195 0
mknod /dev/nvidia1 c 195 1
mknod /dev/nvidiactl c 195 255
chown root.video /dev/nvidia*
chmod 0666 /dev/nvidia*
# need this to spool down the fan
You would need to adjust for the number of devices (this system has two) and change the group when setting the permissions. Note that pgaccelinfo is left in because until the Nvidia driver is loaded, the fans on our devices run at full speed.
Hope this helps,