Just an FYI…we recently upgraded from CUDA 1.0 to CUDA 1.1 on our GPU computing cluster. The techs had to run the commands listed below on each host at reboot to get things working properly. Not sure why this was necessary; the techs speculated that it might be necessary because the cluster nodes don’t run X, which probably initializes the device files on its own.
Setup Nvidia device files
mknod /dev/nvidiactl c 195 255
for n in 0 1 2 3 ; do
mknod /dev/nvidia$n c 195 $n
done
chmod a+w /dev/nvidia*