I start my headless machine, ssh into it and start a vncserver. Then I use a TightVNCviewer to remote into the machine and execute my code. I get “no cuda devices found” and “/dev/nvidia1 no such file or directory” errors. I check the /dev folder and nvidia1 does not exist. Is it because my computer starts up without a monitor because it works when I start up with a monitor. How do I load up cuda drivers after the login? Or is this a VNC problem? Thanks.
The short answer is that if you aren’t running X11 on the CUDA machine, you need to create the /dev entries manually. How this is done is is discussed in the CUDA toolkit release notes for linux