I have a box with multiple GPUs running ubuntu 16.04. The box is going to have multiple users running different experiments and I need to restrict access to the GPUs so that only certain users have access to certain GPUs. As suggested on Restrict user access to GPU - Ask Ubuntu I’ve tried changing the group and permission of the /dev/nvidia* files however, when a new session is opened they revert back to their original settings. I’ve also created /etc/modprode.d/nvidia.conf which contains options nvidia NVreg_DeviceFileGID=120 but this doesn’t work either. Am I placing nvidia.conf in the wrong place or is the syntax wrong? Any other suggestions would be greatly appreciated!
put the gpus in persistence mode so that the device files are not re-created.
[url]https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#install-persistenced[/url]
you can automate this:
[url]http://www.resultsovercoffee.com/2011/01/cuda-in-runlevel-3.html[/url]
Hi there, thanks for the suggestion but it doesn’t seem to work. I’ve tried changing the group of the /dev/nvidia* file and then placing in persistance mode using both nvidia-persistenced and nvidia-smi -pm 1 however neither work. Strange thing also happens when I use nvidia-persistanced --user=USERNAME --persistance-mode and reboot - running ps -aux|grep -e nvidia displays /usr/bin/nvidia-persistenced --user root --no-persistence-mode --verbose ??
I wouldn’t expect any of this to survive a reboot unless you have built some init scripts to set things up at boot.
[url]https://docs.nvidia.com/deploy/driver-persistence/index.html#installation[/url]
I haven’t used any scripts at boot time?
Putting a question mark at the end of your statement is confusing to me. Are you asking me a question? If you are asking me whether you have used any scripts at boot time, I would not be able to answer that. If you are making a statement that you have not used any scripts at boot time, then I have already suggested that is something you need to address, and it’s unclear why you would put a question mark at the end of a declarative statement.
If you reboot, any settings changes you’ve made either to persistence mode or to the permissions applied to the device files will be lost. They will not magically reappear on the next boot cycle.
Do you know what a init script is in linux?
In order to make your changes take effect on each and every boot cycle, you will need to do some plumbing in linux. If you google “linux init script” you’ll get some ideas, and at least 2 of the links I’ve provided already in this thread reference this need and even give some examples, or tell you where to find examples of init scripts.
Hi Robert. Apologies for my poor grammar but, is the sarcasm entirely necessary? Obviously I don’t know as much as you which is why I am asking for help. I meant that I wasn’t using a script at boot time and was therefore questioning why the daemon was still running after reboot. I appreciate that this shouldn’t be the case and was therefore wondering what might be the cause. I will now try with a script and see if that works however, I am still unsure about how to change the /dev/nvidia* files? All the links you’ve given are only concerned with the daemon. I’m guessing that I change the dev/nvidia* files prior to starting the daemon in the script. Would I be correct?
I apologize for offending you.
I wrote a short tutorial on restricting GPUs to certain users that you might find useful: