We are setting all our cards to “exclusive access” mode using nvidia-smi. This works great.
My question: is there a way to programmatically determine which, or how many, cards are currently in use without trying to create contexts on them?
Jim
not really. The only way I’m aware of to even get this information is to examine the output of lsof /dev/nvidia* and look for which GPUs are open as “mem” files. Also, it only works as root because users cannot list the open files of other users.