I’ve recently started working on a multi-GPU system and noticed that my programs were running slow. I tracked it down to the querying of the devices by the system.
The system is an Ubuntu Linux x86_64 2.6.32-25 with NVIDIA driver 260.19.26. It has 4 Geforce GTX 295 (nvidia-smi reports 8 devices, which I assume is because of the 2 GPUs on each board, but I’m not sure since I haven’t seen the actual hardware). X is not running, nor is anything else which I might suspect of interfering.
When listing devices the system is very slow. ‘nvidia-smi -a -q’ takes about 5 seconds, my own enum-gpu program as well, and my real program spends the same amout of time. When looking at a strace, one can see that the time is spent between the call to
open("/dev/nvidiactl", O_RDWR) = 3
and after going through the devices
open("/dev/nvidia7", O_RDWR) = 11
After that, the opens to the /dev/nvidiaX are repeated, but don’t take an unusual amount of time.