This isn’t strictly a CUDA topic but I hope you’ll pardon me. This question has sat unanswered for several days in the general application forum. I hope that with all the linux gurus hanging out here I can get an anwer.
Is there any way to check the GPU load on nVidia GPU’s on a linux system? I’ve got several systems with 3 nVidia graphics cards each, used for 3D thin client servers. I’m seeing high load that I can’t correlate with CPU use, network I/O, or disk access. I suspect that I may be facing a GPU bottleneck on one or more of the graphics cards, but can’t figure out how to check that.
# lspci | grep nVid
10:00.0 VGA compatible controller: nVidia Corporation NV34 [GeForce FX 5200] (rev a1)
10:01.0 VGA compatible controller: nVidia Corporation NV34 [GeForce FX 5200] (rev a1)
40:00.0 VGA compatible controller: nVidia Corporation NV45GL [Quadro FX 3400/4400] (rev a2)
# cat /etc/redhat-release
Red Hat Enterprise Linux WS release 3 (Taroon Update 8)
Thanks in advance for any ideas. I need a CLI option by the way, too many servers to try running any sort of gui.
Whenever a CUDA app is executing a kernel, the GPU usage is 100%. So as long as any CUDA apps you are running call kernels constantly, the GPU usage by that application will be 100% of however many GPUs it was written for.
Thanks very much for writing back. Maybe I wasn’t explicit enough when I said this wasn’t specifically a CUDA issue… this isn’t at all related to CUDA but I came to this sub-forum because I couldn’t get any answers in the proper forum for the question. This is the only place I’ve been able to find the linux gurus.
Plain vanilla linux system, running older nVidia cards that don’t support CUDA regardless – how do I see the load on the GPUs (3 of them)? Running a variety of 3d apps for multiple users.
Oh, well I only skimmed your original post. This is the CUDA forum after all. And most posters here are pure CUDA and not much in the way of OpenGL/graphics.
Anyways, there isn’t any “gputop” command that will tell you the GPU usage. The closest thing I know about is PerfHUD (NVIDIA PerfHUD | NVIDIA Developer), but that is windows/DirectX only.
Umm, here is one idea that might help. Any process accessing the GPU must do so through the device node files. I’ve never tried it, but these might show up in lsof. That will at least tell you how many processes are accessing GPUs. If you have more than on a single GPU, it would be safe to say that GPU is overextended.