nvidia-smi LINUX correct?

Accelerated Computing CUDA CUDA Programming and Performance
#1

Hi, don’t know how else to word topic question.
in Linux, RHEL 7.6 with cuda 10.1.105, using either a P100 or V100 card,
doing an nvidia-smi -a it lists the graphics card but power draw never changes from around 26.xx watts, I think it may change to something like 29.xx but i keep forgetting to check. And for Gpu Utilization it’s always at 0%.

Have a piece of software, in a work environment, it runs only in linux not windows, we paid to have re-coded from cpu version to gpu version to take advantage of cuda, and it does. We see obvious gains like an 8 hour job took 1 hour, something like that. I believe it mostly makes use of cuBLAS.
However when i run a test case job that takes ~3 hours, trying to use nvidia-smi I never see gpu utilization change from 0% nor do I see power draw go above 30 watts.

Now I did have just one P620 card installed in server (for monitor) and that could be used (we normally set nvblas.conf and set the GPU variable to numbers not including this p620). So Itired using just the p620 (V100 and P100 removed from server), tried a small test case that ran in ~4 minutes with only the p620 card installed in server and I did see Utilization go to 100% on it (don’t remember power draw number).

Can someone make sense of this to me?

#2

It sounds to me like the P100 or V100 is not being used. Or, if it is being used, the utilization is so small and in such short bursts that the sampling interval associated with nvidia-smi doesn’t “see” it. However if the application has a long run time, it should be evident which GPUs are being used by the application (regardless of exact utilization pattern) by running the nvidia-smi summary command:

nvidia-smi

and looking at the bottom of the output. It will show which processes are using which GPUs, regardless of their exact activity pattern.

If the P620 is being used which is presumably not according to your intent, you should be able to prevent that with careful use of the CUDA_VISIBLE_DEVICES environment variable. Googling that will quickly give you descriptions of how it is used.