Power Monitoring on Titan V

Developer Tools Other Tools System Management and Monitoring (NVML)
1

I have a system with Titan V’s where I would really like to monitor the power. I was under the impression that these systems could not do so, after seeing N/A / N/A under power. My NVML tool also reported that it was not supported. However, today I realized that nvidia-smi was returning a power measurement for some GPUs. It seemed like it was active, changing, and tracking. However, it eventually went back to N/A / N/A.

Is this actually supported on Titan V? Is there some special setting that needs to be changed to enable it? Or did I maybe stumble upon a bug?

Thanks


nvidia-smi3.png
nvidia-smi3.png712×228 38.7 KB

2

“nvidia-smi” sometimes miss some measurements (running so fast :-). Try to activate driver persistence “nvidia-smi -pm 1”, wait few seconds and try again.

3

Wow, this works. Ok, it seems it needs a combination of (1) persistence mode, (2) some CUDA op to trigger driver persistence (empty kernel for example). For some reason, I had never encountered this issue on my prior devices. Thanks a lot for the info and advice!