nvidia 387.12 breaks power reading in nvidia-smi.

[jbase]

I have just encountered exactly the same problem. I do mining, in my rig I have a 1050 GPU. There was some emergency situation (I was getting the “low graphics mode” error message on OS boot) and I had to reinstall CUDA using sudo apt install cuda. Obviously, a newer version (390.30) got installed. Old one was some 381.*, I think. Now, all of a sudden instead of a power reading in nvidia-smi I’m getting N/A for this card! Other cards (970s and 1060s) are ok.

How do I bring it back?! My shell scripts use it for processing!

[gerhard.hintermayer]

I have just encountered exactly the same problem. I do mining, in my rig I have a 1050 GPU. There was some emergency situation (I was getting the “low graphics mode” error message on OS boot) and I had to reinstall CUDA using sudo apt install cuda. Obviously, a newer version (390.30) got installed. Old one was some 381.*, I think. Now, all of a sudden instead of a power reading in nvidia-smi I’m getting N/A for this card! Other cards (970s and 1060s) are ok.

How do I bring it back?! My shell scripts use it for processing!

I’m based on Gentoo, running the latest driver 390.25 without any problems. My power readings are oll ok and power cap works as expecte.

[hussam]

I have just encountered exactly the same problem. I do mining, in my rig I have a 1050 GPU. There was some emergency situation (I was getting the “low graphics mode” error message on OS boot) and I had to reinstall CUDA using sudo apt install cuda. Obviously, a newer version (390.30) got installed. Old one was some 381.*, I think. Now, all of a sudden instead of a power reading in nvidia-smi I’m getting N/A for this card! Other cards (970s and 1060s) are ok.

How do I bring it back?! My shell scripts use it for processing!

I’m based on Gentoo, running the latest driver 390.25 without any problems. My power readings are oll ok and power cap works as expecte.

Yes, from my observations, not all boards have this issue. Only ones which were reporting power readings below the minimal power consumption and other accuracy issues were affected.

[jbase]

That’s strange, since power draw reporting is probably tied to GPU, not card model. Are we all talking about the same card? GTX 1050? Not the Ti version?

[hussam]

That’s strange, since power draw reporting is probably tied to GPU, not card model. Are we all talking about the same card? GTX 1050? Not the Ti version?

This comment suggests it is the board:

@aplattner

I looked into this a bit to track down the history of what happened. Apparently the power measurement circuitry on this particular GPU board isn’t very accurate, so reporting was disabled in nvidia-smi. However, there was a bug in the way it was disabled that causes it to report “ERR!” instead of “N/A”. A future driver should change the reporting to “N/A”.

[jbase]

I’ve seen this comment, I read the entire thread. Does the driver really differentiate between manufacturers? How would it do it if device ids are the same?

[gerhard.hintermayer]

I have just encountered exactly the same problem. I do mining, in my rig I have a 1050 GPU. There was some emergency situation (I was getting the “low graphics mode” error message on OS boot) and I had to reinstall CUDA using sudo apt install cuda. Obviously, a newer version (390.30) got installed. Old one was some 381.*, I think. Now, all of a sudden instead of a power reading in nvidia-smi I’m getting N/A for this card! Other cards (970s and 1060s) are ok.

How do I bring it back?! My shell scripts use it for processing!

I’m based on Gentoo, running the latest driver 390.25 without any problems. My power readings are oll ok and power cap works as expecte.

Sorry for the confusion guys. I forgot that I traded my 1050 for a 1070.

[p.gay454]

Hi,

Taking this topic more generally, is there information available on the current configurations that support power reading?

A repo reported that it works on
NVIDIA Titan X
NVIDIA Titan V

GitHub - Breakend/experiment-impact-tracker

It also worked for me on
Tesla K80 with the driver version 450.51.05.

However, it failed for a
GeForce GTX 950M with the driver version 418.56

Energy consumption is a must in my case, so I need to check this feature before buying new GPUs.

Thanks for your answer.