New tool to get more GPU sensor information (Memory temp/Voltages/Blackwell hotspot)

Graphics / Linux Linux
1

Hi all.

With all the discussion of the newly discovered Blackwell hotspot sensors, I felt inspired to put together a Linux tool that would expose this information - why do the Windows users get all the fun?

Along the way, I reverse-engineered the mechanism that nvidia-smi and libnvidia-ml use to get temperature and voltage (partially documented in the kernel module source) and that unexpectedly provided the memory temperature as well as a second voltage reading - at least on a Blackwell GPU.

I have no idea whether those extra temperatures and voltages will show up for other GPUs - let me know, I guess.

It shouldn’t be hugely sensitive to driver version - probably it’ll break on anything older than 550 or so, but after that should be ok.

I can’t promise it won’t freak out your GPU and require a reset (I did that to myself a few times) but it requires direct PCIE access for the hotspot temperature and that means running as root. If you don’t run as root, it will just skip trying to read that information.

Enjoy!

2 
sudo ./nvidia-gpu-sensors
  GPU    Core Temp    Mem Temp    Hot Spot          NVVDD          MSVDD
  ----  ----------  ----------  ----------  -------------  -------------
  0         41.8 C      52.0 C         n/a        0.650 V            n/a

ChatGPT failed.

3

I know you don’t do constructive feedback, but what is the actual complaint? I certainly can’t guess what GPUs you are using, so I don’t know whether to be surprised by missing data or not.

4

5060 TI.

5

Thanks for sharing your reverse engineering work and explaining the limitations clearly. I’ll definitely keep an eye on how compatibility tool improves across different NVIDIA GPUs.

6

5060 TI.

Thanks. Reading through various discussions of this, my current theory is that the set of valid memory locations is not fixed across all models, and might vary between individual devices. So I’ve pushed an update that scans the region for anything that looks like a valid reading (valid ones have a 0x4000 tag bit set on them). I don’t know what the absolute boundaries of the memory region are, but this should be enough to see if there’s something there on your model.

Note that I’m away from my machine with my Blackwell GPU in it, so this untested. Hope I didn’t break it completely.

7

I also create a kermel module to read vram / hotspot for 5090 and export them to hwmon

8

ChatGPT failed again. You might haveto wait a few years for The Singularity to get an update.

9

Hi, I have A40 and L4 GPUs. I am looking for a tool to monitor and collect voltage usage when I run my experiment. Does your tool measure voltage for a specific workload while running on the GPU?

10

If you mean, can you observe the voltage while a workload is running, then yes. If you are asking about a single workload on a GPU running many workloads in parallel, then no - the voltage is a global characteristic of the chip and will always be based on the total load.

11

For example, I want to run the SGEMM benchmark on an A40 GPU in parallel and determine the voltage required to run a specific SGEMM workload. Can I use this tool to obtain that information?

12

Assuming it shows a voltage reading for your hardware; I’d assume it would as it’s enterprise gear, but you’ll have to test that for yourself. In terms of trying to find the required voltage, good luck - I have not had a good experience getting the GPU to recover if it errors due to insufficient voltage without a full system reboot.

13

Thanks for sharing nvidia-gpu-sensors; it’s a great tool and works fine on my 5060 Ti. I was finally able to identify a GPU Mem overheating issue while playing Isonzo. When using the ‘Quiet Fan’ setting, the Mem Temp can go up to 78°C, which is too much, and I’m getting psychedelic colors in the skies. I have now switched to performance settings, which keeps the mem below 70°C.