Monitoring GPU utilization or memory/temperature

I’ve noticed nvidia-smi is not included with the NVIDIA driver on my TK1. Is there any way to monitor GPU utilization by writing a CUDA application? I’ve perused the APIs, but haven’t found any mechanism to get this information.

Thanks!

Without the NVML library or the NV-CONTROL X extension used by the nvidia-settings app, there is no other way that I know of to get that information. I would suggest to request that the NVML library (and thus nvidia-smi) be made available on Jetson if this is what you want.

I’m able to use the stock linux sensors utility from the lm-sensors package to see a bunch of different temperature sensors. It won’t get you memory utilization, but you will be able to see the temperature and make sure you’re not overheating:

$ sensors
CPU-therm-virtual-0
Adapter: Virtual device
temp1:        +35.5°C  (crit = +101.0°C)

GPU-therm-virtual-0
Adapter: Virtual device
temp1:        +32.5°C  (crit = +101.0°C)

MEM-therm-virtual-0
Adapter: Virtual device
temp1:        +31.5°C  (crit = +101.0°C)

PLL-therm-virtual-0
Adapter: Virtual device
temp1:        +33.5°C  

Tboard_tegra-virtual-0
Adapter: Virtual device
temp1:        +25.0°C  

Tdiode_tegra-virtual-0
Adapter: Virtual device
temp1:        +29.0°C

The sensors tool is built on top of a standard Linux API called libsensors, and there are a number of graphical front-ends for it as well.

I’m seeing similar temps on my TK1. Running a GPU task continuously drove the temps up by a few degrees.

The rather robust fan seems to be effective. :)

FYI, I installed lm-sensors by downloading and building the package as well as installing flex and bison.

Thank everyone, lm-sensors seems to be the best option for now. Under full load I have yet to see the GPU get above 39°C. At idle, it’s around 29°C.

I installed lm-sensors through the package manager; much easier than trying to compile it from souce:

sudo apt-get install lm-sensors

@namniart, interesting! I first tried to install lm-sensors with apt-get but the package wasn’t available.

Sigh, I must have something wrong with my apt-get configuration.

Did you uncomment the universe repository in /etc/apt/sources.list?

apt-get fails to get lm-sensors when I try, and I get similar errors on other packages, I wonder if some step of the Jetson setup messed up the package system?:

apt-get install lm-sensors

Reading package lists… Done
Building dependency tree
Reading state information… Done
Package lm-sensors is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package ‘lm-sensors’ has no installation candidate

Strange, it installed for me without problems, and I haven’t altered my repos. What version of Linux for Tegra do you have? I have Rel 19.

I have the same result as @tachyon_john.

Yeah; I enabled the universe repos. It’s usually one of the first things I do when I set up a new Ubuntu machine, and a bunch of the other stuff I’m doing requires packages from universe.

Yes, it would appear that many of the “missing” packages are in the universe repo, I found a few of the ones I was looking for there.