Monitoring GPU Utilization "Top" like utility for GPU

Accelerated Computing CUDA CUDA Programming and Performance
#1

I’m trying to figure out how I can monitor the utilization of GPUs running CUDA kernels. Ideally, I’d like to measure on an on-going basis how idle or busy the GPUs are, something akin to “top”.

At the least, I’d like some way of measuring when kernels start and stop running.

The goal is to monitor performance on a production compute farm, so I don’t have the luxury of (1) running code under profiling, (2) forcing developers to use a wrapper API.

Would it be possible to hack the NVIDIA driver to trap kernel launch and completion events?

I’d greatly appreciate any suggestions! Thanks :-)

#2

This is a request I hear a lot and is something I’m actively looking into.

#3

I´.m looking for info too. I´m interested on it

#4

Hi,

Any news regarding this yet?

I am running a farm of 12 PCs each having two 8800GTXs running CUDA processing applications under Windows XP and would dearly like to measure the performance / utilisation of each GPU in order to optimize the software.

Regards,

Jury

#5

Coming in an upcoming release, but don’t confuse GPU utilization statistics with the results of a profiler. GPU utilization statistics will not necessarily enable you to improve performance.

#6

Thanks,

Any idea when?

Regards Jury

#7

I should know better than to make release date promises, but the release after CUDA 3.1.

#8

Thanks, won’t keep you to it.

By the way is there a way to direct Performance Monitor API (NVPMAPI) to a specific GPU?

#9

Now that 3.1 is out, do you have any update on the timing of this utility? Basically, we’re using an S1070 in a multiuser environment and I’d like to be able to tell if anyone else is using it. As the original poster said, something akin to ‘top’ for the gpu.

Thanks!