NVIDIA-SMI: Great starting point for monitoring GPU...

NVIDIA Virtual GPU Forums Virtual GPU Monitoring/Assessment Tools
#1

With NVIDIA drivers installs there is the nvidia-smi tool. See the following link for info:
https://developer.nvidia.com/nvidia-system-management-interface

See this link for the man page and various switches/tools to use:
http://developer.download.nvidia.com/compute/cuda/5_5/rel/nvml/nvidia-smi.5.319.43.pdf

For GRID specifically, keep in mind the location of the driver. As an example, if using vSGA (shared) in VMware then the driver is the VIB loaded on the ESXi hypervisor, so the nvidia-smi executable is located there, on the hypervisor. If on that same deployment you chose to use on GPU as pass-through then that GPU is looking to the NVIDIA driver on the guest and so nvidia-smi is located on the guest.

#2

For vGPU do make sure you run "nvidia-smi -l" without the loop flag you can see stalls as it reenters the API - I added some other tools here: http://www.xenserver.org/partners/developing-products-for-xenserver/20-dev-hints/133-xs-dev-tools.html

and GPU specific ones: http://www.xenserver.org/partners/developing-products-for-xenserver/18-sdk-development/136-xs-dev-gpu-tools.html

For pass-through you will need to use something in guest e.g. Fraps

#3

There was a bug in the tech preview / beta release of vGPU where running nvidia-smi would cause a visible glitch or stall in VMs running vGPU - but this was fixed in the vGPU 1.0 production release. You shouldn’t see any stall when running nvidia-smi now.