Capture vGPU performance data in csv format using nvidia-smi

Capture the GPU utilization, GPU memory utilization, Total GPU memory, GPU memory free, GPU memory used performance statistics to a CSV formatted file.

In the XenServer dom-0 console, issue the following command to get the PCIe bus IDs for the vGPU enabled GPUs.

root@nvgrid:~ # nvidia-smi | grep GRID
| 1 GRID K2 On | 0000:09:00.0 Off | Off |
| 2 GRID K2 On | 0000:44:00.0 Off | Off |

To capture the GPU utilization for all vGPUs enabled virtual machines on PCIe ID "0000:09:00.0" and display to the console:

root@nvgrid:~ # nvidia-smi --query-gpu=utilization.gpu,utilization.memory,,,memory.used --format=csv --id=0000:09:00 -l 1

To redirect the output to a file:

root@nvgrid:~ # nvidia-smi --query-gpu=utilization.gpu,utilization.memory,,,memory.used --format=csv --id=0000:09:00 -l 1 -f ./GPU-09-stats.csv

The data will be captured in one second intervals and written to the file.

To monitor the other vGPU enabled GPUs, replace the "–id" value "0000:09:00" with the PCIe ID you wish to monitor.

NOTE: GPUs designated for use as PCI pass-through will not be displayed in the output from nvidia-smi as access to those GPUs are handled by the hypervisor.

(a future post will address the monitoring of GPU pass-through enabled virtual machines)