I am measuring an application using nvidia-smi.
It seems higher value for utilization.gpu compared to nsys (nsight systems) time chart.
From reading the nvidia-smi man, the sample period for nvidia-smi is 1 or 1/6seconds. It is good for long time CUDA kernel, but not good for short time CUDA kernel.
Can nvidia-smi change the sample period (especially for short time multi-CUDA kernel) ?
Reference
https://developer.download.nvidia.com/compute/DCGM/docs/nvidia-smi-367.38.pdf
To my knowledge, nvidia-smi does not have a user-selectable setting to choose sampling frequencies from a fairly wide range. Such functionality is not needed for a tool designed for general system monitoring and management.
In my experience, specifying a high sampling frequency actually has a negative impact on the kernels execution on the GPU. On Windows, I use a third-party tool called TechPowerUp GPU-Z to monitor my GPUs. Its lowest sampling interval setting is 0.1 seconds. I tried that and judged it unusable due to the reason I stated and went back to sampling at 1 Hz. nvidia-smi (and presumably GPU-Z) makes use of the NVML (NVIDIA management library), which you can use in your own applications.
What metrics in particular are you trying to capture with fine granularity, and what kind of sampling frequency do you hope to achieve? Have you checked the literature what set-ups others use for that purpose?
That’s a fairly old driver and therefore fairly old documentation.
nvidia-smi has command line help, have you checked it?
nvidia-smi --help
for example, I have driver 470.57.02 and I see in the help output:
-l, --loop= Probe until Ctrl+C at specified second interval.
-lms, --loop-ms= Probe until Ctrl+C at specified millisecond interval.
If you study the command line help a bit more, you might try something like this:
nvidia-smi -lms 100 -q -d UTILIZATION
NVIDIA’s main web page for nvidia-smi
links this documentation:
This does mention the -lms option, but I overlooked it:
-lms ms, --loop-ms=ms
Same as -l,–loop but in milliseconds.
Sorry about that, did not mean to mislead.
I recommend that people use the command line help for detailed nvidia-smi usage.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.