I am measuring an application using nvidia-smi.
It seems higher value for utilization.gpu compared to nsys (nsight systems) time chart.
From reading the nvidia-smi man, the sample period for nvidia-smi is 1 or 1/6seconds. It is good for long time CUDA kernel, but not good for short time CUDA kernel.
Can nvidia-smi change the sample period (especially for short time multi-CUDA kernel) ?
To my knowledge, nvidia-smi does not have a user-selectable setting to choose sampling frequencies from a fairly wide range. Such functionality is not needed for a tool designed for general system monitoring and management.
In my experience, specifying a high sampling frequency actually has a negative impact on the kernels execution on the GPU. On Windows, I use a third-party tool called TechPowerUp GPU-Z to monitor my GPUs. Its lowest sampling interval setting is 0.1 seconds. I tried that and judged it unusable due to the reason I stated and went back to sampling at 1 Hz. nvidia-smi (and presumably GPU-Z) makes use of the NVML (NVIDIA management library), which you can use in your own applications.
What metrics in particular are you trying to capture with fine granularity, and what kind of sampling frequency do you hope to achieve? Have you checked the literature what set-ups others use for that purpose?