I tried to use “nvprof --profile-from-start on --system-profiling on --continuous-sampling-interval 50 --export-profile xxxx.prof python xxxx.py” to profile the application, where
- “–system-profiling on” enables to profile the GPU status (e.g., power, thermal, clock);
- “–continuous-sampling-interval 50” is to set the sampling rate to 50ms;
- “xxxx.prof” is the profiled file that can be imported to Visual Profiler to understand the behavior of GPU during the application;
- “xxxx.py” is a python application (e.g., training of neural network).
However, I found some issues on using nvprof as follows,
a. When I change the argument “xxxx” in “–continuous-sampling-interval xxxx”, the sampling rate does not change at all. Besides, the sampling rate differs from what I set. What is the reason for this issue?
b. When I import the profiled file ‘xxxx.prof’ into Visual Profiler, I can see the small circles that represent the sampled power. And in the “Properties” window, I can find some information of power such as “Avg, Min, Max, Samples”. For instance, “Samples” represents the number of samples. However, I failed to find a way to export all the power samples. Is there any way for exporting all the sampled power data?