I try to use nvprof to fetch metrics from my GPU in AWS P3 instance. The installed Cuda is 10.1.168 and the NVIDIA Driver is 418.67.
I try the following command:
nvprof --metrics flop_count_sp_fma,flop_count_dp_fma,inst_compute_ld_st,inst_executed,inst_fp_32,inst_fp_64,inst_integer,dram_read_transactions,dram_write_transactions --log-file output.log ./xx.out
But get the following info:
==3155== Warning: ERR_NVGPUCTRPERM - The user does not have permission to profile on the target device. See the following link for instructions to enable permissions and get more information: https://developer.nvidia.com/ERR_NVGPUCTRPERM
==3155== Profiling application: ./xx.out
==3155== Profiling result:
No events/metrics were profiled.
==3155== Warning: Some profiling data are not recorded. Make sure cudaProfilerStop() or cuProfilerStop() is called before application exit to flush profile data.
when I use this:
nvprof --print-gpu-trace --log-file output.log ./xx.out
I can get the: No kernels were profiled.
So where the error come from?
This error will only come for nvprof features such as “–metrics” (for some of the metrics) which accesses the GPU Performance Counters.
These will not come for nvprof “–print-gpu-trace” as this features does not use GPU Performance counters.
As suggested in the nvprof message see the following link for instructions to enable permissions and get more information: https://developer.nvidia.com/ERR_NVGPUCTRPERM
Dear ssatoor I also have the same problem and I made everything about permission in the link which is u sended. But still I got the same result
-----------------------WARP DIVERGENCE EXAMPLE------------------------
==9624== NVPROF is profiling process 9624, command: warp_divergence.out
==9624== Profiling application: warp_divergence.out
==9624== Profiling result:
No events/metrics were profiled.
This does not seem to be the same issue. You are not getting the nvprof warning:
ERR_NVGPUCTRPERM - The user does not have permission to profile on the target device.
Can you please share the nvprof command line options used?
Also what is the CUDA Toolkit version, CUDA driver version and GPU you are using?
Also check if you see any kernels in the nvprof summary output (without using the options --events or --metrics):
nvprof
nvprof result
C:\Users\sulo4\source\repos\warp_divergence\warp_divergence>nvprof warp_divergence.out
-----------------------WARP DIVERGENCE EXAMPLE------------------------
==11160== NVPROF is profiling process 11160, command: warp_divergence.out
==11160== Profiling application: warp_divergence.out
==11160== Profiling result:
Type Time(%) Time Calls Avg Min Max Name
GPU activities: 50.02% 52.480us 1 52.480us 52.480us 52.480us code_without_divergence(void)
49.98% 52.448us 1 52.448us 52.448us 52.448us divergence_code(void)
API calls: 85.59% 252.49ms 2 126.24ms 12.800us 252.47ms cudaLaunchKernel
14.19% 41.866ms 1 41.866ms 41.866ms 41.866ms cudaDeviceReset
0.13% 379.50us 97 3.9120us 100ns 194.90us cuDeviceGetAttribute
0.08% 236.40us 2 118.20us 99.900us 136.50us cudaDeviceSynchronize
0.01% 20.200us 1 20.200us 20.200us 20.200us cuDeviceTotalMem
0.00% 8.0000us 1 8.0000us 8.0000us 8.0000us cuDeviceGetPCIBusId
0.00% 1.0000us 3 333ns 100ns 600ns cuDeviceGetCount
0.00% 1.0000us 2 500ns 100ns 900ns cuDeviceGet
0.00% 600ns 1 600ns 600ns 600ns cuDeviceGetName
0.00% 300ns 1 300ns 300ns 300ns cuDeviceGetLuid
0.00% 200ns 1 200ns 200ns 200ns cuDeviceGetUuid
And my CUDA driver verison: V10.2.89
C:\Users\sulo4\source\repos\warp_divergence\warp_divergence>nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:32:27_Pacific_Daylight_Time_2019
Cuda compilation tools, release 10.2, V10.2.89
I am using RTX2060
As I mentioned on the other post:
nvprof does not support metric collection on RTX2060 (compute capability 7.5).
You need to use Nsight Compute.
Though nvprof supports tracing on RTX2060, it is recommended that you move to using Nsight Systems.
Please refer the blog posts:
-
Migrating to NVIDIA Nsight Tools from NVVP and nvprof
https://devblogs.nvidia.com/migrating-nvidia-nsight-tools-nvvp-nvprof/
-
Transitioning to Nsight Systems from NVIDIA Visual Profiler / nvprof
https://devblogs.nvidia.com/transitioning-nsight-systems-nvidia-visual-profiler-nvprof/