Hello,
I’m trying to profile a CUDA kernel running on a remote machine using Nsight Compute.
I installed CUDA 10 on both local (where I run Nsight Compute) and target machine (where I run the kernel on a Tesla V100 GPU).
I’m able to profile the kernel using nvprofbut if I try to use nv-nsight-cu I get the error: kernel profiling is not supported on this devicedespite the command nv-nsight-cu-cli --list-chips returns the architecture gv100 which is the one of the Tesla V100.
I tried to build the program with and without the --arch flag but nothing changes.
What can I do? Thanks :-)
Are you still stuck with this issue? Can you try using a newer version of Nsight Compute?
I had the same problem.
when I add the path of nv-nsight-cu-cli into /etc/sudoers, and run nv-nsight-cu-cli as sudo nv-nsight-cu-cli, the problem is disappear.
/etc/sudoers
Defaults secure_path=“/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/usr/local/cuda/bin”
Any updates on this?
I am having this issue when doing remote interactive profiling.
In case it worths knowing, I am using a Tesla P40, and this is the supported chips listed
(base) [edreis@compute-0-0 build]$ nv-nsight-cu-cli --list-chips
gp102, gp104, gp106, gp107, gp108, gv100, gv11b, tu102, tu104, tu106, tu116, tu117
It doesn’t seem the P40 is listed. Would that be the cause of my issue?
I am not sure if the solution from @chen.bo would solve my case. My admin just helped giving permission to access NVIDIA GPU Performance Counters on the target device, after I had the problem below (which is now solved)
(base) [edreis@compute-0-0 build]$ nv-nsight-cu-cli --kernel-id ::mygemm:6 --section ".*" -o 1-1-pinned-basic 1-1-pinned-basic
generate data
==PROF== Connected to process 293082 (/home/edreis/Repo/nvidia-performance-tools/sgemm/build/1-1-pinned-basic)
transfer to GPU
==ERROR== Error: ERR_NVGPUCTRPERM - The user does not have permission to access NVIDIA GPU Performance Counters on the target device 0. For instructions on enabling permissions and to get more information see https://developer.nvidia.com/ERR_NVGPUCTRPERM
Tesla P40 is a gp102. Support for this was dropped from Nsight Compute version 2020.1.
Release notes for 2020.1 mentions this:
Removed support for the Pascal SM 6.x GPU architecture
Which version of Nsight Compute are you using? Can be checked using:
nv-nsight-cu-cli --version
nv-nsight-cu-cli --version
NVIDIA (R) Nsight Compute Command Line Profiler
Copyright (c) 2012-2019 NVIDIA Corporation
Version 2019.5.0 (Build 27346997)
To summarize - you are facing an issue when doing remote interactive profiling on a Tesla P40 using Nsight Compute Version 2019.5.0.
It is not clear why.
Was the permissions error ERR_NVGPUCTRPERM solved. Can you confirm that you can collect metrics using nv-nsight-cu-cli on the Tesla P40?
Since we have dropped Pascal GPU support from Nsight Compute - I would suggest that you move to using Visual Profiler and/or nvprof on Tesla P40.
