Cannot run profiling

Accelerated Computing CUDA CUDA Programming and Performance
1

How can I enable permission for a user on my Centos 7, linux machine who is not an admin to perform profiling for CUDA ?

The user tried all of the following three profiling apps:

Visual Profiler
nSight Systems
nSight Compute

In the case of the visual profiler, the user gets the message:
==16463== Warning: ERR_NVGPUCTRPERM - The user does not have permission to profile on the target device. See the following link for instructions to enable permissions and get more information: https://developer.nvidia.com/ERR_NVGPUCTRPERM

for nSight Systems:
the user sees only partial information which does not include: the GPU back tracing and communications.

for nSight Compute:
get the error: ERR_NVGPUCTRPERM - The user does not have permission to access NVIDIA GPU Performance Counters on the target device.

I tried to follow the steps here: https://developer.nvidia.com/nvidia-development-tools-solutions-err_nvgpuctrperm-permission-issue-performance-counters

I did:

systemctl isolate multi-user

modprobe -r nvidia_uvm nvidia_drm nvidia_modeset nvidia-vgpu-vfio nvidia

modprobe nvidia NVreg_RestrictProfilingToAdminUsers=0

but the last command resulted in the message: modprobe: ERROR: could not insert 'nvidia': Unknown symbol in module, or unknown parameter (see dmesg)

I run: dmesg

And see:

[ 530.581447] nvidia: Unknown parameter“NVreg_RestrictProfilingToAdminUsers’ [ 530.763627] nvidia: Unknown parameter “NVreg_RestrictProfilingToAdminUsers' [ 531.799233] nvidia: Unknown parameter “NVreg_RestrictProfilingToAdminUsers’ [ 532.733628] nvidia: Unknown parameter “NVreg_RestrictProfilingToAdminUsers' [ 533.653000] nvidia: Unknown parameter “NVreg_RestrictProfilingToAdminUsers’ [ 534.618366] nvidia: Unknown parameter “NVreg_RestrictProfilingToAdminUsers' [ 535.529668] nvidia: Unknown parameter “NVreg_RestrictProfilingToAdminUsers’ [ 536.579298] nvidia: Unknown parameter “NVreg_RestrictProfilingToAdminUsers' [ 537.587419] nvidia: Unknown parameter “NVreg_RestrictProfilingToAdminUsers’ [ 538.515868] nvidia: Unknown parameter “NVreg_RestrictProfilingToAdminUsers' [ 539.432322] nvidia: Unknown parameter “NVreg_RestrictProfilingToAdminUsers’`

Any advice how to solve this issue?

2

what driver do you have installed?

3

How do I check that?

4

run nvidia-smi

5

Driver Version: 418.165.02 CUDA Version: 10.1

6

I’m not sure what the problem is. I can think of a couple things you could try:

1, Update the driver to a newer version (e.g. latest for your GPU) and try again
2. Use the alternate method suggested, i.e.:

Alternatively, A file containing ‘options nvidia “NVreg_RestrictProfilingToAdminUsers=0”’ may be saved to /etc/modprobe.d

Note the use of the double quotes. The outer single quotes are used to delineate the text to put in the file and should not be included in the file itself. In my opinion this is the preferred method anyway, because it will persist through a machine reboot.

7

It turns out for this to work need newer driver
I install CUDA 11.2 which comes with a higher version of the driver. Now profiling works for a non admin user