Using CUPTI from a non-admin user on Windows

Hi,

We have some Windows machines with 8x V100 GPUs. These machines are shared by multiple users (via HTCondor). However, individual GPUs are never shared, and always run in EXCLUSIVE_PROCESS mode.

We have some users who want to use CUPTI profiling.

However, they run as non-admin, and when they try to use CUPTI they get error

2021-06-25 15:17:54.403117: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1441] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI_ERROR_INSUFFICIENT_PRIVILEGES

It says at NVIDIA Development Tools Solutions - ERR_NVGPUCTRPERM: Permission issue with Performance Counters | NVIDIA Developer that you can open up performance counters to everyone via a setting in NVIDIA control panel.

However the control panel won’t start. It fails with “You are not currently using a display attached to an NVIDIA GPU”. This is true even if you run as administrator.

What do you suggest? It would be great if you could set this via a script rather than the control panel GUI. Ideally nvidia-smi would support setting this. Alternatively, is this setting in the registry somewhere?

Thanks,

Rob

This can be set in the registry as described at Microsoft Windows Server :: NVIDIA Virtual GPU Software Documentation

[HKLM\SYSTEM\CurrentControlSet\Services\nvlddmkm\Global\NVTweak]
Value: “RmProfilingAdminOnly”
Type: DWORD
Data: 00000000