We have some Windows machines with 8x V100 GPUs. These machines are shared by multiple users (via HTCondor). However, individual GPUs are never shared, and always run in EXCLUSIVE_PROCESS mode.
We have some users who want to use CUPTI profiling.
However, they run as non-admin, and when they try to use CUPTI they get error
2021-06-25 15:17:54.403117: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1441] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI_ERROR_INSUFFICIENT_PRIVILEGES
It says at NVIDIA Development Tools Solutions - ERR_NVGPUCTRPERM: Permission issue with Performance Counters | NVIDIA Developer that you can open up performance counters to everyone via a setting in NVIDIA control panel.
However the control panel won’t start. It fails with “You are not currently using a display attached to an NVIDIA GPU”. This is true even if you run as administrator.
What do you suggest? It would be great if you could set this via a script rather than the control panel GUI. Ideally nvidia-smi would support setting this. Alternatively, is this setting in the registry somewhere?