Any solutions? Similar issues here. I also tried following command line in host to grant permission to all users:
systemctl isolate multi-user
modprobe -r nvidia_uvm nvidia_drm nvidia_modeset nvidia
modprobe nvidia NVreg_RestrictProfilingToAdminUsers=0
systemctl isolate graphical
Still not work.
OS: Ubuntu 16.04
Card: Tesla P100
CUDA Version: 10.2 (Host)
Driver version 440.95.01
Docker Image: Nvidia NGC nvcr.io/nvidia/tensorrt:20.02-py3 (cuda verison 10.2)