"NvRmMemInitNvmap failed with Permission denied" error when running nvidia-docker in rootless mode on Jetson Orin Nano

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin Nano
1

I’m using a Jetson Orin Nano Developer Kit with Jetpack Version 6.1+b123 installed. I created a new user “test_user” and installed a rootless docker engine (Version: 27.4.1) on this user following: Rootless mode | Docker Docs .

I’m trying to run a (custom) docker image based on l4t-jetpack:r36.4.0 (where I just install pytorch to test gpu availability) which fails with the following error:

“”"NvRmMemInitNvmap failed with Permission denied
356: Memory Manager Not supported

NvRmMemMgrInit failed error type: 196626

/usr/local/lib/python3.10/dist-packages/torch/cuda/init.py:129: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at /opt/pytorch/pytorch/c10/cuda/CUDAFunctions.cpp:108.)
return torch._C._cuda_getDeviceCount() > 0
Traceback (most recent call last):
File “”, line 1, in “”"

I built the same image with the rootful docker engine where the container has access to the gpu without any problems…

I’m using the built-in nvidia-container-toolkit (Version: 1.14.2-1) and configured the runtime for rootless mode following: Installing the NVIDIA Container Toolkit — NVIDIA Container Toolkit 1.17.3 documentation . The user “test_user” is member of the sudo, video and i2c groups.

I read here: JetPack 6.3 containerd and kubernetes - #12 by AastaLLL that there are issues with Orin series regarding nvidia-container-cli and kubernetes, but the solution there didn’t help me.

3

Hi,

Please find the below link to setup docker:

Thanks.

4

Sorry but I don’t see how this link should help me with my problem: nvidia-container-toolkit and docker were already installed with Jetpack 6.1 where the GPU access works without any problems.

I then create a new user where I installed a rootless docker engine. The rootless docker engine works but has no access to the GPU with the error I mentioned above.

5

Hi,

Do you want to run the docker without using root or need the rootless docker?

In the above document, the steps show how to set up the docker without using sudo.
Is this sufficient for you?

  1. Restart the Docker service and add your user to the docker group, so that you don’t need to use the command with sudo .
    …

Thanks.

6

I need both rootful and rootless docker as seperate docker engines for my specific application.

7

Hi,

To run a standard docker tool with a $USER account should work.
But we need to check with our internal team for the rootless docker.

Will keep you updated.

Thanks.

1 Like
10

Update

rootless docker requires extra config and such custom config is not verified in our QA process.
So for forum users, we recommend using rootful docker instead.

Thanks.

11

Hi,

Could you help to test if the rootless docker can work with our official container?
We can access the GPU without issue inside the pytorch:24.12-py3-igpu container:

$ docker run -it --rm --net=host --runtime nvidia -e DISPLAY=$DISPLAY -v /home/nvidia/cuda-samples:/home/nvidia/cuda-samples -v /tmp/.X11-unix/:/tmp/.X11-unix nvcr.io/nvidia/pytorch:24.12-py3-igpu

=============
== PyTorch ==
=============

NVIDIA Release 24.12 (build 126674151)
PyTorch Version 2.6.0a0+df5bbc0
Container image Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
Copyright (c) 2014-2024 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies    (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU                      (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006      Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015      Google Inc.
Copyright (c) 2015      Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

NOTE: The SHMEM allocation limit is set to the default of 64MB.  This may be
   insufficient for PyTorch.  NVIDIA recommends the use of the following flags:
   docker run --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 ...

root@tegra-ubuntu:/workspace# python3
Python 3.12.3 (main, Nov  6 2024, 18:32:19) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.rand(10).to(torch.device("cuda"))
tensor([0.1448, 0.6821, 0.3180, 0.2998, 0.6193, 0.1839, 0.4616, 0.3445, 0.3689,
        0.2040], device='cuda:0')
>>>

Thanks.

12

Hi,

unfortunately it does not work for me:

$ docker run -it --rm --net=host --runtime nvidia -e DISPLAY=$DISPLAY -v /tmp/.X11-unix/:/tmp/.X11-unix nvcr.io/nvidia/pytorch:24.12-py3-igpu

=============
== PyTorch ==

NVIDIA Release 24.12 (build 126674151)
PyTorch Version 2.6.0a0+df5bbc0
Container image Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
Copyright (c) 2014-2024 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006 Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015 Google Inc.
Copyright (c) 2015 Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:

ERROR: The NVIDIA Driver is present, but CUDA failed to initialize. GPU functionality will not be available.
[[ Operation not supported (error 801) ]]

NVIDIA Tegra driver detected.

NOTE: The SHMEM allocation limit is set to the default of 64MB. This may be
insufficient for PyTorch. NVIDIA recommends the use of the following flags:
docker run --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 …

root@industrial:/workspace# python3
Python 3.12.3 (main, Nov 6 2024, 18:32:19) [GCC 13.2.0] on linux
Type “help”, “copyright”, “credits” or “license” for more information.

import torch
print(torch.cuda.is_available())
/usr/local/lib/python3.12/dist-packages/torch/cuda/init.py:129: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 801: operation not supported (Triggered internally at /opt/pytorch/pytorch/c10/cuda/CUDAFunctions.cpp:108.)
return torch._C._cuda_getDeviceCount() > 0
False

13

Hi,

Could you share the docker info of your environment with us?

$ docker info

Thanks.

14

Hi there,

It seems like you’re encountering a permission issue when trying to run a container with GPU support in rootless mode, which is a common challenge with rootless Docker setups. The error NvRmMemInitNvmap failed with Permission denied indicates that the container doesn’t have the correct permissions to access the GPU resources.

Here are a few things you can try:

Check Permissions: Even though your test_user is part of the sudo, video, and i2c groups, sometimes rootless Docker containers may still lack necessary permissions to access the GPU. You can try adding the user to the docker group, and ensure the environment is set correctly, especially CUDA_VISIBLE_DEVICES.

Adjust Docker Runtime Configurations: You mentioned using nvidia-container-toolkit, but in rootless mode, you may need to make sure that nvidia-docker and the NVIDIA runtime are configured to allow GPU access in a rootless setup. Double-check that the correct runtime (nvidia) is specified in the Docker config files for your rootless Docker setup.

Test on a Different Version: There might be a compatibility issue between the Jetson Orin and the version of the NVIDIA container toolkit you’re using. Sometimes, upgrading or downgrading the toolkit can resolve these types of issues. Make sure that the versions of nvidia-container-toolkit and docker are fully compatible with JetPack 6.1.

Check for Known Issues: Since you’ve already read the thread on the JetPack 6.3 containerd and Kubernetes issues, I would suggest keeping an eye on any official patches or updates released by NVIDIA for better support in rootless Docker mode. Sometimes, these updates address specific compatibility problems with newer hardware.

For additional support, you can reach out to Bettlink, which is a trusted supplier for various electronic components and developer kits. They might have insights into compatible hardware or kits that could support smoother GPU access in such environments. Visit Bettlink.com for more details on their offerings.

Best of luck with your setup, and I hope this helps resolve the issue!

1 Like
15

Hi,

I had to flash my Jetson Orin Nano DevBoard, since some standard ubuntu updates killed my docker. Anyways, I have now Jetpack Version 6.2+b77 together with nvidia-container-toolkit Version 1.16.2-1 installed.

When I run the official pytorch container with rootless docker, the error is now more elaborate:

$ docker run -it --rm --net=host --runtime nvidia -e DISPLAY=$DISPLAY -v /tmp/.X11-unix/:/tmp/.X11-unix nvcr.io/nvidia/pytorch:24.12-py3-igpu
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as ‘legacy’
NvRmMemInitNvmap failed with Permission denied
356: Memory Manager Not supported

NvRmMemMgrInit failed error type: 196626

libnvrm_gpu.so: NvRmGpuLibOpen failed, error=196626
NvRmMemInitNvmap failed with Permission denied
356: Memory Manager Not supported

NvRmMemMgrInit failed error type: 196626

libnvrm_gpu.so: NvRmGpuLibOpen failed, error=196626
NvRmMemInitNvmap failed with Permission denied
356: Memory Manager Not supported

NvRmMemMgrInit failed error type: 196626

libnvrm_gpu.so: NvRmGpuLibOpen failed, error=196626
nvidia-container-cli: detection error: nvml error: unknown error: unknown.

And what you requested:
$ docker info
Client: Docker Engine - Community
Version: 27.5.0
Context: rootless
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.19.3
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.32.3
Path: /usr/libexec/docker/cli-plugins/docker-compose

Server:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 2
Server Version: 27.5.0
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: true
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 nvidia runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
runc version: v1.2.4-0-g6c52b3f
init version: de40ad0
Security Options:
seccomp
Profile: builtin
rootless
cgroupns
Kernel Version: 5.15.148-tegra
Operating System: Ubuntu 22.04.5 LTS
OSType: linux
Architecture: aarch64
CPUs: 6
Total Memory: 7.441GiB
Name: johannes-desktop
ID: 5805035d-80f5-47e7-8654-8f3bc8678d6b
Docker Root Dir: /home/test_user/.local/share/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support

16

Hi,

It looks like we have different permission settings.
Please try our rules file and test it again.

Edit /etc/udev/rules.d/99-tegra-devices.rules
99-tegra-devices.rules (5.4 KB)

$ sudo udevadm control --reload-rules && sudo udevadm trigger

Thanks.

17

Hi,

sorry for my late reply! We again tested on two different devices (both with JP 6.2 installed):

One device gave this error message:

"ERROR: The NVIDIA Driver is present, but CUDA failed to initialize. GPU functionality will not be available.
[[ Operation not supported (error 801) ]]

NVIDIA Tegra driver detected.

NOTE: The SHMEM allocation limit is set to the default of 64MB. This may be
insufficient for PyTorch. NVIDIA recommends the use of the following flags:
docker run --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 …"

the other one gave this:

“docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as ‘legacy’
nvidia-container-cli: detection error: nvml error: unknown error: unknown.”

18

Hi,

Did you try with the below command?

$ docker run -it --rm --net=host --runtime nvidia -e DISPLAY=$DISPLAY -v /tmp/.X11-unix/:/tmp/.X11-unix nvcr.io/nvidia/pytorch:24.12-py3-igpu

Thanks.

19

Yes we did.

Alsp pls find here my groups configuration:

root:x:0:
daemon:x:1:
bin:x:2:
sys:x:3:
adm:x:4:syslog,johannes
tty:x:5:
disk:x:6:
lp:x:7:
mail:x:8:
news:x:9:
uucp:x:10:
man:x:12:
proxy:x:13:
kmem:x:15:
dialout:x:20:
fax:x:21:
voice:x:22:
cdrom:x:24:johannes
floppy:x:25:
tape:x:26:
sudo:x:27:johannes,test_user
audio:x:29:pulse,johannes
dip:x:30:johannes
www-data:x:33:
backup:x:34:
operator:x:37:
list:x:38:
irc:x:39:
src:x:40:
gnats:x:41:
shadow:x:42:
utmp:x:43:
video:x:44:gdm,johannes,test_user
sasl:x:45:
plugdev:x:46:johannes
staff:x:50:
games:x:60:
users:x:100:
nogroup:x:65534:
input:x:101:
sgx:x:102:
kvm:x:103:
render:x:104:johannes
_ssh:x:105:
dhcpd:x:106:
netdev:x:107:
systemd-journal:x:108:
systemd-network:x:109:
systemd-resolve:x:110:
messagebus:x:111:
systemd-timesync:x:112:
rdma:x:113:
syslog:x:114:
bluetooth:x:115:
i2c:x:116:johannes,test_user
rtkit:x:117:
tss:x:118:
crontab:x:119:
lpadmin:x:120:johannes
ssl-cert:x:121:
systemd-oom:x:122:
geoclue:x:124:
pulse:x:125:
pulse-access:x:126:
scanner:x:127:saned
avahi-autoipd:x:128:
saned:x:129:
nm-openvpn:x:130:
colord:x:131:
avahi:x:132:
sssd:x:133:
fwupd-refresh:x:134:
gdm:x:135:johannes
uuidd:x:123:
gpio:x:999:johannes
trusty:x:998:
crypto:x:997:
weston-launch:x:996:johannes
docker:x:995:johannes,test_user
debug:x:994:
johannes:x:1000:
sambashare:x:136:johannes
test_user:x:1001:

20

Hi,

We test this on another device and it can work normally.
Here are the detailed steps to enable GPU with rootless docker.
Hope this will help:

rootless docker

$ sudo apt-get install -y dbus-user-session
$ sudo apt-get install -y uidmap
$ sudo apt-get install -y systemd-container

$ sudo systemctl disable --now docker.service docker.socket
$ sudo rm /var/run/docker.sock
$ dockerd-rootless-setuptool.sh install

nvidia-container-toolkit

$ nvidia-ctk runtime configure --runtime=docker --config=$HOME/.config/docker/daemon.json
$ systemctl --user restart docker
$ sudo nvidia-ctk config --set nvidia-container-cli.no-cgroups --in-place

Configure

Find the 99-tegra-devices.rules on the comment Feb 25.

$ sudo mv /etc/udev/rules.d/99-tegra-devices.rules  /etc/udev/rules.d/99-tegra-devices.rules_bk
$ sudo mv [99-tegra-devices.rules] /etc/udev/rules.d/99-tegra-devices.rules
$ sudo udevadm control --reload-rules && sudo udevadm trigger

Test

$ docker info
Client: Docker Engine - Community
 Version:    28.0.1
 Context:    rootless
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.21.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.33.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 28.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc version: v1.2.4-0-g6c52b3f
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 5.15.148-tegra
 Operating System: Ubuntu 22.04.5 LTS
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 15.29GiB
 Name: tegra-ubuntu
 ID: 979fd791-f729-414f-8420-229c8bccf181
 Docker Root Dir: /home/nvidia/.local/share/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support

$ docker run -it --rm --net=host --runtime nvidia -e DISPLAY=$DISPLAY -v /tmp/.X11-unix/:/tmp/.X11-unix nvcr.io/nvidia/pytorch:24.12-py3-igpu

=============
== PyTorch ==
=============

NVIDIA Release 24.12 (build 126674151)
PyTorch Version 2.6.0a0+df5bbc0
Container image Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
Copyright (c) 2014-2024 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies    (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU                      (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006      Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015      Google Inc.
Copyright (c) 2015      Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

NOTE: The SHMEM allocation limit is set to the default of 64MB.  This may be
   insufficient for PyTorch.  NVIDIA recommends the use of the following flags:
   docker run --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 ...

root@tegra-ubuntu:/workspace# python3
Python 3.12.3 (main, Nov  6 2024, 18:32:19) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.rand(10).to(torch.device("cuda"))
tensor([0.1687, 0.3456, 0.6423, 0.1486, 0.8193, 0.6526, 0.5585, 0.8619, 0.7614,
        0.4501], device='cuda:0')
>>>

Thanks.

21

Ah yes, now I know what we did differently:

I always created an additional test_user whereas you always tested on the standard (ubuntu) user.

Indeed following your instructions on the standard user I can now access the gpu even in rootless docker. Thx for providing this solution.

However, when testing on a second user (test_user):
$ sudo machinectl shell test_user@
$ docker run -it --rm --net=host --runtime nvidia -e DISPLAY=$DISPLAY -v /tmp/.X11-unix/:/tmp/.X11-unix nvcr.io/nvidia/pytorch:24.12-py3-igpu

I get the error I already mentioned above:
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #0: exit status 1, stdout: , stderr: Auto-detected mode as ‘legacy’
nvidia-container-cli: detection error: nvml error: unknown error: unknown

The group settings are as mentioned above.

Ideally what I would like to do is to run the rootless docker on a second user who is not even member of the sudo group…

22

Hi,

We try a second user but can still get GPU access with the rootless docker.
It looks like the $USER is added to some default group when generating.

Could you check if any difference between us?
The test_user doesn’t belong to the sudo group.

$ sudo adduser test_user
Adding user `test_user' ...
Adding new group `test_user' (1002) ...
Adding new user `test_user' (1001) with group `test_user' ...
Creating home directory `/home/test_user' ...
Copying files from `/etc/skel' ...
New password: 
BAD PASSWORD: The password fails the dictionary check - it is based on a dictionary word
Retype new password: 
passwd: password updated successfully
Changing the user information for test_user
Enter the new value, or press ENTER for the default
	Full Name []: test_user
	Room Number []: 
	Work Phone []: 
	Home Phone []: 
	Other []: 
Is the information correct? [Y/n] y
Adding new user `test_user' to extra groups ...
Adding user `test_user' to group `audio' ...
Adding user `test_user' to group `gdm' ...
Adding user `test_user' to group `gpio' ...
Adding user `test_user' to group `i2c' ...
adduser: The group `lightdm' does not exist.
Adding user `test_user' to group `render' ...
Adding user `test_user' to group `video' ...
Adding user `test_user' to group `weston-launch' ...

Then login with test_user:

$ echo ${USER}
test_user

$ groups 
test_user audio video render i2c gdm weston-launch gpio

Client: Docker Engine - Community
 Version:    28.0.1
 Context:    rootless
 Debug Mode: false

$ docker run -it --rm --net=host --runtime nvidia -e DISPLAY=$DISPLAY -v /tmp/.X11-unix/:/tmp/.X11-unix nvcr.io/nvidia/pytorch:24.12-py3-igpu
 
=============
== PyTorch ==
=============

NVIDIA Release 24.12 (build 126674151)
PyTorch Version 2.6.0a0+df5bbc0
Container image Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
Copyright (c) 2014-2024 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies    (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU                      (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006      Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015      Google Inc.
Copyright (c) 2015      Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

ERROR: The NVIDIA Driver is present, but CUDA failed to initialize.  GPU functionality will not be available.
   [[ Operation not supported (error 801) ]]

Failed to detect NVIDIA driver version.

NOTE: The SHMEM allocation limit is set to the default of 64MB.  This may be
   insufficient for PyTorch.  NVIDIA recommends the use of the following flags:
   docker run --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 ...
23

Sry but before I test, your logs obviously state:

ERROR: The NVIDIA Driver is present, but CUDA failed to initialize. GPU functionality will not be available.
[[ Operation not supported (error 801) ]]