"NvRmMemInitNvmap failed with Permission denied" error when running nvidia-docker in rootless mode on Jetson Orin Nano

Autonomous Machines Jetson & Embedded Systems Jetson Orin Nano
1

I’m using a Jetson Orin Nano Developer Kit with Jetpack Version 6.1+b123 installed. I created a new user “test_user” and installed a rootless docker engine (Version: 27.4.1) on this user following: Rootless mode | Docker Docs .

I’m trying to run a (custom) docker image based on l4t-jetpack:r36.4.0 (where I just install pytorch to test gpu availability) which fails with the following error:

“”"NvRmMemInitNvmap failed with Permission denied
356: Memory Manager Not supported

NvRmMemMgrInit failed error type: 196626

/usr/local/lib/python3.10/dist-packages/torch/cuda/init.py:129: UserWarning: CUDA initialization: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero. (Triggered internally at /opt/pytorch/pytorch/c10/cuda/CUDAFunctions.cpp:108.)
return torch._C._cuda_getDeviceCount() > 0
Traceback (most recent call last):
File “”, line 1, in “”"

I built the same image with the rootful docker engine where the container has access to the gpu without any problems…

I’m using the built-in nvidia-container-toolkit (Version: 1.14.2-1) and configured the runtime for rootless mode following: Installing the NVIDIA Container Toolkit — NVIDIA Container Toolkit 1.17.3 documentation . The user “test_user” is member of the sudo, video and i2c groups.

I read here: JetPack 6.3 containerd and kubernetes - #12 by AastaLLL that there are issues with Orin series regarding nvidia-container-cli and kubernetes, but the solution there didn’t help me.

3

Hi,

Please find the below link to setup docker:

Thanks.

4

Sorry but I don’t see how this link should help me with my problem: nvidia-container-toolkit and docker were already installed with Jetpack 6.1 where the GPU access works without any problems.

I then create a new user where I installed a rootless docker engine. The rootless docker engine works but has no access to the GPU with the error I mentioned above.

5

Hi,

Do you want to run the docker without using root or need the rootless docker?

In the above document, the steps show how to set up the docker without using sudo.
Is this sufficient for you?

  1. Restart the Docker service and add your user to the docker group, so that you don’t need to use the command with sudo .
    …

Thanks.

6

I need both rootful and rootless docker as seperate docker engines for my specific application.

7

Hi,

To run a standard docker tool with a $USER account should work.
But we need to check with our internal team for the rootless docker.

Will keep you updated.

Thanks.

1 Like