This is a follow up question to this topic: "NvRmMemInitNvmap failed with Permission denied" error when running nvidia-docker in rootless mode on Jetson Orin Nano
Thanks to Nvidia’s support we are now able to access the GPU in a rootless docker container (running on a rootless user). However, for our very specific application we need to be able to access the GPU in rootless docker in docker.
To achieve this I created a docker image based on http://nvcr.io/nvidia/l4t-cuda:12.6.11-runtime and install docker and add the executable “deviceQuery” to check the availability of the GPU: create_nvidia_dind.zip (9.4 KB)
When running this image in rootful docker with “docker run --runtime=nvidia --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock -it image_name”, I can then start a docker container inside the container using the same image with “docker run --runtime=nvidia -it image_name” and checking with “deviceQuery” shows that the GPU can be accessed in rootful docker in docker.
However, this does not work for rootless docker:
I start the outer container using “docker run -it --mount type=bind,src=/var/run/user/1001/docker.sock,dst=/var/run/docker.sock --rm --net=host --runtime nvidia --group-add=video --group-add=104 image_name” and can access the gpu in this container. However, when starting the docker container in the container (again using the same image) with “docker run --runtime=nvidia -it image_name”, the “deviceQuery” executable returns:
cudaGetDeviceCount returned 801
→ operation not supported
Result = FAIL
What configurations do I need to set in the outer docker container or how do I need to run the inner docker conainer, such that the GPU is accessible in rootless docker in docker?
Hi,
We need to discuss this internally.
Will provide more info to you shortly.
Thanks.
Hi,
Since running rootless docker inside docker is not a common usage we expect.
You will need to try it on your own.
Maybe other users can share their experience.
Thanks.
Hi,
We are going to test this issue internally.
Will provide more info with you later.
Thanks.
1 Like
Hi,
Sorry for the late update.
We test your Dockerfile and it installs the default docker inside the rootless docker.
$ docker run -it --rm --runtime=nvidia -v /var/run/docker.sock:/var/run/docker.sock dind:latest
..
root@c5327f106d29:/# docker info
Client: Docker Engine - Community
Version: 28.2.2
Context: default
Debug Mode: false
Suppose default docker requires more system permission compared to the rootless docker.
Could you try to install a rootless docker inside the rootless docker to see if it can work?
Thanks.
Hi!
I ran the following command:
$ docker run -it --rm --runtime=nvidia -v /var/run/user/1001/docker.sock:/var/run/docker.sock docker:latest
which gives me the following error:
Certificate request self-signature ok
subject=CN=docker:dind server
/certs/server/cert.pem: OK
Certificate request self-signature ok
subject=CN=docker:dind client
/certs/client/cert.pem: OK
cat: can’t open ‘/proc/net/arp_tables_names’: No such file or directory
ip: can’t find device ‘nf_tables’
modprobe: can’t change directory to ‘/lib/modules’: No such file or directory
ip: can’t find device ‘ip_tables’
ip_tables 32768 2 iptable_nat,iptable_filter
x_tables 45056 9 xt_conntrack,xt_MASQUERADE,ip6table_nat,ip6table_filter,ip6_tables,iptable_nat,xt_addrtype,iptable_filter,ip_tables
modprobe: can’t change directory to ‘/lib/modules’: No such file or directory
ip: can’t find device ‘ip6_tables’
ip6_tables 32768 2 ip6table_nat,ip6table_filter
x_tables 45056 9 xt_conntrack,xt_MASQUERADE,ip6table_nat,ip6table_filter,ip6_tables,iptable_nat,xt_addrtype,iptable_filter,ip_tables
modprobe: can’t change directory to ‘/lib/modules’: No such file or directory
iptables v1.8.11 (nf_tables)
mount: permission denied (are you root?)
Could not mount /sys/kernel/security.
AppArmor detection and --privileged mode might break.
mount: permission denied (are you root?)
So it cannot even start the container, not to mention accessing to the GPU. Note that we want to use the rootless docker, therefore I try to mount the rootless socket into the container.
Hi,
Do you install a rootless docker inside the container?
If so, could you share the Dockerfile with us to try?
The previous one used the standard docker.
Thanks.
True, I install standard Docker in the container. But rootless docker inside the container does not really make sense, as there is only a root user in the container.
Or do I misunderstand your point?
So the setup we aim for is: have a non-sudo user with a rootless docker engine installed, be able to mount the gpu inside a rootless docker container and be able to start a docker container insider the rootless docker container and mount the gpu into the container in the container. So we want to mount the gpu into a (rootful) docker container inside a rootless docker container.
Hi,
Thanks for your patience.
Sorry that we don’t have much experience with the dind use case so need to investigate further.
Will update more info with you once we have progress.
Thanks.
1 Like