Let’s break down the problem with docker-in-docker a bit.
Build the following image:
docker build -t test-dind - <<EOF
FROM ubuntu
RUN apt-get update && \
apt-get -y install ca-certificates curl && \
install -m 0755 -d /etc/apt/keyrings && \
curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc && \
chmod a+r /etc/apt/keyrings/docker.asc
RUN echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
tee /etc/apt/sources.list.d/docker.list > /dev/null && \
apt-get update
RUN apt-get -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
EOF
Start an outer docker-in-docker container using this image:
docker run --rm -d --name outer \
--mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \
--runtime=nvidia \
-e NVIDIA_VISIBLE_DEVICES=all \
test-dind \
sleep infinity
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
920fa4ede65c test-dind "sleep infinity" 3 seconds ago Up 2 seconds outer
Note that this same container is visible from the container itself.
$ docker exec -ti outer docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
920fa4ede65c test-dind "sleep infinity" About a minute ago Up About a minute outer
If we now start a new container in the outer container (note that we don’t mount the docker socket and we’re running a standard ubuntu container):
docker exec -ti outer \
docker run --rm -d --name inner \
--runtime=nvidia \
-e NVIDIA_VISIBLE_DEVICES=all \
ubuntu \
sleep infinity
We see the following on the host:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a973c2e7ec22 ubuntu "sleep infinity" 4 seconds ago Up 3 seconds inner
920fa4ede65c test-dind "sleep infinity" 3 minutes ago Up 3 minutes outer
and once again this is visible to the outer container:
$ docker exec -ti outer docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a973c2e7ec22 ubuntu "sleep infinity" 53 seconds ago Up 52 seconds inner
920fa4ede65c test-dind "sleep infinity" 4 minutes ago Up 4 minutes outer
Note that even containers started on the host will be visible in the outer container:
$ docker run --rm -d --name external ubuntu sleep infinity
$ docker exec -ti outer docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d6b7bc11b5e2 ubuntu "sleep infinity" 5 seconds ago Up 4 seconds external
a973c2e7ec22 ubuntu "sleep infinity" 2 minutes ago Up 2 minutes inner
920fa4ede65c test-dind "sleep infinity" 5 minutes ago Up 5 minutes outer
This is to say, for the setup you’re describing, it is still the docker daemon on the host
that is running the inner container. Here I would expect the arguments that you pass to the
inner container to match the ones that you would pass on the host – although I
don’t have a definitive answer here in the rootless case.
What may be an option is to start the outer container WITHOUT injecting the
docker socket and then install docker in this container and start the docker
daemon there. (This is something similar to what kind does, but in the container
runtime that is installed is containerd specifically).
Note that at least the DEFAULT configuration requires a --privileged container:
docker run --privileged --rm -d --name outer \
--runtime=nvidia \
-e NVIDIA_VISIBLE_DEVICES=all \
test-dind \
dockerd
One can then start other containers in the outer container, but note that
in order to enable GPU support, the NVIDIA Container Toolkit needs to be installed
in this container. In the case of Tegra-based systems, the CSV files from
/etc/nvidia-container-runtime/host-files-for-container.d/ (or the whole folder)
would also need to be mounted into the outer container (or added through some other mechanism)
for device detection to work properly