Docker and Container Toolkit on Yocto

Autonomous Vehicles DRIVE AGX Thor DRIVE AGX Thor General
1

DRIVE OS Version: 7.0.3

Issue Description:

I’m currently trying to run Docker with GPU access on DriveOS’s Yocto build.
I’ve added the meta-virtualization layer and have successfully built Docker into my image which is working.
I’m able to create and run containers with the default runtime without issue.

I next installed NVIDIA container toolkit (version 1.19.0) and its required dependencies (libcap2 and libseccomp2) from it’s ARM64 .deb package that I downloaded from the GitHub releases page: Releases · NVIDIA/nvidia-container-toolkit · GitHub

I run the following to configure Docker to use the runtime:

nvidia-ctk runtime configure --runtime=docker
systemctl restart docker

At this point, I’m able to start containers using the NVIDIA runtime:

sudo docker run --rm --privileged --net host --runtime nvidia -it ubuntu

Attempting to run a container with access the GPU however results in the container immediately stopping with exit code 135.
There are no logs I could find that provide any details or indicate what the exact issue is.

sudo docker run --rm --privileged --net host --runtime nvidia --gpus all -it ubuntu

As Docker and NVIDIA Container Toolkit works on DriveOS’s Ubuntu installation, it in theory should be possible on Yocto as well. Is there any additional configurations done on Ubuntu that makes GPU access from containers possible?

I’m aware that Docker/CTK isn’t supported on DriveOS Yocto but any suggestions would be greatly appreciated. Thanks in advance.

5

Dear @andy188 ,
Could you please share the followed steps and logs to review/repro for further guidance from engineering team
Note that docker support on drive is for experimental purpose and not all features expected to work.

6

Steps to reproduce:

  1. Set up Yocto build environment as per described in To build NVIDIA Yocto Project-based components — NVIDIA DriveOS 7.0.3 Linux SDK Developer Guide
    • Not using dual branch support, Scarthgap only
    • Using default IMAGE_TYPE, KERNEL_CONFIG_TYPE, COPYTARGET_VARIANT, PCT
  2. Modify environment to add Docker
    1. Clone meta-virtualization layer
      $ cd /drive/drive-linux_src/yocto/oss/scarthgap
$ git clone https://git.yoctoproject.org/meta-virtualization -b scarthgap
    2. Modify /drive/drive-linux_src/yocto/build/bblayers.conf
      # add to BBLAYERS (line 34)
/drive/drive-linux_src/yocto/oss/scarthgap/poky/../meta-virtualization \
    3. Modify /drive/drive-linux_src/yocto/build/local.conf
      # add the following
DISTRO_FEATURES:append = " virtualization"
KERNEL_FEATURES:append = " features/docker/docker.cfg"
    4. Modify /drive/drive-linux_src/yocto/layers/meta-drive/recipes-core/images/tegra-drive-os-av-image.bb
      IMAGE_INSTALL:append = " docker-moby"
  3. Add additional CUDA, CuDNN, TensorRT packages (I think only CUDA matters here) as described in the following:
  4. Start building the images
    $ bitbake tegra-drive-os-av-image
$ bitbake tegra-initramfs-boot
  5. Flash the images to Thor as described in Flashing Yocto Built Images via Bootburn — NVIDIA DriveOS 7.0.3 Linux SDK Developer Guide
    • board variant = p3960-10-sw03
  6. Reset MCU to boot into Yocto
  7. Update /etc/resolv.conf
    echo "nameserver 1.1.1.1" > /etc/resolv.conf
  8. Download packages/dependencies to Thor
  9. Install Packages
    $ dpkg -i libcap2_2.78-1_arm64.deb
$ dpkg -i libseccomp2_2.6.1-1_arm64.deb
$ tar -zxf nvidia-container-toolkit_1.19.0_deb_arm64.tar.gz
$ cd release-v1.19.0-stable/packages/ubuntu18.04/arm64/
$ dpkg -i nvidia-container-toolkit-base_1.19.0-1_arm64.deb
$ dpkg -i libnvidia-container1_1.19.0-1_arm64.deb
$ dpkg -i libnvidia-container-tools_1.19.0-1_arm64.deb
$ dpkg -i nvidia-container-toolkit_1.19.0-1_arm64.deb
  10. Setup/Start Docker
    $ nvidia-ctk runtime configure --runtime=docker
$ systemctl restart docker
  11. Run container
    root@p3960-0010:~# sudo docker run --privileged --net host --runtime nvidia --hostname ubuntu -it ubuntu:latest
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
55237ac9880d: Pull complete 
693710ba2039: Pull complete 
Digest: sha256:3131b4cc82a783df6c9df078f86e01819a13594b865c2cad47bd1bca2b7063bb
Status: Downloaded newer image for ubuntu:latest
root@ubuntu:/#
root@ubuntu:/# exit
root@p3960-0010:~# sudo docker run --privileged --net host --runtime nvidia --gpus all --hostname ubuntu -it ubuntu:latest
root@p3960-0010:~#
root@p3960-0010:~# docker ps -a
CONTAINER ID   IMAGE           COMMAND       CREATED          STATUS                          PORTS     NAMES
671c36a25e4b   ubuntu:latest   "/bin/bash"   25 seconds ago   Exited (135) 24 seconds ago               gracious_williamson
7

The immediate container exit seems to be related to the file /etc/nvidia-container-runtime/host-files-for-container.d/drivers.csv.

This file is getting generated automatically on boot, and I understand that it is expected to be modified to add whatever is needed for a given GPU-accelerated application, but what are the minimum libraries, files, directories that are required?

Removing the following from the file seems to stop the container from immediately exiting but CUDA/GPU access doesn’t work correctly when doing so.

lib, /usr/lib/ld-linux-aarch64.so.1
lib, /lib/ld-linux-aarch64.so.1

Trying to run the cudaQuery sample application described in this Target Container Support documentation prints out the following which isn’t correct.

Running Sample Cuda Query... 

Device 0: 
CUDA Capability major/minor: 1.65535

The drivers.csv file does seem to be generated based on the contents of the host system, where the cudaQuery application is working correctly, so it isn’t quite clear what the issue is.

8

Does that mean you notice different CUDA capability only when running via docker and sample is giving correct output if run on target directly?

10

Does that mean you notice different CUDA capability only when running via docker and sample is giving correct output if run on target directly?

That is correct. Running the sample directly on the host results in the following:

root@p3960-0010:~# ./cudaQuery 
Running Sample Cuda Query... 

Device 0: Thor
CUDA Capability major/minor: 10.1
12

Let me check with latest release and get back to you on this.