Building docker image in CI with CUDA runnable on Jetson

Hi, we have a docker image built on top of timongentzsch’s pytorch image. We are building this image via our gitlab continuous integration using qemu for aarch64 emulation and it works great. We’re able to build the image and install dependencies and everything. The snag comes in with CUDA. The images we’re using rely on the CUDA which gets passed into the container by the nvidia container toolkit at runtime and therefore isn’t there during our CI build. So any code we’d like to build also in CI will not build (unless we were to deploy a jetson in our CI server farm, which we definitely do not want to have to do). Has anyone dealt with this and have any ideas? One suggestion has been to install CUDA in the image which then makes it available at CI time, then just assume the nvidia docker runtime will stomp over that installation at container creation time, but I’m not 100% sure how all the linked libraries and such would work or if any headers or libraries may be different in the jetpack compiled version of CUDA vs a standard gcc compiled version. Any thoughts or ideas greatly appreciated.

The default image should be only run on a jetson host. If you use another host, such as x86_64 and qemu, you need to build your own image with all packages (such as cuda) directly installed. Here is what I did:

cat Dockerfile.l4t 
#
# docker build -t l4t-base-jp46 -f Dockerfile.l4t .
#

FROM nvcr.io/nvidia/l4t-base:r32.7.1

ENV DEBIAN_FRONTEND=noninteractive
ENV CUDA_HOME="/usr/local/cuda"

RUN apt-get update \
    && apt-get -y purge --auto-remove humanity-icon-theme \
        "gsettings*" "libavahi*" "console-setup*" wpasupplicant \
    && apt-get -y upgrade \
    && mkdir -p /opt/nvidia/l4t-packages \
    && touch /opt/nvidia/l4t-packages/.nv-l4t-disable-boot-fw-update-in-preinstall \
    && apt-get install -y --no-install-recommends \
         ca-certificates curl git \
         gstreamer1.0-plugins-bad gstreamer1.0-plugins-good \
         gstreamer1.0-pocketsphinx gstreamer1.0-rtsp gstreamer1.0-vaapi \
         gstreamer1.0-x libv4l-0 libv4lconvert0 libxi-dev \
         libxmu-dev vim devmem2 libgtk-3-0 \
    && cd /opt/nvidia/l4t-packages \
    && curl -sLOJ https://repo.download.nvidia.com/jetson/t194/pool/main/n/nvidia-l4t-core/nvidia-l4t-core_32.7.2-20220417024839_arm64.deb \
    && curl -sLOJ https://repo.download.nvidia.com/jetson/t194/pool/main/n/nvidia-l4t-apt-source/nvidia-l4t-apt-source_32.7.2-20220417024839_arm64.deb \
    && apt-get install -o Dpkg::Options::="--force-confold" --force-yes -y \
         ./nvidia-l4t-core*deb \
    && apt-get -y install ./nvidia-l4t-apt-source*deb \
    && sed -i "s/<SOC>/t194/g" \
         "/etc/apt/sources.list.d/nvidia-l4t-apt-source.list" \
    && rm -f ./*.deb \
    && apt-get update \
    && apt-get -y install libcublas10 cuda-cudart-10-2 \
         libcurand-10-2 libcusolver-10-2 libcusparse-10-2 \
	 libnvinfer8 cuda-command-line-tools-10-2 libcufft-10-2 \
	 libnvinfer-plugin8 cuda-nvprune-10-2 cuda-compiler-10-2 \
	 gir1.2-gstreamer-1.0 libdouble-conversion1 liblua5.2-0 \
	 libqt5gstreamer-1.0-0 \
         nvidia-l4t-3d-core nvidia-l4t-camera nvidia-l4t-configs \
	 nvidia-l4t-cuda nvidia-l4t-firmware nvidia-l4t-gputools \
	 nvidia-l4t-init nvidia-l4t-libvulkan nvidia-l4t-multimedia \
	 nvidia-l4t-multimedia-utils nvidia-l4t-tools \
	 nvidia-l4t-wayland nvidia-l4t-weston nvidia-l4t-x11 \
	 nvidia-l4t-xusb-firmware \
    && rm -f /opt/nvidia/l4t-packages/.nv-l4t-disable-boot-fw-update-in-preinstall \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

We could build a plain image but the main idea here is to build an image using CI that can then be pulled down and run on the jetson hardware. Example:

I have a project that depends on CUDA. I’d like to build a docker image which has that project and builds the project in CI, then pushes the built docker to our registry. Then I could pull down that docker and just run it on the jetson hardware. The idea being that the build project would now just link against the nvidia docker runtime loaded CUDA instead of the one we installed during CI. To be 100% honest I’m not even sure what I’m asking for is possible.

Yes, you can do two-stage build. What I posted could be used as stage 1 for your project build/compilation. The binaries can then be copied to stage 2 (nvidia provided jetson image) that is used for runtime. The final image that is pushed to the registry is mostly stage 2 image plus your application binaries.

Ahh I see, and so those compiled binaries that I built against the CI’s CUDA will link no problem to the Jetson based CUDA at container runtime? Interesting.

yes, since the first stage build container uses the exact same packages/libraries as the real jetson, your compiled binaries will just work.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.