Cmake cannot detect Cuda dev toolkit inside docker when running in Jetson AGX Xaver dev kit

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1

Hello all

I am having trouble to make the following config work:

  1. Host: A jetson AGX jetson developer kit

  2. The host is trying to act as a cuda application docker builder. Meaning it will compile some cuda codes in container/Dockerfile.

  3. The Dockerfile is way too long to share. However, the idea will be something like this

=======================
o FROM nvcr.io/nvidia/l4t-base:r32.3.1

o Install some dependencies, including cmake

o Compile our cuda application by using cmake …

=======================

This config will fail when I am detecting cuda inside cmake when running command

enable_language(“CUDA”)

with the error

“CMake Error at /usr/local/share/cmake-3.19/Modules/CMakeDetermineCUDACompiler.cmake:191 (message):
Couldn’t find CUDA library root.”

I inspect the base image and I see nvcc can be found inside docker image and can be run as well. I also specify cuda root manually when running cmake at the /usr/local/cuda and didnt help either. tried with and without
FindPackage(CUDA) isnide cmakelist and the same problem

I wonder have you had similar experience or know some ideas what is happening?

Cheers

3

Hi,

Please noted that l4t-base:r32.3.1 mounts the CUDA library from the host.
The mount is only available at running time, not building time.

So to link the CUDA in the Dockerfile, you might need to copy the whole CUDA library into the container first.
A simple workaround is to compile it on the Xavier first and copy the application into the container.

Thanks.

1 Like
4

I see & sincerely appreciate the response.

5

If you set your docker default-runtime to nvidia, then the mounts are available during building time too - then you don’t need to copy in CUDA manually. See here for the steps:

https://github.com/dusty-nv/jetson-containers#docker-default-runtime

1 Like
7

@dusty_nv @AastaLLL it works like a charm, until I enable a docker buildkit feature. And seems like it overrides the default nvidia runtime somehow. Any idea how to merge these 2? As buildkit is really useful.

8

Hi @sskorol , I believe docker buildkit allows you to specify --runtime nvidia on the command line, whereas normal docker build does not (which is why it needs set in the daemon config)

9

@dusty_nv the following command:
DOCKER_BUILDKIT=1 docker build --runtime nvidia -t [image] .

gives an error:
unknown flag: --runtime

The same error if I enable this feature in a daemon config.

10

Ah sorry, that must have been the --mount option I was thinking that buildkit supports during build, not --runtime. I did find a couple GitHub issues on the buildkit repo which seems to indicate this should be possible, but I’m not personally sure how:

11

Got an official answer. Unfortunately, buildkit uses only runc runtime at the moment.