Nvcc -- broken, cicc not found with DOCKER_BUILDKIT=1

I get a CMake error: sh: 1: cicc not found when compiling our software from an L4T image.

It occurs with docker build using DOCKER_BUILDKIT=1 but not DOCKER_BUILDKIT=0.

From the steps below, /usr/local/cuda-10.2/nvvm/bin/cicc is not available when using the docker buildkit.
(Similarly, headers are missing, such as /usr/include/aarch64-linux-gnu/NvCaffeParser.h)

This is a problem for us b/c we use buildkit to access git over ssh from the build (--mount=type=ssh), it’s faster, and it’s easier to manage multi stage builds.

  • Is it possible to use the DOCKER_BUILDKIT=1 docker build on Jetson and access the cicc compiler? (and the other headers?)
  • Is there a better way to compile our software from the deepstream images?

steps to reproduce

You can see that cicc does not exist in the filesystem with buildkit enabled, but does when disabled:

# syntax=docker/dockerfile:1.0.0-experimental
FROM nvcr.io/nvidia/deepstream-l4t:5.0-dp-20.04-samples
RUN test -e /usr/local/cuda-10.2/nvvm/bin/cicc
$ DOCKER_BUILDKIT=1 docker build -f cicc_test.dockerfile .
...
executor failed running [/bin/sh -c test -e /usr/local/cuda-10.2/nvvm/bin/cicc]: exit code: 1
$ DOCKER_BUILDKIT=0 docker build -f cicc_test.dockerfile .
...
Successfully built

You can reproduce the CMake error like so:

cmake_minimum_required(VERSION 3.8)
project(test_cicc LANGUAGES CUDA CXX)
find_package(CUDA 10.2 REQUIRED)
# syntax=docker/dockerfile:1.0.0-experimental
FROM nvcr.io/nvidia/deepstream-l4t:5.0-dp-20.04-samples
RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections \
	&& apt-get clean && apt-get update -y && apt-get install -y \
	g++ cmake
COPY CMakeLists.txt .
RUN cmake .
$ DOCKER_BUILDKIT=0 docker build --progress plain -f cicc_cmake.dockerfile .
...

Successfully built ef7f81143a14
$ DOCKER_BUILDKIT=1 docker build --progress plain -f cicc_cmake.dockerfile .
...

#13 [4/4] RUN cmake .
#13 sha256:cec00802be0114d420fa54cd5400e8decd52d2a8b20d1404a6df8134c1560d6a
#13 0.843 -- The CUDA compiler identification is unknown
#13 0.993 -- The CXX compiler identification is GNU 7.5.0
#13 1.004 -- Check for working CUDA compiler: /usr/local/cuda-10.2/bin/nvcc
#13 1.196 -- Check for working CUDA compiler: /usr/local/cuda-10.2/bin/nvcc -- broken
#13 1.197 CMake Error at /usr/share/cmake-3.10/Modules/CMakeTestCUDACompiler.cmake:46 (message):
#13 1.197   The CUDA compiler
#13 1.197 
#13 1.197     "/usr/local/cuda-10.2/bin/nvcc"
#13 1.197 
#13 1.197   is not able to compile a simple test program.
#13 1.197 
#13 1.197   It fails with the following output:
#13 1.197 
#13 1.197     Change Dir: /opt/nvidia/deepstream/CMakeFiles/CMakeTmp
#13 1.197     
#13 1.197     Run Build Command:"/usr/bin/make" "cmTC_b2444/fast"
#13 1.197     /usr/bin/make -f CMakeFiles/cmTC_b2444.dir/build.make CMakeFiles/cmTC_b2444.dir/build
#13 1.197     make[1]: Entering directory '/opt/nvidia/deepstream/CMakeFiles/CMakeTmp'
#13 1.197     Building CUDA object CMakeFiles/cmTC_b2444.dir/main.cu.o
#13 1.197     /usr/local/cuda-10.2/bin/nvcc     -x cu -c /opt/nvidia/deepstream/CMakeFiles/CMakeTmp/main.cu -o CMakeFiles/cmTC_b2444.dir/main.cu.o
#13 1.197     sh: 1: cicc: not found
#13 1.197     CMakeFiles/cmTC_b2444.dir/build.make:65: recipe for target 'CMakeFiles/cmTC_b2444.dir/main.cu.o' failed
#13 1.197     make[1]: *** [CMakeFiles/cmTC_b2444.dir/main.cu.o] Error 127
#13 1.197     make[1]: Leaving directory '/opt/nvidia/deepstream/CMakeFiles/CMakeTmp'
#13 1.197     Makefile:126: recipe for target 'cmTC_b2444/fast' failed
#13 1.197     make: *** [cmTC_b2444/fast] Error 2
#13 1.197     
#13 1.197 
#13 1.197   
#13 1.197 
#13 1.197   CMake will not be able to correctly generate this project.
#13 1.197 Call Stack (most recent call first):
#13 1.197   CMakeLists.txt:2 (project)
#13 1.197 
#13 1.197 
#13 1.199 -- Configuring incomplete, errors occurred!
#13 1.199 See also "/opt/nvidia/deepstream/CMakeFiles/CMakeOutput.log".
#13 1.199 See also "/opt/nvidia/deepstream/CMakeFiles/CMakeError.log".
#13 ERROR: executor failed running [/bin/sh -c cmake .]: exit code: 1
------
 > [4/4] RUN cmake .:
------
executor failed running [/bin/sh -c cmake .]: exit code: 1

other posts

This post says to make sure cuda is installed correctly. Since I can build with DOCKER_BUILDKIT=0, and the Jetpack install suceeded, I assume it is installed correctly.

This post says to make sure the compiler is found.
However nvcc is found, and adding the /usr/local/cuda-10.2/nvvm/bin folder does not change the result.
(default path: /usr/local/cuda-10.2/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)

This post says that nvcc is accessible but cicc is not.
That’s from March 2020, has there been an update?

system specs

  • Xavier AGX / Jetpack 4.4.1
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_21:14:42_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89

$ docker --version
Docker version 20.10.2, build 20.10.2-0ubuntu1~18.04.2

$ cat /etc/docker/daemon.json
{
    "experimental": true,
    "runtimes": {
	"nvidia": {
	    "path": "nvidia-container-runtime",
	    "runtimeArgs": []
	}
    },

    "default-runtime": "nvidia"
}

I have exactly the same problem. Using NVIDIA driver version 460 with Cuda 11.2 support, and using CUDA Toolkit 10.2 for compiling, gives me the same error saying “is not able to compile a simple test program”.

I’m guessing it has to do with the CUDA Toolkit not setup properly. I just don’t know where.

are you also running on a Jetson and did the jetpack install succeed?