Unable to build OpenCV with CUDA in Docker on Jetson

Autonomous Machines Jetson & Embedded Systems Jetson AGX Orin
, ,
1

Setup info

• Hardware Platform (Jetson / GPU) Jetson AGX Orin
• DeepStream Version 6.2 (inside of docker)
• JetPack Version (valid for Jetson only) 5.1.1 (L4T 35.3.1)

Problem

I try to build a docker image with OpenCV supporting CUDA and Gstreamer.
But I am unable to do it, because during compilation I get info:

CMake Warning at cmake/OpenCVFindLibsPerf.cmake:45 (message):
  OpenCV is not able to find/configure CUDA SDK (required by WITH_CUDA).

  CUDA support will be disabled in OpenCV build.
  To eliminate this warning remove WITH_CUDA=ON CMake configuration option.

Setup and reproduction

1. Create Dockerfile:

FROM nvcr.io/nvidia/deepstream-l4t:6.2-samples

# Usefull tools, apps and libraries
RUN apt-get update && apt install -y \
    g++ gcc git automake \
    ffmpeg wget sudo htop x11-apps nano xonsh \
    libopenmpt-dev python3-pip python3-gi  

RUN useradd -ms /bin/bash -G sudo,audio,video,render myuser && echo "myuser ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/ubuntu
ENV TERM=xterm-256color
USER myuser
WORKDIR /home/myuser

RUN wget https://forums.developer.nvidia.com/uploads/short-url/3kLERQgB4ZR0q0wgUdO9qY6lxBq.sh -O install_opencv4.6.0_Jetson.sh
RUN sudo chmod +x install_opencv4.6.0_Jetson.sh
# To check whether CUDA libs are availavle
RUN ls -1 /usr/local/ | grep cuda-
RUN echo yes | ./install_opencv4.6.0_Jetson.sh
RUN python3 -c "import cv2; print(cv2.getBuildInformation())" | grep CUDA

It uses OpenCV script provided here: Compiling OpenCV on Jetpack 5 - #5 by AastaLLL

2. Edit /etc/docker/daemon.json to be sure that docker builder uses nvidia Runtime

{
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "/usr/bin/nvidia-container-runtime"
        }
    },
    "default-runtime": "nvidia" ,
}

This step is based on:

and

and

3. Reboot machine

4. Build docker

with command:

DOCKER_BUILDKIT=0 docker build  .

5. Observe output

Cuda libraries check

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  54.73MB
////Not important lines
 ---> bb4a6c3df745
Step 10/12 : RUN ls /usr/local/cuda-11.4/lib64
 ---> Running in 676c9518f1ea
libcublas.so                 libnppidei.so
libcublas.so.11              libnppidei.so.11
libcublas.so.11.6.6.84       libnppidei.so.11.4.0.287
libcublasLt.so.11            libnppif.so
libcublasLt.so.11.6.6.84     libnppif.so.11
libcudart.so                 libnppif.so.11.4.0.287
////CUDA libs are available

Check compilation

------------------------------------
** Build opencv 4.6.0 (3/4)
------------------------------------
-- The CXX compiler identification is GNU 9.4.0
-- The C compiler identification is GNU 9.4.0
////Not important lines
-- Performing Test HAVE_CXX_WNO_CLASS_MEMACCESS - Success
CMake Warning at cmake/OpenCVFindLibsPerf.cmake:45 (message):
  OpenCV is not able to find/configure CUDA SDK (required by WITH_CUDA).

  CUDA support will be disabled in OpenCV build.

  To eliminate this warning remove WITH_CUDA=ON CMake configuration option.

Call Stack (most recent call first):
  CMakeLists.txt:733 (include)

-- Could not find OpenBLAS include. Turning OpenBLAS_FOUND off
-- Could not find OpenBLAS lib. Turning OpenBLAS_FOUND off
////Not important lines
-- General configuration for OpenCV 4.6.0 =====================================
--   Version control:               unknown
-- 
--   Extra modules:
--     Location (extra):            /home/myuser/workspace/opencv_contrib-4.6.0/modules
--     Version control (extra):     unknown
-- 
--   Platform:
--     Timestamp:                   2024-01-10T15:56:47Z
--     Host:                        Linux 5.10.104-tegra aarch64
--     CMake:                       3.16.3
--     CMake generator:             Unix Makefiles
--     CMake build tool:            /usr/bin/make
--     Configuration:               RELEASE
....................
--     GStreamer:                   YES (1.16.3)
--     v4l/v4l2:                    YES (linux/videodev2.h)
-- 
--   Parallel framework:            pthreads
-- 
--   Trace:                         YES (with Intel ITT)
-- 
--   Other third-party libraries:
--     Lapack:                      NO
--     Eigen:                       NO
--     Custom HAL:                  YES (carotene (ver 0.0.1))
--     Protobuf:                    build (3.19.1)
-- 
--   NVIDIA CUDA:                   NO
-- 
--   cuDNN:                         NO
-- 
--   OpenCL:                        YES (no extra features)

Watch the last build step (and fail):

** Install opencv 4.6.0 successfully
** Bye :)
Removing intermediate container 0b34e1f472f2
 ---> 5ea8722e2133
Step 12/12 : RUN python3 -c "import cv2; print(cv2.getBuildInformation())" | grep CUDA
 ---> Running in 49b928daaae4
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'cv2'
The command '/bin/sh -c python3 -c "import cv2; print(cv2.getBuildInformation())" | grep CUDA' returned a non-zero code: 1
3

Hi,

  OpenCV is not able to find/configure CUDA SDK (required by WITH_CUDA).

It looks like CUDA is not available in your container.
Could you give it a check?

To compile, you will need both the runtime and dev CUDA packages.
Thanks.

4

I just showed content of the directory with cuda files (/usr/local/cuda-11.4/lib64).
What are the other ways of checking it?

5

Hi,

You will need the CUDA dev package that contains the header.
For example:

$ ll /usr/local/cuda/include/cuda_runtime.h

Thanks.

6

Thank you for response.

cuda_runtime.h was missing, so I added package cuda-libraries-dev-11-4 to my docker file and now in /usr/local/cuda/include/ you can see

Step 11/14 : RUN ls  /usr/local/cuda/include/
 ---> Running in 3773e99864aa
builtin_types.h                device_atomic_functions.hpp
channel_descriptor.h           device_double_functions.h
common_functions.h             device_functions.h
cooperative_groups             device_launch_parameters.h
cooperative_groups.h           device_types.h
cuComplex.h                    driver_functions.h
cub                            driver_types.h
cublas.h                       host_config.h
cublasLt.h                     host_defines.h
cublasXt.h                     library_types.h
cublas_api.h                   math_constants.h
cublas_v2.h                    math_functions.h
cuda                           mma.h
cuda.h                         npp.h
cudaEGL.h                      nppcore.h
cudaEGLTypedefs.h              nppdefs.h
cudaGL.h                       nppi.h
cudaGLTypedefs.h               nppi_arithmetic_and_logical_operations.h
cudaProfiler.h                 nppi_color_conversion.h
cudaProfilerTypedefs.h         nppi_data_exchange_and_initialization.h
cudaTypedefs.h                 nppi_filtering_functions.h
cudaVDPAU.h                    nppi_geometry_transforms.h
cudaVDPAUTypedefs.h            nppi_linear_transforms.h
cuda_awbarrier.h               nppi_morphological_operations.h
cuda_awbarrier_helpers.h       nppi_statistics_functions.h
cuda_awbarrier_primitives.h    nppi_support_functions.h
cuda_bf16.h                    nppi_threshold_and_compare_operations.h
cuda_bf16.hpp                  npps.h
cuda_device_runtime_api.h      npps_arithmetic_and_logical_operations.h
cuda_egl_interop.h             npps_conversion_functions.h
cuda_fp16.h                    npps_filtering_functions.h
cuda_fp16.hpp                  npps_initialization.h
cuda_gl_interop.h              npps_statistics_functions.h
cuda_occupancy.h               npps_support_functions.h
cuda_pipeline.h                nv
cuda_pipeline_helpers.h        nvToolsExt.h
cuda_pipeline_primitives.h     nvToolsExtCuda.h
cuda_profiler_api.h            nvToolsExtCudaRt.h
cuda_runtime.h                 nvToolsExtOpenCL.h
cuda_runtime_api.h             nvToolsExtSync.h
cuda_surface_types.h           nvblas.h
cuda_texture_types.h           nvfunctional
cuda_vdpau_interop.h           nvrtc.h
cudalibxt.h                    nvtx3
cudart_platform.h              sm_20_atomic_functions.h
cudla.h                        sm_20_atomic_functions.hpp
cufft.h                        sm_20_intrinsics.h
cufftXt.h                      sm_20_intrinsics.hpp
cufftw.h                       sm_30_intrinsics.h
curand.h                       sm_30_intrinsics.hpp
curand_discrete.h              sm_32_atomic_functions.h
curand_discrete2.h             sm_32_atomic_functions.hpp
curand_globals.h               sm_32_intrinsics.h
curand_kernel.h                sm_32_intrinsics.hpp
curand_lognormal.h             sm_35_atomic_functions.h
curand_mrg32k3a.h              sm_35_intrinsics.h
curand_mtgp32.h                sm_60_atomic_functions.h
curand_mtgp32_host.h           sm_60_atomic_functions.hpp
curand_mtgp32_kernel.h         sm_61_intrinsics.h
curand_mtgp32dc_p_11213.h      sm_61_intrinsics.hpp
curand_normal.h                surface_functions.h
curand_normal_static.h         surface_functions.hpp
curand_philox4x32_x.h          surface_indirect_functions.h
curand_poisson.h               surface_indirect_functions.hpp
curand_precalc.h               surface_types.h
curand_uniform.h               texture_fetch_functions.h
cusolverDn.h                   texture_fetch_functions.hpp
cusolverRf.h                   texture_indirect_functions.h
cusolverSp.h                   texture_indirect_functions.hpp
cusolverSp_LOWLEVEL_PREVIEW.h  texture_types.h
cusolver_common.h              thrust
cusparse.h                     vector_functions.h
cusparse_v2.h                  vector_functions.hpp
device_atomic_functions.h      vector_types.h

But I still got this error:

Make Warning at cmake/OpenCVFindLibsPerf.cmake:45 (message):
  OpenCV is not able to find/configure CUDA SDK (required by WITH_CUDA).

  CUDA support will be disabled in OpenCV build.

  To eliminate this warning remove WITH_CUDA=ON CMake configuration option.

Call Stack (most recent call first):
  CMakeLists.txt:733 (include)
7

Hi,

Could you share the complete cmake log so we can know which path/lib is missing?

Thanks.

8

Hi @AastaLLL ,

I provided you entire docker file and every command required for easy reproduction of this problem.
You can quickly reproduce this problem by yourself. Just follow instructions in the first post.

It would be much faster for you and me.
I provided you minimal working example to make reproduction on your side as easy as possible.

9

building_opencv_docker.log (1.1 MB)

10

Hi,

[100%] Built target opencv_gapi
** Build opencv 4.8.0 successfully
** Bye :)
Removing intermediate container 389727a8fc99
 ---> cc5b7ab78ece
Step 11/12 : RUN python3 -c "import cv2; print(cv2.getBuildInformation())"
 ---> Running in 31d281c81710
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'cv2'

Based on your log, the OpenCV compiling works without error.
Please try to add the PYTHONPATH and LD_LIBRARY_PATH to fix the import issue.

echo 'export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH' >> ~/.bashrc
echo 'export PYTHONPATH=/usr/local/lib/python3.8/site-packages/:$PYTHONPATH' >> ~/.bashrc
source ~/.bashrc

(Please update to the dockerfile command)
Thanks.

11

@AastaLLL

Your fix doesn’t address the core issue (OpenCV does not build with CUDA)

Unluckily your fix doesn’t help with importing cv2. (/usr/local/lib/python3.8/site-packages Doesn’t seem to exist).

Step 10/17 : RUN echo yes | sudo ./install_opencv4.8.0_Jetson.sh
 ---> Using cache
 ---> cc5b7ab78ece
Step 11/17 : RUN ls /usr/local/lib
 ---> Running in 115177ee09b2
librdkafka++.a   librdkafka++.so.1  librdkafka.so    pkgconfig  python3.8
librdkafka++.so  librdkafka.a       librdkafka.so.1  python2.7
Removing intermediate container 115177ee09b2
 ---> 79ae4c78fe61
Step 12/17 : RUN ls /usr/local/lib/python3.8/
 ---> Running in c47db1c0bf13
dist-packages
Removing intermediate container c47db1c0bf13
 ---> 8f54545983cc
Step 13/17 : RUN echo 'export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH' >> ~/.bashrc
 ---> Running in 989bb14b3fb4
Removing intermediate container 989bb14b3fb4
 ---> cacca9af7178
Step 14/17 : RUN echo 'export PYTHONPATH=/usr/local/lib/python3.8/site-packages/:$PYTHONPATH' >> ~/.bashrc
 ---> Running in 891b3fdf7ab0
Removing intermediate container 891b3fdf7ab0
 ---> 09ec3d3324a0
Step 15/17 : RUN . ~/.bashrc
 ---> Running in 0e1ae971c1b3
Removing intermediate container 0e1ae971c1b3
 ---> 13c9a784274e
Step 16/17 : RUN PYTHONPATH=/usr/local/lib/python3.8/site-packages/:$PYTHONPATH python3 -c "import cv2; print(cv2.getBuildInformation())"
 ---> Running in 6b25fd2d52cc
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'cv2'
The command '/bin/sh -c PYTHONPATH=/usr/local/lib/python3.8/site-packages/:$PYTHONPATH python3 -c "import cv2; print(cv2.getBuildInformation())"' returned a non-zero code: 1
wbuser@ubuntu:~/eyeq$
12

Why you haven’t just tested your recommendation?

13

@AastaLLL

It appears, that there is a mistake in the installation script from Cannot build opencv 4.8.0 with nvidia docker - #13 by AastaLLL

echo "------------------------------------"
echo "** Build opencv "${version}" (3/4)"
echo "------------------------------------"
mkdir release
cd release/
cmake -D CMAKE_TOOLCHAIN_FILE=../platforms/linux/aarch64-gnu.toolchain.cmake -D WITH_CUDA=ON -D WITH_CUDNN=ON -D CUDA_ARCH_BIN="7.2,8.7" -D CUDA_ARCH_PTX="" -D OPENCV_GENERATE_PKGCONFIG=ON -D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib-${version}/modules -D WITH_GSTREAMER=ON -D WITH_LIBV4L=ON -D BUILD_opencv_python3=ON -D BUILD_TESTS=OFF -D BUILD_PERF_TESTS=OFF -D BUILD_EXAMPLES=OFF -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local ..
make -j$(nproc)


echo "** Build opencv "${version}" successfully"
echo "** Bye :)"

There is no installation step. (only building).

14

Hi,

Is your issue fixed after installation?
Thanks.

15

@AastaLLL

No it did not solve the issue.
It only helped with installing OpenCV. But the main problem is with compilation not including CUDA support.

 Protobuf:                    build (3.19.1)
 NVIDIA CUDA:                   NO
 cuDNN:                         NO
16

Hi,

You will need a dev container to get the CUDA header and compiler.
For example:

diff --git a/Dockerfile b/Dockerfile
index 52f5b85..815a4ea 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,10 +1,10 @@
-FROM nvcr.io/nvidia/deepstream-l4t:6.2-samples
-
+#FROM nvcr.io/nvidia/deepstream-l4t:6.2-samples
+FROM nvcr.io/nvidia/l4t-cuda:11.4.19-devel
 # Usefull tools, apps and libraries
 RUN apt-get update && apt install -y \
     g++ gcc git automake \
     ffmpeg wget sudo htop x11-apps nano xonsh \
-    libopenmpt-dev python3-pip python3-gi
+    libopenmpt-dev python3-pip python3-gi unzip

 RUN useradd -ms /bin/bash -G sudo,audio,video,render myuser && echo "myuser ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/ubuntu
 ENV TERM=xterm-256color

Not sure why the import fails in the Dockerfile.
But when you launching the container, the same command works well without issues.

$ sudo docker run -it --rm --runtime nvidia 3a3f8372ef53

==========
== CUDA ==
==========

...
myuser@24da2cd41af6:~$ python3 -c "import cv2; print(cv2.getBuildInformation())" | grep CUDA
  NVIDIA CUDA:                   YES (ver 11.4, CUFFT CUBLAS)

Thanks.

1 Like
17

Thanks for your answer.

Using docker image image nvcr.io/nvidia/l4t-cuda:11.4.19-devel as base makes building CUDA work.

It means, that the nvcr.io/nvidia/deepstream-l4t:6.2-samples misses some cuda-related packages.

18

Please find the comment on below link:

Thanks.

21

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.