Building open3d docker container - libcudaart path issue

_av · September 7, 2020, 6:09pm

trying to build the dockerfile below using docker build . -tt open3d. Unfortunately it misses the libcudaart.

ERRORUnable to find third-party library CUDAToolkit
-- Building library 3rdparty_poisson from source
-- Unable to find cudart library.
CMake Error at /usr/local/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:165 (message):
  Could NOT find CUDAToolkit (missing: CUDA_CUDART) (found version "10.2.89")
Call Stack (most recent call first):
  /usr/local/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:458 (_FPHSA_FAILURE_MESSAGE)
  /usr/local/share/cmake-3.18/Modules/FindCUDAToolkit.cmake:731 (find_package_handle_standard_args)
  src/Cuda/CMakeLists.txt:10 (find_package)

FROM nvcr.io/nvidia/l4t-base:r32.4.3
RUN apt-get update -y
RUN apt upgrade -y
ENV TZ America/Los_Angeles
RUN DEBIAN_FRONTEND="noninteractive" apt install -y libssl-dev wget sudo  tzdata  software-properties-common apt-utils  curl git && apt-get install -y build-essential
RUN DEBIAN_FRONTEND="noninteractive" apt-get purge cmake && wget https://github.com/Kitware/CMake/releases/download/v3.18.2/cmake-3.18.2.tar.gz && tar -zxvf cmake-3.18.2.tar.gz && cd cmake-3.18.2 &&  ./bootstrap && make && make install

RUN apt-get install -y libssl-dev libusb-1.0-0-dev pkg-config libgtk-3-dev
RUN apt-get install -y libglfw3-dev libgl1-mesa-dev libglu1-mesa-dev

RUN  apt remove --purge libeigen3-dev -y && wget https://gitlab.com/libeigen/eigen/-/archive/3.3.7/eigen-3.3.7.tar.gz && tar -xvf eigen-3.3.7.tar.gz && cd eigen-3.3.7  && mkdir build && cd build && cmake -DCMAKE_INSTALL_PREFIX=/usr/loca .. && make && make install

#RUN  git clone --recursive https://github.com/devshank3/Open3D-for-Jetson.git && cd Open3D-for-Jetson/ && cd util/scripts/ && ./install-deps-ubuntu.sh
RUN mkdir -p git_src/Open3D/theNded && cd git_src/Open3D/theNded && git clone --recursive https://github.com/theNded/Open3D.git && cd Open3D &&  git checkout 0ca8fd19d355d80e5998efec31858fe19791ccf5  && wget https://raw.githubusercontent.com/nanotuxi/jetsonNanoPatches/master/Open3DJetsonCudaRealSense.patch && git apply Open3DJetsonCudaRealSense.patch
# git checkout 0ca8fd19d355d80e5998efec31858fe19791ccf5 && cd Open3d && wget https://raw.githubusercontent.com/nanotuxi/jetsonNanoPatches/master/Open3DJetsonCudaRealSense.patch && git apply Open3DJetsonCudaRealSense.patch &&  cd util/scripts/ && ./install-deps-ubuntu.sh && sudo apt autoremove && make clean && cmake .. && make -j6 && make install-pip-package
RUN  apt-get install libhdf5-serial-dev hdf5-tools libhdf5-dev zlib1g-dev zip libjpeg8-dev liblapack-dev libblas-dev gfortran -y
RUN ldconfig
RUN apt-get install python3-pip -y
RUN pip3 install -U pip testresources setuptools
RUN pip3 install -U numpy==1.16.1 future==0.17.1 mock==3.0.5 h5py==2.9.0 keras_preprocessing==1.0.5 keras_applications==1.0.8 gast==0.2.2 futures protobuf pybind11
RUN pip3 install Cython
RUN apt update -y && apt upgrade -y
RUN apt install -y  python3-matplotlib libopencv-dev python3-opencv python3-joblib
#RUN export CUDA_curand_LIBRARY:FILEPATH=/usr/local/cuda-10.2/targets/aarch64-linux/lib/libcurand.so
#RUN apt-get install -y libssl-dev libusb-1.0-0-dev pkg-config libgtk-3-dev
#RUN apt-get install -y libglfw3-dev libgl1-mesa-dev libglu1-mesa-dev
#RUN cd Open3D-for-Jetson/ && mkdir build && cd build && rm -rf * && cmake -DBUILD_EIGEN3=OFF .. && sudo make -j8
#RUN wget https://github.com/IntelRealSense/librealsense/archive/2.38.1.2169.tar.gz && tar -xvf 2.38.1.2169.tar.gz && cd librealsense-2.38.1.2169/ && mkdir build && cd b$
ENV CPATH=/usr/local/cuda-10.2/targets/aarch64-linux/include:$CPATH
ENV LD_LIBRARY_PATH=/usr/local/cuda-10.2/targets/aarch64-linux/lib:$LD_LIBRARY_PATH
ENV PATH=/usr/local/cuda-10.2/bin:$PATH
RUN ln -s /usr/local/cuda-10.2 /usr/local/cuda
RUN export CPATH=/usr/local/cuda-10.2/targets/aarch64-linux/include:$CPATH && export LD_LIBRARY_PATH=/usr/local/cuda-10.2/targets/aarch64-linux/lib:$LD_LIBRARY_PATH &&  export PATH=/usr/local/cuda-10.2/bin:$PATH && ldconfig && cd git_src/Open3D/theNded/Open3D && mkdir build && cd build && rm -rf * && cmake -DBUILD_EIGEN3=OFF .. && sudo make -j8

AastaLLL · September 8, 2020, 4:43am

Hi,

L4t-base container will mount CUDA library from the XavierNX host.
Do you install CUDA toolkit on NX first?

Thanks.

_av · September 8, 2020, 4:48am

Hi @AastaLLL ,
Thank you for following up!
The Nx uses default stock Jetpack installation with all components installed.
The system has the library installed, as it seems to me.
Moreover, given the cmake - > build commands are executed not by docker build .m but from the very same container from bash session the install will suffice. the issue is only observable if the container is built from dockerfile using docker build …

nx:~$ locate cudart
/usr/include/hwloc/cudart.h
/usr/lib/pkgconfig/cudart-10.2.pc
/usr/local/cuda-10.2/doc/man/man7/libcudart.7
/usr/local/cuda-10.2/doc/man/man7/libcudart.so.7
/usr/local/cuda-10.2/targets/aarch64-linux/include/cudart_platform.h
/usr/local/cuda-10.2/targets/aarch64-linux/lib/libcudart.so
/usr/local/cuda-10.2/targets/aarch64-linux/lib/libcudart.so.10.2
/usr/local/cuda-10.2/targets/aarch64-linux/lib/libcudart.so.10.2.89
/usr/local/cuda-10.2/targets/aarch64-linux/lib/libcudart_static.a
/usr/share/doc/cuda-cudart-10-2
/usr/share/doc/cuda-cudart-dev-10-2
/usr/share/doc/cuda-cudart-10-2/changelog.Debian.gz
/usr/share/doc/cuda-cudart-10-2/copyright
/usr/share/doc/cuda-cudart-dev-10-2/changelog.Debian.gz
/usr/share/doc/cuda-cudart-dev-10-2/copyright
/var/lib/dpkg/info/cuda-cudart-10-2.conffiles
/var/lib/dpkg/info/cuda-cudart-10-2.list
/var/lib/dpkg/info/cuda-cudart-10-2.md5sums
/var/lib/dpkg/info/cuda-cudart-10-2.shlibs
/var/lib/dpkg/info/cuda-cudart-10-2.triggers
/var/lib/dpkg/info/cuda-cudart-dev-10-2.list
/var/lib/dpkg/info/cuda-cudart-dev-10-2.md5sums

AastaLLL · September 9, 2020, 2:14am

Hi,

Thanks.
We are going to reproduce this and share more info later.

AastaLLL · September 9, 2020, 8:39am

Hi,

Please noticed that our l4t-base container directly mounts CUDA toolkit from the host to reduce image size.
However, there is no mechanism to mount a volume in building time and cause this issue.

A workaround is to copy the CUDA toolkit into the container manually and delete it after compiling.

$ cp -r /usr/local/cuda-10.2/ .
$ sudo docker build -t test .

Dockerfile

...
COPY ./cuda-10.2 /usr/local/cuda-10.2
RUN cd git_src/Open3D/theNded/Open3D && mkdir build && cd build && rm -rf * && cmake -DBUILD_EIGEN3=OFF .. && sudo make -j8

We can generate the cmake file correctly with above Dockerfile:

Step 20/20 : RUN cd git_src/Open3D/theNded/Open3D && mkdir build && cd build && rm -rf * && cmake -DBUILD_EIGEN3=OFF .. && sudo make -j8
 ---> Running in 4b7516f49eda
...
-- Found CUDA: /usr/local/cuda-10.2 (found version "10.2") 
-- Building CUDA enabled
-- The CUDA compiler identification is NVIDIA 10.2.89
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /usr/local/cuda-10.2/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Automatic GPU detection failed. Building for common architectures.
-- Autodetected CUDA architecture(s): 3.5;5.0;3.0;5.2;6.0;6.1;7.0;7.5;7.5+PTX

Thanks.

_av · September 16, 2020, 2:44pm

@AastaLLL Thanks, It seems got through the cmake process creation.
However, further at the build step it seems not able to finish the build process.

[ 32%] Building CXX object src/Open3D/Geometry/CMakeFiles/Geometry.dir/PointCloudCluster.cpp.o
[ 32%] Building CUDA object src/Open3D/Core/CMakeFiles/Core.dir/MemoryManagerCUDA.cu.o
nvcc fatal   : A single input file is required for a non-link phase when an outputfile is specified
src/Open3D/Core/CMakeFiles/Core.dir/build.make:172: recipe for target 'src/Open3D/Core/CMakeFiles/Core.dir/MemoryManagerCUDA.cu.o' failed
make[2]: *** [src/Open3D/Core/CMakeFiles/Core.dir/MemoryManagerCUDA.cu.o] Error 1
make[1]: *** [src/Open3D/Core/CMakeFiles/Core.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
CMakeFiles/Makefile2:1099: recipe for target 'src/Open3D/Core/CMakeFiles/Core.dir/all' failed
[ 32%] Building CXX object src/Open3D/Geometry/CMakeFiles/Geometry.dir/PointCloudFactory.cpp.o
[ 32%] Building CXX object src/Open3D/Visualization/CMakeFiles/Visualization.dir/Shader/TextureSimpleShader.cpp.o
[ 32%] Building CXX object src/Open3D/Geometry/CMakeFiles/Geometry.dir/PointCloudSegmentation.cpp.o
[ 32%] Built target Utility
---------
[50%] Building CXX object src/Open3D/Geometry/CMakeFiles/Geometry.dir/VoxelGrid.cpp.o
[ 51%] Building CXX object src/Open3D/Geometry/CMakeFiles/Geometry.dir/VoxelGridFactory.cpp.o
[ 51%] Built target Geometry
Makefile:148: recipe for target 'all' failed
make: *** [all] Error 2
The command '/bin/sh -c cd git_src/Open3D/theNded/Open3D && mkdir build && cd build && rm -rf * && cmake -DBUILD_EIGEN3=OFF .. && sudo make -j8' returned a non-zero code: 2

was you able to complete the build step also at your side testing?
the Dockerfile url in the previous post - is it a placeholder or actually there is a downloadable file? it doen’t seem downloadable.
reference file that I tried

github.com

AndreV84/Jetson/blob/master/Dockerfile_open3d_non_funct

FROM nvcr.io/nvidia/l4t-base:r32.4.3
RUN apt-get update -y
RUN apt upgrade -y
ENV TZ America/Los_Angeles
RUN DEBIAN_FRONTEND="noninteractive" apt install -y libssl-dev wget sudo  tzdata  software-properties-common apt-utils  curl git && apt-get install -y build-essential
RUN DEBIAN_FRONTEND="noninteractive" apt-get purge cmake && wget https://github.com/Kitware/CMake/releases/download/v3.18.2/cmake-3.18.2.tar.gz && tar -zxvf cmake-3.18.2.tar.gz && cd cmake-3.18.2 &&  ./bootstrap && make && make install

RUN apt-get install -y libssl-dev libusb-1.0-0-dev pkg-config libgtk-3-dev
RUN apt-get install -y libglfw3-dev libgl1-mesa-dev libglu1-mesa-dev

RUN  apt remove --purge libeigen3-dev -y && wget https://gitlab.com/libeigen/eigen/-/archive/3.3.7/eigen-3.3.7.tar.gz && tar -xvf eigen-3.3.7.tar.gz && cd eigen-3.3.7  && mkdir build && cd build && cmake -DCMAKE_INSTALL_PREFIX=/usr/loca .. && make && make install

#RUN  git clone --recursive https://github.com/devshank3/Open3D-for-Jetson.git && cd Open3D-for-Jetson/ && cd util/scripts/ && ./install-deps-ubuntu.sh
RUN mkdir -p git_src/Open3D/theNded && cd git_src/Open3D/theNded && git clone --recursive https://github.com/theNded/Open3D.git && cd Open3D &&  git checkout 0ca8fd19d355d80e5998efec31858fe19791ccf5  && wget https://raw.githubusercontent.com/nanotuxi/jetsonNanoPatches/master/Open3DJetsonCudaRealSense.patch && git apply Open3DJetsonCudaRealSense.patch
# git checkout 0ca8fd19d355d80e5998efec31858fe19791ccf5 && cd Open3d && wget https://raw.githubusercontent.com/nanotuxi/jetsonNanoPatches/master/Open3DJetsonCudaRealSense.patch && git apply Open3DJetsonCudaRealSense.patch &&  cd util/scripts/ && ./install-deps-ubuntu.sh && sudo apt autoremove && make clean && cmake .. && make -j6 && make install-pip-package
RUN  apt-get install libhdf5-serial-dev hdf5-tools libhdf5-dev zlib1g-dev zip libjpeg8-dev liblapack-dev libblas-dev gfortran -y
RUN ldconfig
RUN apt-get install python3-pip -y
RUN pip3 install -U pip testresources setuptools
RUN pip3 install -U numpy==1.16.1 future==0.17.1 mock==3.0.5 h5py==2.9.0 keras_preprocessing==1.0.5 keras_applications==1.0.8 gast==0.2.2 futures protobuf pybind11

This file has been truncated. show original

However, I noticed, if to commect the latter line, then container will build;
Once it got build it becomes possible to run it in bash mode, then the execution of eigen patch from makes everything seems to build successfully

_av · September 22, 2020, 9:29am

another open3d issue that I probably should create a separate topic for

./RunSystem /home/user/o3d/git_src/Open3D/theNded/Open3D/examples/Cuda/ReconstructionSystem/config/intel/test.json
[Open3D INFO] Processing fragment 0 / 0
[Open3D INFO] RGBD Odometry between (0 1)
[Open3D INFO] RGBD Odometry between (1 2)
[Open3D INFO] RGBD Odometry between (2 3)
[Open3D INFO] RGBD Odometry between (3 4)
[Open3D INFO] RGBD Odometry between (4 5)
[Open3D INFO] RGBD Odometry between (5 6)
[Open3D INFO] RGBD Odometry between (6 7)
[Open3D INFO] RGBD Odometry between (7 8)
[Open3D INFO] RGBD Odometry between (8 9)
[Open3D INFO] RGBD Odometry between (9 10)
[Open3D INFO] RGBD Odometry between (10 11)
[Open3D INFO] RGBD Odometry between (11 12)
[Open3D INFO] RGBD Odometry between (12 13)
[Open3D INFO] RGBD Odometry between (13 14)
[Open3D INFO] RGBD Odometry between (14 15)
[Open3D INFO] RGBD Odometry between (15 16)
[Open3D INFO] RGBD Odometry between (16 17)
[Open3D INFO] RGBD Odometry between (17 18)
[Open3D INFO] RGBD Odometry between (18 19)
[Open3D INFO] RGBD Odometry between (19 20)
[Open3D INFO] RGBD Odometry between (20 21)
[Open3D INFO] RGBD Odometry between (21 22)
[Open3D INFO] RGBD Odometry between (22 23)
[Open3D INFO] RGBD Odometry between (23 24)
[Open3D INFO] Active subvolumes in volume: 389
[Open3D INFO] Active subvolumes in volume: 440
[Open3D INFO] Active subvolumes in volume: 460
[Open3D INFO] Active subvolumes in volume: 464
[Open3D INFO] Active subvolumes in volume: 473
[Open3D INFO] Active subvolumes in volume: 474
[Open3D INFO] Active subvolumes in volume: 485
[Open3D INFO] Active subvolumes in volume: 486
[Open3D INFO] Active subvolumes in volume: 500
[Open3D INFO] Active subvolumes in volume: 503
[Open3D INFO] Active subvolumes in volume: 509
[Open3D INFO] Active subvolumes in volume: 510
[Open3D INFO] Active subvolumes in volume: 512
[Open3D INFO] Active subvolumes in volume: 514
[Open3D INFO] Active subvolumes in volume: 515
[Open3D INFO] Active subvolumes in volume: 517
[Open3D INFO] Active subvolumes in volume: 521
[Open3D INFO] Active subvolumes in volume: 523
[Open3D INFO] Active subvolumes in volume: 524
[Open3D INFO] Active subvolumes in volume: 529
[Open3D INFO] Active subvolumes in volume: 530
[Open3D INFO] Active subvolumes in volume: 530
[Open3D INFO] Active subvolumes in volume: 534
[Open3D INFO] Active subvolumes in volume: 536
[Open3D INFO] Active subvolumes in volume: 541
[Open3D INFO] Writing 697 subvolumes.

CUDA error at /home/user/o3d/git_src/Open3D/theNded/Open3D/src/Cuda/Integration/ScalableMeshVolumeCudaKernel.cuh:117 code=701(cudaErrorLaunchOutOfResources) "cudaGetLastError()"

_av · September 24, 2020, 12:28pm

AastaLLL · October 14, 2020, 3:03am

Hi,

Sorry for the late update.

Would you mind to share the latest status with us?
Have you successfully built the library?

The first issue listed in this comment seems related to this topic.
Would you mind to check if there is any similar cmake include command in the open3d first.

The other issue for CUDA error 701 is cudaErrorLaunchOutOfResources.
This is a common problem when compiling a desktop library on Jetson.
The root cause is the resource assignment is too large to deploy on Jetson and leads to this error.

This can be solved by reassigning the CUDA resource properly.
Here is an example for pyTorch for your reference.

gist.github.com

https://gist.github.com/dusty-nv/ce51796085178e1f38e3c6a1663a93a1#file-pytorch-1-6-rc2-jetpack-4-4-ga-patch

pytorch-1.4-diff-jetpack-4.4.patch

diff --git a/aten/src/ATen/cuda/CUDAContext.cpp b/aten/src/ATen/cuda/CUDAContext.cpp
index e48c020b03..0ecc111c4b 100644
--- a/aten/src/ATen/cuda/CUDAContext.cpp
+++ b/aten/src/ATen/cuda/CUDAContext.cpp
@@ -24,6 +24,8 @@ void initCUDAContextVectors() {
 void initDeviceProperty(DeviceIndex device_index) {
   cudaDeviceProp device_prop;
   AT_CUDA_CHECK(cudaGetDeviceProperties(&device_prop, device_index));
+  // patch for "too many resources requested for launch"
+  device_prop.maxThreadsPerBlock = device_prop.maxThreadsPerBlock / 2;

This file has been truncated. show original

pytorch-1.5-diff-jetpack-4.4.patch

diff --git a/aten/src/ATen/cuda/CUDAContext.cpp b/aten/src/ATen/cuda/CUDAContext.cpp
index fd51cc45e7..e3be2fd3bc 100644
--- a/aten/src/ATen/cuda/CUDAContext.cpp
+++ b/aten/src/ATen/cuda/CUDAContext.cpp
@@ -24,6 +24,8 @@ void initCUDAContextVectors() {
 void initDeviceProperty(DeviceIndex device_index) {
   cudaDeviceProp device_prop;
   AT_CUDA_CHECK(cudaGetDeviceProperties(&device_prop, device_index));
+  // patch for "too many resources requested for launch"
+  device_prop.maxThreadsPerBlock = device_prop.maxThreadsPerBlock / 2;

This file has been truncated. show original

pytorch-1.6-rc2-jetpack-4.4-GA.patch

diff --git a/aten/src/ATen/cuda/CUDAContext.cpp b/aten/src/ATen/cuda/CUDAContext.cpp
index fd51cc45e7..e3be2fd3bc 100644
--- a/aten/src/ATen/cuda/CUDAContext.cpp
+++ b/aten/src/ATen/cuda/CUDAContext.cpp
@@ -24,6 +24,8 @@ void initCUDAContextVectors() {
 void initDeviceProperty(DeviceIndex device_index) {
   cudaDeviceProp device_prop;
   AT_CUDA_CHECK(cudaGetDeviceProperties(&device_prop, device_index));
+  // patch for "too many resources requested for launch"
+  device_prop.maxThreadsPerBlock = device_prop.maxThreadsPerBlock / 2;

This file has been truncated. show original

Thanks.

nadeemm · October 18, 2021, 6:43pm