How to build dlib in jetson nano in DeepStream-l4t container

manjunathapoojari9 · July 29, 2020, 7:34am

Hi,
I’m trying to build dlib from the source in DeepStream-l4t docker container on a jetson nano. while building CMake is detecting Cuda-10.2. but it’s not enabling Cuda since cudnn not found.
Is DeepStream-l4t contains cudnn files or do I have to install it? Please help me out. Im not sure from where to download cudnn files for aarch64 system.

The build log is

Found CUDA: /usr/local/cuda-10.2 (found suitable version “10.2”, minimum required is “7.5”)
– Looking for cuDNN install…
– *** cuDNN V5.0 OR GREATER NOT FOUND.
***
– *** Dlib requires cuDNN V5.0 OR GREATER. Since cuDNN is not found DLIB WILL NOT USE CUDA. ***
– *** If you have cuDNN then set CMAKE_PREFIX_PATH to include cuDNN’s folder. ***
– Disabling CUDA support for dlib. DLIB WILL NOT USE CUDA

AastaLLL · July 30, 2020, 3:15am

Hi,

cuDNN is pre-installed in the Deepstream-l4t image.

The root cause is that cuDNN version information is moved from cudnn.h into cudnn_version.h.
So you may need some update in the dlib CMakeLists.txt.’

Thanks.

manjunathapoojari9 · July 30, 2020, 5:15am

Hi,

I replaced cudnn.h to cudnn_version.h in
./dlib/dlib/cmake_utils/test_for_cudnn/find_cudnn.txt but it didn’t work.

and also I did a quick search for both cudnn.h and cudnn_version.h using
find / -type f -name cudnn_version.h in
nvcr.io/nvidia/deepstream-l4t:5.0-dp-20.04-base image I didn’t find these files.

Can you please help me out on this. Thanks

AastaLLL · July 31, 2020, 4:30am

Hi,

Sure.
Would you mind to share which dlib version do you use first?

Thanks.

manjunathapoojari9 · July 31, 2020, 7:16am

Hi,

I tried with 19.16 and 19.4, both was giving same errors.

Thanks.

AastaLLL · August 4, 2020, 9:13am

Hi,

Here are some detail of the failure.
The cuDNN cannot be included since the cudnn_test is failed.

github.com

davisking/dlib/blob/v19.16/dlib/CMakeLists.txt#L661


      
          
          
set(CUDA_TEST_CMAKE_FLAGS 
             "-DCMAKE_PREFIX_PATH=${CMAKE_PREFIX_PATH}"
             "-DCMAKE_INCLUDE_PATH=${CMAKE_INCLUDE_PATH}"
             "-DCMAKE_LIBRARY_PATH=${CMAKE_LIBRARY_PATH}")
          
          
if (NOT MSVC) # see https://github.com/davisking/dlib/issues/363
             list(APPEND CUDA_TEST_CMAKE_FLAGS "-DCUDA_HOST_COMPILER=${CUDA_HOST_COMPILER}")
          endif()
          
          
try_compile(cuda_test_compile_worked 
             ${PROJECT_BINARY_DIR}/cuda_test_build 
             ${PROJECT_SOURCE_DIR}/cmake_utils/test_for_cuda cuda_test
             CMAKE_FLAGS ${CUDA_TEST_CMAKE_FLAGS}
             OUTPUT_VARIABLE try_compile_output_message
             )
          if (NOT cuda_test_compile_worked)
             string(REPLACE "\n" "\n   ***   " try_compile_output_message "${try_compile_output_message}")
             message(STATUS "*****************************************************************************************************************")
             message(STATUS "*** CUDA was found but your compiler failed to compile a simple CUDA program so dlib isn't going to use CUDA. ")
             message(STATUS "*** The output of the failed CUDA test compile is shown below: ")

The reason why cudnn_test failed is that some API has changed in the cuDNN v8.0.
So the corresponding update is needed for the dlib to ensure the compatibility:

root@nvidia-desktop:/home/nvidia/dlib-19.16/dlib/cmake_utils/test_for_cudnn# make   
Scanning dependencies of target cudnn_test
[ 50%] Building CXX object CMakeFiles/cudnn_test.dir/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp.o
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp: In member function 'void dlib::cuda::tensor_conv::setup(const dlib::tensor&, const dlib::tensor&, int, int, int, int)':
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:851:57: error: 'CUDNN_CONVOLUTION_FWD_PREFER_FASTEST' was not declared in this scope
                         dnn_prefer_fastest_algorithms()?CUDNN_CONVOLUTION_FWD_PREFER_FASTEST:CUDNN_CONVOLUTION_FWD_NO_WORKSPACE,
                                                         ^
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:43:33: note: in definition of macro 'CHECK_CUDNN'
     const cudnnStatus_t error = call;                                         \
                                 ^~~~
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:851:57: note: suggested alternative: 'CUDNN_CONVOLUTION_BWD_FILTER_ALGO_3'
                         dnn_prefer_fastest_algorithms()?CUDNN_CONVOLUTION_FWD_PREFER_FASTEST:CUDNN_CONVOLUTION_FWD_NO_WORKSPACE,
                                                         ^
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:43:33: note: in definition of macro 'CHECK_CUDNN'
     const cudnnStatus_t error = call;                                         \
                                 ^~~~
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:851:94: error: 'CUDNN_CONVOLUTION_FWD_NO_WORKSPACE' was not declared in this scope
                         dnn_prefer_fastest_algorithms()?CUDNN_CONVOLUTION_FWD_PREFER_FASTEST:CUDNN_CONVOLUTION_FWD_NO_WORKSPACE,
                                                                                              ^
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:43:33: note: in definition of macro 'CHECK_CUDNN'
     const cudnnStatus_t error = call;                                         \
                                 ^~~~
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:851:94: note: suggested alternative: 'CUDNN_CONVOLUTION_FWD_ALGO_WINOGRAD'
                         dnn_prefer_fastest_algorithms()?CUDNN_CONVOLUTION_FWD_PREFER_FASTEST:CUDNN_CONVOLUTION_FWD_NO_WORKSPACE,
                                                                                              ^
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:43:33: note: in definition of macro 'CHECK_CUDNN'
     const cudnnStatus_t error = call;                                         \
                                 ^~~~
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:845:29: error: 'cudnnGetConvolutionForwardAlgorithm' was not declared in this scope
                 CHECK_CUDNN(cudnnGetConvolutionForwardAlgorithm(
                             ^
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:43:33: note: in definition of macro 'CHECK_CUDNN'
     const cudnnStatus_t error = call;                                         \
                                 ^~~~
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:845:29: note: suggested alternative: 'cudnnGetConvolutionForwardAlgorithm_v7'
                 CHECK_CUDNN(cudnnGetConvolutionForwardAlgorithm(
                             ^
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:43:33: note: in definition of macro 'CHECK_CUDNN'
     const cudnnStatus_t error = call;                                         \
                                 ^~~~
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:873:57: error: 'CUDNN_CONVOLUTION_BWD_DATA_PREFER_FASTEST' was not declared in this scope
                         dnn_prefer_fastest_algorithms()?CUDNN_CONVOLUTION_BWD_DATA_PREFER_FASTEST:CUDNN_CONVOLUTION_BWD_DATA_NO_WORKSPACE,
                                                         ^
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:43:33: note: in definition of macro 'CHECK_CUDNN'
     const cudnnStatus_t error = call;                                         \
                                 ^~~~
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:873:57: note: suggested alternative: 'CUDNN_CONVOLUTION_BWD_DATA_ALGO_FFT'
                         dnn_prefer_fastest_algorithms()?CUDNN_CONVOLUTION_BWD_DATA_PREFER_FASTEST:CUDNN_CONVOLUTION_BWD_DATA_NO_WORKSPACE,
                                                         ^
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:43:33: note: in definition of macro 'CHECK_CUDNN'
     const cudnnStatus_t error = call;                                         \
                                 ^~~~
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:873:99: error: 'CUDNN_CONVOLUTION_BWD_DATA_NO_WORKSPACE' was not declared in this scope
                         dnn_prefer_fastest_algorithms()?CUDNN_CONVOLUTION_BWD_DATA_PREFER_FASTEST:CUDNN_CONVOLUTION_BWD_DATA_NO_WORKSPACE,
                                                                                                   ^
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:43:33: note: in definition of macro 'CHECK_CUDNN'
     const cudnnStatus_t error = call;                                         \
                                 ^~~~
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:873:99: note: suggested alternative: 'CUDNN_CONVOLUTION_BWD_DATA_ALGO_WINOGRAD'
                         dnn_prefer_fastest_algorithms()?CUDNN_CONVOLUTION_BWD_DATA_PREFER_FASTEST:CUDNN_CONVOLUTION_BWD_DATA_NO_WORKSPACE,
                                                                                                   ^
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:43:33: note: in definition of macro 'CHECK_CUDNN'
     const cudnnStatus_t error = call;                                         \
                                 ^~~~
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:867:29: error: 'cudnnGetConvolutionBackwardDataAlgorithm' was not declared in this scope
                 CHECK_CUDNN(cudnnGetConvolutionBackwardDataAlgorithm(
                             ^
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:43:33: note: in definition of macro 'CHECK_CUDNN'
     const cudnnStatus_t error = call;                                         \
                                 ^~~~
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:867:29: note: suggested alternative: 'cudnnGetConvolutionBackwardDataAlgorithm_v7'
                 CHECK_CUDNN(cudnnGetConvolutionBackwardDataAlgorithm(
                             ^
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:43:33: note: in definition of macro 'CHECK_CUDNN'
     const cudnnStatus_t error = call;                                         \
                                 ^~~~
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:896:57: error: 'CUDNN_CONVOLUTION_BWD_FILTER_PREFER_FASTEST' was not declared in this scope
                         dnn_prefer_fastest_algorithms()?CUDNN_CONVOLUTION_BWD_FILTER_PREFER_FASTEST:CUDNN_CONVOLUTION_BWD_FILTER_NO_WORKSPACE,
                                                         ^
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:43:33: note: in definition of macro 'CHECK_CUDNN'
     const cudnnStatus_t error = call;                                         \
                                 ^~~~
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:896:57: note: suggested alternative: 'CUDNN_CONVOLUTION_BWD_FILTER_ALGO_FFT'
                         dnn_prefer_fastest_algorithms()?CUDNN_CONVOLUTION_BWD_FILTER_PREFER_FASTEST:CUDNN_CONVOLUTION_BWD_FILTER_NO_WORKSPACE,
                                                         ^
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:43:33: note: in definition of macro 'CHECK_CUDNN'
     const cudnnStatus_t error = call;                                         \
                                 ^~~~
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:896:101: error: 'CUDNN_CONVOLUTION_BWD_FILTER_NO_WORKSPACE' was not declared in this scope
                         dnn_prefer_fastest_algorithms()?CUDNN_CONVOLUTION_BWD_FILTER_PREFER_FASTEST:CUDNN_CONVOLUTION_BWD_FILTER_NO_WORKSPACE,
                                                                                                     ^
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:43:33: note: in definition of macro 'CHECK_CUDNN'
     const cudnnStatus_t error = call;                                         \
                                 ^~~~
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:896:101: note: suggested alternative: 'CUDNN_CONVOLUTION_BWD_FILTER_ALGO_WINOGRAD'
                         dnn_prefer_fastest_algorithms()?CUDNN_CONVOLUTION_BWD_FILTER_PREFER_FASTEST:CUDNN_CONVOLUTION_BWD_FILTER_NO_WORKSPACE,
                                                                                                     ^
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:43:33: note: in definition of macro 'CHECK_CUDNN'
     const cudnnStatus_t error = call;                                         \
                                 ^~~~
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:890:29: error: 'cudnnGetConvolutionBackwardFilterAlgorithm' was not declared in this scope
                 CHECK_CUDNN(cudnnGetConvolutionBackwardFilterAlgorithm(
                             ^
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:43:33: note: in definition of macro 'CHECK_CUDNN'
     const cudnnStatus_t error = call;                                         \
                                 ^~~~
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:890:29: note: suggested alternative: 'cudnnGetConvolutionBackwardFilterAlgorithm_v7'
                 CHECK_CUDNN(cudnnGetConvolutionBackwardFilterAlgorithm(
                             ^
/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp:43:33: note: in definition of macro 'CHECK_CUDNN'
     const cudnnStatus_t error = call;                                         \
                                 ^~~~
CMakeFiles/cudnn_test.dir/build.make:62: recipe for target 'CMakeFiles/cudnn_test.dir/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp.o' failed
make[2]: *** [CMakeFiles/cudnn_test.dir/home/nvidia/dlib-19.16/dlib/cuda/cudnn_dlibapi.cpp.o] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/cudnn_test.dir/all' failed
make[1]: *** [CMakeFiles/cudnn_test.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

It looks like the latest source on the GitHub already has the cuDNN v8.0 support.
Please compile it with the GitHub source directly.

Thanks.

Topic		Replies	Views
Dlib testing on Jetson Nano, TX2 and Xavier Jetson TX2	6	2051	October 18, 2021
OpenCV 4.2.0 and CuDNN for Jetson Nano? Jetson Nano opencv	56	11208	October 18, 2021
Installing dlib_for_arm Jetson Nano cudnn	2	767	October 18, 2021
I can't use CUDA dnn on Jetson nano + python Jetson Nano opencv	11	3905	October 18, 2021
deepstream for tesla in docker DeepStream SDK	8	1770	June 5, 2018
Problems with deepstream docker on Jetson nano DeepStream SDK	13	1058	October 12, 2021
Jetson Nano 2GB, dlib in dockerfile Jetson Nano docker	5	918	June 6, 2022
Issue with dlib Jetson Nano python	4	2257	October 18, 2021
Issue installing dlib on Jetson AGX Xavier Jetson AGX Xavier	8	2576	October 18, 2021
Jetson Nano, Build dlib in dockerfile Jetson Nano cuda , docker	2	866	January 18, 2023

How to build dlib in jetson nano in DeepStream-l4t container

Related topics