Jetpack 4.4 Broke one of my programs

Autonomous Machines Jetson & Embedded Systems Jetson Nano
1

Went to run my bird camera this morning after loading 4.4, got the error below…

020-07-09 11:46:01.977034: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library ‘libcudart.so.10.0’; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory
2020-07-09 11:46:01.977101: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2020-07-09 11:46:01.977319: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library ‘libcudart.so.10.0’; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory
2020-07-09 11:46:01.977357: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Segmentation fault (core dumped)

2

What do you mean by “loading 4.4”? Did you flash the Jetson with a new release? Basically your program wants version 10.0 of libcudart.so dynamic library, and could not find it. The question is one of whether it was lost during an upgrade, or what actually is meant by “loading 4.4”.

3

Yes. “Loading 4.4” means flashing the new Jetson release from 4.3.

4

appears to be a torch issue post upgrade.

import torch
Traceback (most recent call last):
File “”, line 1, in
File “/usr/local/lib/python3.6/dist-packages/torch/init.py”, line 81, in
from torch._C import *
ImportError: libcudart.so.10.0: cannot open shared object file: No such file or directory

Found this on the Pytorch Forum page:
The JetPack 4.4 production release (L4T R32.4.3) only supports PyTorch 1.6.0 or newer, due to updates in cuDNN.

5

plot thickens, my opencv install is also busted post jetpack 4.4.

I don’t think this release was ready for prime time.

6

You can install PyTorch 1.6.0 from wheel from the link:

That should fix your issue.

7

I installed 1.6. That didn’t fix the issue. Went through my import list and tested them all. OpenCV isn’t working either. Currently reinstalling that…

8

You will have to rebuild OpenCV for JetPack 4.4. If you built from a sript you can probably just run it again. In the case of the pre-built docker image, there are multiple versions available, one for 4.3 (jp-r32.3.1-cv-4.3.0), and one for 4.4 (jp-r32.4.2-cv-4.3.0):

9

Yea, I’m rerunning the nvidia script. I just think in the release notes it should say “will break all your favorite things”.

10

Re: PyTorch: You can post in that thread re: 1.6.0 not working and you’ll probably get a response quicker than here considering @dusty_nv watches it.

Re: OpenCV: If you’re running Nvidia’s scipt to build 4.1.1, that may not work out of the box (with cudnn anyway) since the cudnn version needs to be manually specified since 8.0. You might try my script instead.

Re: breakage: it’s mostly a package management issue I think. Much of it could be resolved if Nvidia pre-built and packaged certain things with JetPack itself, but there may be reasons they can’t. When underlying libraries are upgraded, sometimes it breaks things built from source. If there’s an apt package for the thing, then it gets updated at the same time.

12

Thanks, the Nvidia one failed at 54%. Yours at 50%. Getting tempted to switch back to 4.3.

13

Error message? Appreciate you filing an issue on the repo if you still have the log.

15

[ 50%] Building CXX object modules/dnn/CMakeFiles/opencv_dnn.dir/src/layers/blank_layer.cpp.o
In file included from /tmp/build_opencv/opencv/modules/dnn/src/layers/…/cuda4dnn/csl/cudnn.hpp:8:0,
from /tmp/build_opencv/opencv/modules/dnn/src/layers/…/op_cuda.hpp:11,
from /tmp/build_opencv/opencv/modules/dnn/src/layers/blank_layer.cpp:43:
/tmp/build_opencv/opencv/modules/dnn/src/layers/…/cuda4dnn/primitives/…/csl/cudnn/convolution.hpp: In constructor ‘cv::dnn::cuda4dnn::csl::cudnn::ConvolutionAlgorithm::ConvolutionAlgorithm(const cv::dnn::cuda4dnn::csl::cudnn::Handle&, const cv::dnn::cuda4dnn::csl::cudnn::ConvolutionDescriptor&, const cv::dnn::cuda4dnn::csl::cudnn::FilterDescriptor&, const cv::dnn::cuda4dnn::csl::cudnn::TensorDescriptor&, const cv::dnn::cuda4dnn::csl::cudnn::TensorDescriptor&)’:
/tmp/build_opencv/opencv/modules/dnn/src/layers/…/cuda4dnn/primitives/…/csl/cudnn/convolution.hpp:266:21: error: ‘CUDNN_CONVOLUTION_FWD_PREFER_FASTEST’ was not declared in this scope
CUDNN_CONVOLUTION_FWD_PREFER_FASTEST,
^
/tmp/build_opencv/opencv/modules/dnn/src/layers/…/cuda4dnn/csl/cudnn/cudnn.hpp:23:53: note: in definition of macro ‘CUDA4DNN_CHECK_CUDNN’
::cv::dnn::cuda4dnn::csl::cudnn::detail::check((call), CV_Func, FILE, LINE)
^~~~
/tmp/build_opencv/opencv/modules/dnn/src/layers/…/cuda4dnn/primitives/…/csl/cudnn/convolution.hpp:266:21: note: suggested alternative: ‘CUDNN_CONVOLUTION_BWD_FILTER_ALGO_3’
CUDNN_CONVOLUTION_FWD_PREFER_FASTEST,
^
/tmp/build_opencv/opencv/modules/dnn/src/layers/…/cuda4dnn/csl/cudnn/cudnn.hpp:23:53: note: in definition of macro ‘CUDA4DNN_CHECK_CUDNN’
::cv::dnn::cuda4dnn::csl::cudnn::detail::check((call), CV_Func, FILE, LINE)
^~~~
/tmp/build_opencv/opencv/modules/dnn/src/layers/…/cuda4dnn/primitives/…/csl/cudnn/transpose_convolution.hpp: In constructor ‘cv::dnn::cuda4dnn::csl::cudnn::TransposeConvolutionAlgorithm::TransposeConvolutionAlgorithm(const cv::dnn::cuda4dnn::csl::cudnn::Handle&, const cv::dnn::cuda4dnn::csl::cudnn::ConvolutionDescriptor&, const cv::dnn::cuda4dnn::csl::cudnn::FilterDescriptor&, const cv::dnn::cuda4dnn::csl::cudnn::TensorDescriptor&, const cv::dnn::cuda4dnn::csl::cudnn::TensorDescriptor&)’:
/tmp/build_opencv/opencv/modules/dnn/src/layers/…/cuda4dnn/primitives/…/csl/cudnn/transpose_convolution.hpp:42:21: error: ‘CUDNN_CONVOLUTION_BWD_DATA_PREFER_FASTEST’ was not declared in this scope
CUDNN_CONVOLUTION_BWD_DATA_PREFER_FASTEST,
^
/tmp/build_opencv/opencv/modules/dnn/src/layers/…/cuda4dnn/csl/cudnn/cudnn.hpp:23:53: note: in definition of macro ‘CUDA4DNN_CHECK_CUDNN’
::cv::dnn::cuda4dnn::csl::cudnn::detail::check((call), CV_Func, FILE, LINE)
^~~~
/tmp/build_opencv/opencv/modules/dnn/src/layers/…/cuda4dnn/primitives/…/csl/cudnn/transpose_convolution.hpp:42:21: note: suggested alternative: ‘CUDNN_CONVOLUTION_BWD_DATA_ALGO_FFT’
CUDNN_CONVOLUTION_BWD_DATA_PREFER_FASTEST,
^
/tmp/build_opencv/opencv/modules/dnn/src/layers/…/cuda4dnn/csl/cudnn/cudnn.hpp:23:53: note: in definition of macro ‘CUDA4DNN_CHECK_CUDNN’
::cv::dnn::cuda4dnn::csl::cudnn::detail::check((call), CV_Func, FILE, LINE)
^~~~
modules/dnn/CMakeFiles/opencv_dnn.dir/build.make:721: recipe for target ‘modules/dnn/CMakeFiles/opencv_dnn.dir/src/layers/blank_layer.cpp.o’ failed
make[2]: *** [modules/dnn/CMakeFiles/opencv_dnn.dir/src/layers/blank_layer.cpp.o] Error 1
CMakeFiles/Makefile2:3798: recipe for target ‘modules/dnn/CMakeFiles/opencv_dnn.dir/all’ failed
make[1]: *** [modules/dnn/CMakeFiles/opencv_dnn.dir/all] Error 2
Makefile:162: recipe for target ‘all’ failed
make: *** [all] Error 2

16

Thx. Got the report on GitHub. I will try and replicate tomorrow.

17

@mdegans those OpenCV errors look similar to when I went to build PyTorch - there were some previously-deprecated functions removed from cuDNN 8.0. The OpenCV version might need updated (assuming they have patched master) if you want to build it with cuDNN support. What I did with PyTorch was to move to the latest release where they had updated their code to support cuDNN 8.0.

Note that the version of OpenCV that ships with JetPack should still be fine, I am assuming from above that a custom build of OpenCV is being referred to (e.g. with additional build options enabled). When you upgrade versions of JetPack that change the CUDA version, you typically need to also update packages that have dependencies on CUDA (e.g. PyTorch, TensorFlow, ect).

1 Like
18

So far I’ve been unable to fix the OpenCV. The make keeps dying at around 50%. I got pytorch up to 1.6.

19

I am running a build in Docker on an NX using r32.4.3 base image, which has a bunch of fixes (yay!). I will let you know if it suceeds and push the image after some basic tests if it does. After that, I’ll run it outside Docker on a fully updated 4.4 rootfs. If I can replicate the problem, I will let you know and try to fix it.

20

@dusty_nv

I’m building OpenCV 4.3.0 which did not recognize cuDNN out of the box. I had to manually specify -D CUDNN_VERSION='8.0'. It did build with 4.4 DP, but we’ll see…

@vondalej

Dustin is right. OpenCV included in the apt repos should work fine, but I don’t believe it has any of the CUDA stuff, so it’ll be using your CPU exclusively. Depending on your use, that may be acceptable. I believe it’s installed by default. You can check with opencv_version --verbose. My NX shows:

 $ opencv_version --verbose

General configuration for OpenCV 4.1.1 =====================================
  Version control:               4.1.1-2-gd5a58aa75

  Platform:
    Timestamp:                   2019-12-13T17:25:11Z
    Host:                        Linux 4.9.140-tegra aarch64
    CMake:                       3.10.2
    CMake generator:             Unix Makefiles
    CMake build tool:            /usr/bin/make
    Configuration:               Release

  CPU/HW features:
    Baseline:                    NEON FP16
      required:                  NEON
      disabled:                  VFPV3

  C/C++:
    Built as dynamic libs?:      YES
    C++ Compiler:                /usr/bin/c++  (ver 7.4.0)
    C++ flags (Release):         -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG
    C++ flags (Debug):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -g  -O0 -DDEBUG -D_DEBUG
    C Compiler:                  /usr/bin/cc
    C flags (Release):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -O3 -DNDEBUG  -DNDEBUG
    C flags (Debug):             -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -g  -O0 -DDEBUG -D_DEBUG
    Linker flags (Release):      -Wl,--gc-sections  
    Linker flags (Debug):        -Wl,--gc-sections  
    ccache:                      NO
    Precompiled headers:         NO
    Extra dependencies:          dl m pthread rt
    3rdparty dependencies:

  OpenCV modules:
    To be built:                 calib3d core dnn features2d flann gapi highgui imgcodecs imgproc ml objdetect photo python2 python3 stitching ts video videoio
    Disabled:                    world
    Disabled by dependency:      -
    Unavailable:                 java js
    Applications:                tests perf_tests examples apps
    Documentation:               NO
    Non-free algorithms:         NO

  GUI: 
    GTK+:                        YES (ver 2.24.32)
      GThread :                  YES (ver 2.56.4)
      GtkGlExt:                  NO

  Media I/O: 
    ZLib:                        /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.11)
    JPEG:                        /usr/lib/aarch64-linux-gnu/libjpeg.so (ver 80)
    WEBP:                        build (ver encoder: 0x020e)
    PNG:                         /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.6.34)
    TIFF:                        /usr/lib/aarch64-linux-gnu/libtiff.so (ver 42 / 4.0.9)
    JPEG 2000:                   build (ver 1.900.1)
    HDR:                         YES
    SUNRASTER:                   YES
    PXM:                         YES
    PFM:                         YES

  Video I/O:
    FFMPEG:                      YES
      avcodec:                   YES (57.107.100)
      avformat:                  YES (57.83.100)
      avutil:                    YES (55.78.100)
      swscale:                   YES (4.8.100)
      avresample:                NO
    GStreamer:                   YES (1.14.5)
    v4l/v4l2:                    YES (linux/videodev2.h)

  Parallel framework:            TBB (ver 2017.0 interface 9107)

  Trace:                         YES (with Intel ITT)

  Other third-party libraries:
    Lapack:                      NO
    Eigen:                       YES (ver 3.3.4)
    Custom HAL:                  YES (carotene (ver 0.0.1))
    Protobuf:                    build (3.5.1)

  Python 2:
    Interpreter:                 /usr/bin/python2.7 (ver 2.7.15)
    Libraries:                   /usr/lib/aarch64-linux-gnu/libpython2.7.so (ver 2.7.15+)
    numpy:                       /usr/lib/python2.7/dist-packages/numpy/core/include (ver 1.13.3)
    install path:                lib/python2.7/dist-packages/cv2/python-2.7

  Python 3:
    Interpreter:                 /usr/bin/python3 (ver 3.6.9)
    Libraries:                   /usr/lib/aarch64-linux-gnu/libpython3.6m.so (ver 3.6.9)
    numpy:                       /usr/lib/python3/dist-packages/numpy/core/include (ver 1.13.3)
    install path:                lib/python3.6/dist-packages/cv2/python-3.6

  Python (for build):            /usr/bin/python2.7

  Java:                          
    ant:                         NO
    JNI:                         NO
    Java wrappers:               NO
    Java tests:                  NO

  Install to:                    /usr
-----------------------------------------------------------------
21

What’s the point of the Nano if I can’t use the GPU?

Problem solved thanks to JetsonHacks.

Program is still broken…have to fix tensorflow now.

#worst"upgrade"ever

22

@vondalej

OpenCV itself does not use the GPU for the most part since it’s:
a. very old (this is not necessarily bad, but most code is written for the CPU)
b. an Intel project

In any case, running my script with ./build_opencv.sh master (specifying master branch explicitly), will likely build, and it will have GPU acceleration (but only for the cuda module, eg. cv2.cuda.resize not cv.resize).

No matter whose script you use, most code you will find on the internet will have to be rewritten to use the cuda module. There is an experimental graph api which aims to be independent of computation backend, but it currently has no cuda support.