TLT-deepstream sample app problems : I found thatFRCNN, SSD , DSSD , RetinaNet and Detectnet_v2 can run successfully, but Yolov3 can’t

What is your TRT version?

I used command ‘dpkg -l | grep TensorRT’,and didn’t see tensorrt in my system, but it was 7.1.3 in pip3 list.

Can you install CUDA/cudnn/TRT with Jetpack 4.4?

yes,I installed these tools using the SDK manager.

What is your device? Nano/xaiver/NX or others?

I use TX2.

OK.
More,

  1. For TX2, when you build the Jetson TensorRT OSS Plugin according to https://github.com/NVIDIA-AI-IOT/deepstream_tlt_apps/tree/master/TRT-OSS/Jetson, please make sure you use GPU_ARCHS 62.

  2. If you install CUDA/cudnn/TRT with Jetpack 4.4, there should be below info. Please double check.

nvidia@nvidia:~$ dpkg -l |grep tensorrt
ii nvidia-container-csv-tensorrt 7.1.3.0-1+cuda10.2 arm64 Jetpack TensorRT CSV file
ii tensorrt 7.1.3.0-1+cuda10.2 arm64 Meta package of TensorRT

when I build the Jetson TensorRT OSS Plugin
command:
/usr/local/bin/cmake … -DGPU_ARCHS=62 -DTRT_LIB_DIR=/usr/lib/aarch64-linux-gnu/ -DCMAKE_C_COMPILER=/usr/bin/gcc -DTRT_BIN_DIR=pwd/out

nvidia@nvidia-desktop:~$ dpkg -l |grep cuda

ii cuda-command-line-tools-10-2 10.2.89-1 arm64 CUDA command-line tools
ii cuda-compiler-10-2 10.2.89-1 arm64 CUDA compiler
ii cuda-cudart-10-2 10.2.89-1 arm64 CUDA Runtime native Libraries
ii cuda-cudart-dev-10-2 10.2.89-1 arm64 CUDA Runtime native dev links, headers
ii cuda-cufft-10-2 10.2.89-1 arm64 CUFFT native runtime libraries
ii cuda-cufft-dev-10-2 10.2.89-1 arm64 CUFFT native dev links, headers
ii cuda-cuobjdump-10-2 10.2.89-1 arm64 CUDA cuobjdump
ii cuda-cupti-10-2 10.2.89-1 arm64 CUDA profiling tools runtime libs.
ii cuda-cupti-dev-10-2 10.2.89-1 arm64 CUDA profiling tools interface.
ii cuda-curand-10-2 10.2.89-1 arm64 CURAND native runtime libraries
ii cuda-curand-dev-10-2 10.2.89-1 arm64 CURAND native dev links, headers
ii cuda-cusolver-10-2 10.2.89-1 arm64 CUDA solver native runtime libraries
ii cuda-cusolver-dev-10-2 10.2.89-1 arm64 CUDA solver native dev links, headers
ii cuda-cusparse-10-2 10.2.89-1 arm64 CUSPARSE native runtime libraries
ii cuda-cusparse-dev-10-2 10.2.89-1 arm64 CUSPARSE native dev links, headers
ii cuda-documentation-10-2 10.2.89-1 arm64 CUDA documentation
ii cuda-driver-dev-10-2 10.2.89-1 arm64 CUDA Driver native dev stub library
ii cuda-gdb-10-2 10.2.89-1 arm64 CUDA-GDB
ii cuda-libraries-10-2 10.2.89-1 arm64 CUDA Libraries 10.2 meta-package
ii cuda-libraries-dev-10-2 10.2.89-1 arm64 CUDA Libraries 10.2 development meta-package
ii cuda-license-10-2 10.2.89-1 arm64 CUDA licenses
ii cuda-memcheck-10-2 10.2.89-1 arm64 CUDA-MEMCHECK
ii cuda-misc-headers-10-2 10.2.89-1 arm64 CUDA miscellaneous headers
ii cuda-npp-10-2 10.2.89-1 arm64 NPP native runtime libraries
ii cuda-npp-dev-10-2 10.2.89-1 arm64 NPP native dev links, headers
ii cuda-nvcc-10-2 10.2.89-1 arm64 CUDA nvcc
ii cuda-nvdisasm-10-2 10.2.89-1 arm64 CUDA disassembler
ii cuda-nvgraph-10-2 10.2.89-1 arm64 NVGRAPH native runtime libraries
ii cuda-nvgraph-dev-10-2 10.2.89-1 arm64 NVGRAPH native dev links, headers
ii cuda-nvml-dev-10-2 10.2.89-1 arm64 NVML native dev links, headers
ii cuda-nvprof-10-2 10.2.89-1 arm64 CUDA Profiler tools
ii cuda-nvprune-10-2 10.2.89-1 arm64 CUDA nvprune
ii cuda-nvrtc-10-2 10.2.89-1 arm64 NVRTC native runtime libraries
ii cuda-nvrtc-dev-10-2 10.2.89-1 arm64 NVRTC native dev links, headers
ii cuda-nvtx-10-2 10.2.89-1 arm64 NVIDIA Tools Extension
ii cuda-samples-10-2 10.2.89-1 arm64 CUDA example applications
ii cuda-toolkit-10-2 10.2.89-1 arm64 CUDA Toolkit 10.2 meta-package
ii cuda-tools-10-2 10.2.89-1 arm64 CUDA Tools meta-package
ii graphsurgeon-tf 7.1.3-1+cuda10.2 arm64 GraphSurgeon for TensorRT package
ii libcudnn8 8.0.0.180-1+cuda10.2 arm64 cuDNN runtime libraries
ii libcudnn8-dev 8.0.0.180-1+cuda10.2 arm64 cuDNN development libraries and headers
ii libcudnn8-doc 8.0.0.180-1+cuda10.2 arm64 cuDNN documents and samples
ii libnvinfer-bin 7.1.3-1+cuda10.2 arm64 TensorRT binaries
ii libnvinfer-dev 7.1.3-1+cuda10.2 arm64 TensorRT development libraries and headers
ii libnvinfer-doc 7.1.3-1+cuda10.2 all TensorRT documentation
ii libnvinfer-plugin-dev 7.1.3-1+cuda10.2 arm64 TensorRT plugin libraries
ii libnvinfer-plugin7 7.1.3-1+cuda10.2 arm64 TensorRT plugin libraries
ii libnvinfer-samples 7.1.3-1+cuda10.2 all TensorRT samples
ii libnvinfer7 7.1.3-1+cuda10.2 arm64 TensorRT runtime libraries
ii libnvonnxparsers-dev 7.1.3-1+cuda10.2 arm64 TensorRT ONNX libraries
ii libnvonnxparsers7 7.1.3-1+cuda10.2 arm64 TensorRT ONNX libraries
ii libnvparsers-dev 7.1.3-1+cuda10.2 arm64 TensorRT parsers libraries
ii libnvparsers7 7.1.3-1+cuda10.2 arm64 TensorRT parsers libraries
ii nvidia-container-csv-cuda 10.2.89-1 arm64 Jetpack CUDA CSV file
ii nvidia-container-csv-cudnn 8.0.0.180-1+cuda10.2 arm64 Jetpack CUDNN CSV file
ii nvidia-container-csv-tensorrt 7.1.3.0-1+cuda10.2 arm64 Jetpack TensorRT CSV file
ii nvidia-l4t-cuda 32.4.3-20200625213407 arm64 NVIDIA CUDA Package
ii python-libnvinfer 7.1.3-1+cuda10.2 arm64 Python bindings for TensorRT
ii python-libnvinfer-dev 7.1.3-1+cuda10.2 arm64 Python development package for TensorRT
ii python3-libnvinfer 7.1.3-1+cuda10.2 arm64 Python 3 bindings for TensorRT
ii python3-libnvinfer-dev 7.1.3-1+cuda10.2 arm64 Python 3 development package for TensorRT
ii tensorrt 7.1.3.0-1+cuda10.2 arm64 Meta package of TensorRT
ii uff-converter-tf 7.1.3-1+cuda10.2 arm64 UFF converter for TensorRT package

Thanks for the info. I find another topic YOLO model, Convert ETLT into an Engine which has the same error as yours.
Not sure what is happened. I will check it.

Thank you very much for your kind help. I will continue to solve the problem.

I sync the code and follow the guide from https://github.com/NVIDIA-AI-IOT/deepstream_tlt_apps. Then run yolo_v3 without any error. Please double check on your side, especially the replacement of libnvinfer_plugin.so.

I’m sure I compiled and replaced the file, and tried it more than once.

Please share the result of below command.
$ ll -sh /usr/lib/aarch64-linux-gnu/libnvinfer_plugin*

nvidia@nvidia-desktop:~/Public$ ll -sh /usr/lib/aarch64-linux-gnu/libnvinfer_plugin*
0 lrwxrwxrwx 1 root root 26 6月 6 13:00 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so -> libnvinfer_plugin.so.7.1.3*
0 lrwxrwxrwx 1 root root 31 9月 20 14:58 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7 -> libnvinfer_plugin.so.7.1.3_bak2*
8.7M -rwxr-xr-x 1 root root 8.7M 9月 20 14:57 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7.1.3*
5.4M -rw-r–r-- 1 root root 5.4M 6月 6 13:00 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7.1.3.bak
3.4M -rwxr-xr-x 1 root root 3.4M 9月 17 19:35 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7.1.3_bak2*
7.8M -rw-r–r-- 1 root root 7.8M 6月 6 13:00 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin_static.a

I’m afraid there is something wrong in your libs.

Please refer to Failling in building sample from TLT-DEEPSTREAM

More reference as below.

nvidia@nvidia:~/new_github/deepstream_tlt_apps$ ll -sh /usr/lib/aarch64-linux-gnu/libnvinfer_plugin*
0 lrwxrwxrwx 1 root root 26 Jun 6 13:00 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so -> libnvinfer_plugin.so.7.1.3*
0 lrwxrwxrwx 1 root root 26 Jun 6 13:00 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7 -> libnvinfer_plugin.so.7.1.3*
0 lrwxrwxrwx 1 root root 26 Sep 10 14:00 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7.0.0 -> libnvinfer_plugin.so.7.1.3*
4.5M -rwxr-xr-x 1 root root 4.5M Sep 10 14:00 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7.1.3*
7.8M -rw-r–r-- 1 root root 7.8M Jun 6 13:00 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin_static.a

It’s different. I’ll check my operation

I ran into the same issue, here are what I did to fix the issue:

  • I first reflash my Jetson AGX XAVIER to make sure whatever I installed on it is up to date, this step is optional
    Relevant system config after installation is done:
    Jetpack 4.4 (CUDA 10.2)
    TensorRT: 7.1.3
    Deepstream: 5.0

  • Follow instruction from the link shared by Morganh to build the TensorRT OSS, make sure you use release 7.0 NOT 7.1, I used 7.1 at first and YOLOV3 failed while other models (SSD , DSSD , RetinaNet) run without issue.

#back up the original file first
cd ~
mkdir libnvinfer-plugin-backups
sudo mv /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7.1.3 /home/[REPLACE_WITH_YOUR_USERNAME]/libnvinfer-plugin-backups/libnvinfer_plugin.so.7.1.3.bak
#replace with the .so file built in previous step
sudo cp /home/minh/TensorRT/build/out/libnvinfer_plugin.so.7.0.0 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7.1.3
sudo ldconfig
#check
ll -sh /usr/lib/aarch64-linux-gnu/libnvinfer_plugin*
#make sure in the printed ouput you get the same ouput as what Morganh showed, pay close attention to the modification date, the 2 original symlinks (libnvinfer_plugin.so & libnvinfer_plugin.so.7) & the libnvinfer_plugin_static.a file should still shown Jun while the two new file (libnvinfer_plugin.so.7.0.0 & libnvinfer_plugin.so.7.1.3), for me shown Sep

I can then run the Yolov3 example without error.

2 Likes

Thank you, you said in detail, and I followed the steps and solved the problem.

I had the exact same issue, @hyperlight’s answer solved it for me too 🎉️

Although I’m not sure why it works. Shouldn’t we be using the 7.1 release of TensorRT OSS with TRT 7.1.3?