Yolov4 not working in deepstream app?

Could you please double check your command? Above seems to be not correct.

BTW, where did you run tlt-converter? In Jetson device or your host?

I am running on jetson device. above command work well on my system, but not worked on jetson,

There is two “/opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream_python_apps/apps/deepstream_tlt_apps-master/models/yolov4/yolov4_resnet18.etlt_b1_gpu0_fp16.engine” . Can you double check your command?

No, There is no two "/opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream_python_apps/apps/deepstream_tlt_apps-master/models/yolov4/yolov4_resnet18.etlt_b1_gpu0_fp16.engine"


sudo ./tlt-converter -e /opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream_python_apps/apps/deepstream_tlt_apps-master/models/yolov4/yolov4_resnet18.etlt_b1_gpu0_fp16.engine -p Input,1x3x544x960,1x3x544x960,1x3x544x960 -t fp16 -d 3,544,960 -k nvidia_tlt -m 1 tlt_encode /opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream_python_apps/apps/deepstream_tlt_apps-master/models/yolov4/yolov4_resnet18.etlt

Actually these two are different.
one for model output file /opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream_python_apps/apps/deepstream_tlt_apps-master/models/yolov4/yolov4_resnet18.etlt_b1_gpu0_fp16.engine

second for model etlt file /opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream_python_apps/apps/deepstream_tlt_apps-master/models/yolov4/yolov4_resnet18.etlt

See above. It is not expected.

I followed this link

Can you double check your command?
Why there is “tlt_encode” in “-k nvidia_tlt -m 1 tlt_encode” ?

I removed tlt encode from command but same error.

Error:

[WARNING] onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[INFO] ModelImporter.cpp:135: No importer registered for op: BatchedNMSDynamic_TRT. Attempting to import as plugin.
[INFO] builtin_op_importers.cpp:3659: Searching for plugin: BatchedNMSDynamic_TRT, plugin_version: 1, plugin_namespace:
[ERROR] INVALID_ARGUMENT: getPluginCreator could not find plugin BatchedNMSDynamic_TRT version 1
ERROR: builtin_op_importers.cpp:3661 In function importFallbackPluginImporter:
[8] Assertion failed: creator && “Plugin not found, are the plugin name, version, and namespace correct?”
Assertion failed: creator && “Plugin not found, are the plugin name, version, and namespace correct?”
[ERROR] Failed to parse the model, please check the encoding key to make sure it’s correct
[INFO] Detected input dimensions from the model: (-1, 3, 544, 960)
[INFO] Model has dynamic shape. Setting up optimization profiles.
[INFO] Using optimization profile min shape: (1, 3, 544, 960) for input: Input
[INFO] Using optimization profile opt shape: (1, 3, 544, 960) for input: Input
[INFO] Using optimization profile max shape: (1, 3, 544, 960) for input: Input
[ERROR] Network must have at least one output
[ERROR] Network validation failed.
[ERROR] Unable to create engine
Segmentation fault

Code:

sudo ./tlt-converter -e /opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream_python_apps/apps/deepstream_tlt_apps-master/models/yolov4/yolov4_resnet18.etlt_b1_gpu0_fp16.engine -p Input,1x3x544x960,1x3x544x960,1x3x544x960 -t fp16 -d 3,544,960 -k nvidia_tlt -m 1 /opt/nvidia/deepstream/deepstream-5.1/sources/apps/sample_apps/deepstream_python_apps/apps/deepstream_tlt_apps-master/models/yolov4/yolov4_resnet18.etlt

Could you please run similar command as below?
In Tlt-convert for custom trained YoloV4 model failed on Jetson Nano 4G - #34 by Morganh , I generate fp16 trt engine successfully in Nano.

$ ./tlt-converter -k nvidia_tlt -d 3,544,960 -e trt.fp16.engine -t fp16 -p Input,1x3x544x960,1x3x544x960,2x3x544x960 yolov3_resnet18.etlt

Above command throw same error on my side.

Error:
[WARNING] onnx2trt_utils.cpp:220: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[WARNING] onnx2trt_utils.cpp:246: One or more weights outside the range of INT32 was clamped
[INFO] ModelImporter.cpp:135: No importer registered for op: BatchedNMSDynamic_TRT. Attempting to import as plugin.
[INFO] builtin_op_importers.cpp:3659: Searching for plugin: BatchedNMSDynamic_TRT, plugin_version: 1, plugin_namespace:
[ERROR] INVALID_ARGUMENT: getPluginCreator could not find plugin BatchedNMSDynamic_TRT version 1
ERROR: builtin_op_importers.cpp:3661 In function importFallbackPluginImporter:
[8] Assertion failed: creator && “Plugin not found, are the plugin name, version, and namespace correct?”
Assertion failed: creator && “Plugin not found, are the plugin name, version, and namespace correct?”
[ERROR] Failed to parse the model, please check the encoding key to make sure it’s correct
[INFO] Detected input dimensions from the model: (-1, 3, 544, 960)
[INFO] Model has dynamic shape. Setting up optimization profiles.
[INFO] Using optimization profile min shape: (1, 3, 544, 960) for input: Input
[INFO] Using optimization profile opt shape: (1, 3, 544, 960) for input: Input
[INFO] Using optimization profile max shape: (1, 3, 544, 960) for input: Input
[ERROR] Network must have at least one output
[ERROR] Network validation failed.
[ERROR] Unable to create engine
Segmentation fault

OK, to narrow down, could you try to generate fp32 trt engine via “-t fp32” ?

Fp32, FP16 generated successfully on PC, but on jetson FP32 throw error. is it possible to use fp32 file make on system to use in jetson, because fp32 is general mode?

Sorry, I not added my modules versions. Below is my jetson module details.
CUDA = 10.2
TRT = 7.1.3

It is not expected to get these errors when run tlt-converter. So, may I get more info about your results?
In host pc, do you run tlt-covnerter inside tlt docker or ouside the tlt docker?
Both fp16 and fp32 can be generated successfully. Right?
In Jetson, do you mean both fp16 and fp32 can not be generated. Right?

In host PC, I used tlt-converter outside the docker.
Yes both fp16 and fp32 generated successfully on host PC.
Yes in jetson both (fp16, fp32) can not generated.

Which Jetpack version did you install? You are running in Nano, right?
Can you run following command and share the result too?
$ dpkg -l |grep cuda
$ uname -a

The jet-pack version is 4.4, I am using jetson nano (4GB)

$ dpkg -l |grep cuda

ii cuda-command-line-tools-10-2 10.2.89-1 arm64 CUDA command-line tools
ii cuda-compiler-10-2 10.2.89-1 arm64 CUDA compiler
ii cuda-cudart-10-2 10.2.89-1 arm64 CUDA Runtime native Libraries
ii cuda-cudart-dev-10-2 10.2.89-1 arm64 CUDA Runtime native dev links, headers
ii cuda-cufft-10-2 10.2.89-1 arm64 CUFFT native runtime libraries
ii cuda-cufft-dev-10-2 10.2.89-1 arm64 CUFFT native dev links, headers
ii cuda-cuobjdump-10-2 10.2.89-1 arm64 CUDA cuobjdump
ii cuda-cupti-10-2 10.2.89-1 arm64 CUDA profiling tools runtime libs.
ii cuda-cupti-dev-10-2 10.2.89-1 arm64 CUDA profiling tools interface.
ii cuda-curand-10-2 10.2.89-1 arm64 CURAND native runtime libraries
ii cuda-curand-dev-10-2 10.2.89-1 arm64 CURAND native dev links, headers
ii cuda-cusolver-10-2 10.2.89-1 arm64 CUDA solver native runtime libraries
ii cuda-cusolver-dev-10-2 10.2.89-1 arm64 CUDA solver native dev links, headers
ii cuda-cusparse-10-2 10.2.89-1 arm64 CUSPARSE native runtime libraries
ii cuda-cusparse-dev-10-2 10.2.89-1 arm64 CUSPARSE native dev links, headers
ii cuda-documentation-10-2 10.2.89-1 arm64 CUDA documentation
ii cuda-driver-dev-10-2 10.2.89-1 arm64 CUDA Driver native dev stub library
ii cuda-gdb-10-2 10.2.89-1 arm64 CUDA-GDB
ii cuda-libraries-10-2 10.2.89-1 arm64 CUDA Libraries 10.2 meta-package
ii cuda-libraries-dev-10-2 10.2.89-1 arm64 CUDA Libraries 10.2 development meta-package
ii cuda-license-10-2 10.2.89-1 arm64 CUDA licenses
ii cuda-memcheck-10-2 10.2.89-1 arm64 CUDA-MEMCHECK
ii cuda-misc-headers-10-2 10.2.89-1 arm64 CUDA miscellaneous headers
ii cuda-npp-10-2 10.2.89-1 arm64 NPP native runtime libraries
ii cuda-npp-dev-10-2 10.2.89-1 arm64 NPP native dev links, headers
ii cuda-nvcc-10-2 10.2.89-1 arm64 CUDA nvcc
ii cuda-nvdisasm-10-2 10.2.89-1 arm64 CUDA disassembler
ii cuda-nvgraph-10-2 10.2.89-1 arm64 NVGRAPH native runtime libraries
ii cuda-nvgraph-dev-10-2 10.2.89-1 arm64 NVGRAPH native dev links, headers
ii cuda-nvml-dev-10-2 10.2.89-1 arm64 NVML native dev links, headers
ii cuda-nvprof-10-2 10.2.89-1 arm64 CUDA Profiler tools
ii cuda-nvprune-10-2 10.2.89-1 arm64 CUDA nvprune
ii cuda-nvrtc-10-2 10.2.89-1 arm64 NVRTC native runtime libraries
ii cuda-nvrtc-dev-10-2 10.2.89-1 arm64 NVRTC native dev links, headers
ii cuda-nvtx-10-2 10.2.89-1 arm64 NVIDIA Tools Extension
ii cuda-repo-l4t-10-2-local-10.2.89 1.0-1 arm64 cuda repository configuration files
ii cuda-samples-10-2 10.2.89-1 arm64 CUDA example applications
ii cuda-toolkit-10-2 10.2.89-1 arm64 CUDA Toolkit 10.2 meta-package
ii cuda-tools-10-2 10.2.89-1 arm64 CUDA Tools meta-package
ii graphsurgeon-tf 7.1.3-1+cuda10.2 arm64 GraphSurgeon for TensorRT package
ii libcudnn8 8.0.0.180-1+cuda10.2 arm64 cuDNN runtime libraries
ii libcudnn8-dev 8.0.0.180-1+cuda10.2 arm64 cuDNN development libraries and headers
ii libcudnn8-doc 8.0.0.180-1+cuda10.2 arm64 cuDNN documents and samples
ii libnvinfer-bin 7.1.3-1+cuda10.2 arm64 TensorRT binaries
ii libnvinfer-dev 7.1.3-1+cuda10.2 arm64 TensorRT development libraries and headers
ii libnvinfer-doc 7.1.3-1+cuda10.2 all TensorRT documentation
ii libnvinfer-plugin-dev 7.1.3-1+cuda10.2 arm64 TensorRT plugin libraries
ii libnvinfer-plugin7 7.1.3-1+cuda10.2 arm64 TensorRT plugin libraries
ii libnvinfer-samples 7.1.3-1+cuda10.2 all TensorRT samples
ii libnvinfer7 7.1.3-1+cuda10.2 arm64 TensorRT runtime libraries
ii libnvonnxparsers-dev 7.1.3-1+cuda10.2 arm64 TensorRT ONNX libraries
ii libnvonnxparsers7 7.1.3-1+cuda10.2 arm64 TensorRT ONNX libraries
ii libnvparsers-dev 7.1.3-1+cuda10.2 arm64 TensorRT parsers libraries
ii libnvparsers7 7.1.3-1+cuda10.2 arm64 TensorRT parsers libraries
ii nvidia-container-csv-cuda 10.2.89-1 arm64 Jetpack CUDA CSV file
ii nvidia-container-csv-cudnn 8.0.0.180-1+cuda10.2 arm64 Jetpack CUDNN CSV file
ii nvidia-container-csv-tensorrt 7.1.3.0-1+cuda10.2 arm64 Jetpack TensorRT CSV file
ii nvidia-l4t-cuda 32.5.1-20210519110732 arm64 NVIDIA CUDA Package
ii python-libnvinfer 7.1.3-1+cuda10.2 arm64 Python bindings for TensorRT
ii python-libnvinfer-dev 7.1.3-1+cuda10.2 arm64 Python development package for TensorRT
ii python3-libnvinfer 7.1.3-1+cuda10.2 arm64 Python 3 bindings for TensorRT
ii python3-libnvinfer-dev 7.1.3-1+cuda10.2 arm64 Python 3 development package for TensorRT
ii tensorrt 7.1.3.0-1+cuda10.2 arm64 Meta package of TensorRT
ii uff-converter-tf 7.1.3-1+cuda10.2 arm64 UFF converter for TensorRT package

$ uname -a

Linux facit-desktop 4.9.201-tegra #1 SMP PREEMPT Wed May 5 09:31:36 PDT 2021 aarch64 aarch64 aarch64 GNU/Linux

Did you ever build TRT OSS plugin ? See GitHub - NVIDIA-AI-IOT/deepstream_tlt_apps: Sample apps to demonstrate how to deploy models trained with TLT on DeepStream

Can you show the result of
$ ls -l /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so*

No I have not build TRT OSS Plugin, for deepstream-tlt-apps

$ ls -l /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so*

lrwxrwxrwx 1 root root 26 جون 6 2020 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so → libnvinfer_plugin.so.7.1.3
lrwxrwxrwx 1 root root 26 جون 6 2020 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7 → libnvinfer_plugin.so.7.1.3
-rw-r–r-- 1 root root 5630344 جون 6 2020 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7.1.3

Please backup the original libnvinfer_plugin.so.7.1.3 and then follow deepstream_tlt_apps/TRT-OSS/Jetson at master · NVIDIA-AI-IOT/deepstream_tlt_apps · GitHub to generate the new plugin and replace.

sudo mv /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7.1.3   ${HOME}/libnvinfer_plugin.so.7.1.3.bak   
sudo cp `pwd`/out/libnvinfer_plugin.so.7.m.n  /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7.1.3
sudo ldconfig

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.