TAO Toolkit 4.0, Yolo v4 not working with Deepstream nvinfer - parseModel: Failed to parse ONNX model

Accelerated Computing Intelligent Video Analytics TAO Toolkit
, ,
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 61
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only) 510
• Issue Type( questions, new requirements, bugs) bug
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing) Export yolo_v4 etlt model using tao toolkit and try to run with deepstream nvinfer
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

I have trained a yolo_v4 model using TAO toolkit 4.0. The TAO version was installed with pip install nvidia-tao and is the latest version. I exported the model in etlt format and tried to use it in a deepstream application. The NvInfer plugin is supposed to generate an engine file from etlt and run but it throws a ONNX parse error.

WARNING: [TRT]: onnx2trt_utils.cpp:369: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
WARNING: [TRT]: onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
…
WARNING: [TRT]: onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
parseModel: Failed to parse ONNX model
ERROR: tlt/tlt_decode.cpp:389 Failed to build network, error in model parsing.
ERROR: …/nvdsinfer/nvdsinfer_model_builder.cpp:723 Failed to create network using custom network creation function
ERROR: …/nvdsinfer/nvdsinfer_model_builder.cpp:789 Failed to get cuda engine from custom library API
0:00:02.631375039 4115 0x55d7c4af1a30 ERROR nvinfer gstnvinfer.cpp:640:gst_nvinfer_logger: NvDsInferContext[UID 4]: Error in NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1934> [UID = 4]: build engine file failed
ERROR: [TRT]: 2: [logging.cpp::decRefCount::61] Error Code 2: Internal Error (Assertion mRefCount > 0 failed. )
corrupted size vs. prev_size while consolidating

My nvinfer config looks like so
ppev1_3_nvinfer_config.txt (1.3 KB)

The spec file of my TAO model looks like so
yolo_v4_retrain_mobilenet_v1_kitti_seq_ppe_set_3.txt (2.4 KB)

Why is TAO model not working in deepstream?

I also tried to use a separate tao-deploy container to generate the trt engine and pass that to nvinfer, but the engine generated in that container doesn’t work with serialization version mismatch error.

ERROR: [TRT]: 1: [stdArchiveReader.cpp::StdArchiveReader::40] Error Code 1: Serialization (Serialization assertion stdVersionRead == serializationVersion failed.Version tag does not match. Note: Current Version: 213, Serialized Engine Version: 232)
ERROR: [TRT]: 4: [runtime.cpp::deserializeCudaEngine::50] Error Code 4: Internal Error (Engine deserialization failed.)

3

Sorry for delay! Will check and get back to you ASAP

6

Moving into TAO forum.

7

Is the key correct? Can you share your command?

8

This is a common log when the trt version under inference environment is not the same as the trt version under the environment where to generate the trt engine.

9

The original issue was TAO 4.0 toolkit’s etlt models were not being read by nvinfer plugins. So I had to generate engines using tao deploy containers. I checked the tensorrt version in tao deploy container and it was slightly higher from the deepstream version.

I had to update nvinfer to version TENSORRT_VERSION=8.5.2-1+cuda11.8 . It then started working. The key was correct.

export TENSORRT_VERSION=8.5.2-1+cuda11.8
export CUDNN_VERSION=8.7.0.84-1+cuda11.8

apt-get install -y /tmp/nv-tensorrt-repo-ubuntu2004-cuda11.4-trt8.2.5.1-ga-20220505_1-1_amd64.deb
&& apt-key add /var/nv-tensorrt-repo-ubuntu2004-cuda11.4-trt8.2.5.1-ga-20220505/82307095.pub
&& apt-get update && apt-get install -y --no-install-recommends libcudnn8=${CUDNN_VERSION}
libcudnn8-dev=${CUDNN_VERSION}
libnvinfer8=${TENSORRT_VERSION}
libnvinfer-dev=${TENSORRT_VERSION}
libnvparsers8=${TENSORRT_VERSION}
libnvparsers-dev=${TENSORRT_VERSION}
libnvonnxparsers8=${TENSORRT_VERSION}
libnvonnxparsers-dev=${TENSORRT_VERSION}
libnvinfer-plugin8=${TENSORRT_VERSION}
libnvinfer-plugin-dev=${TENSORRT_VERSION} \

It works now.

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.