TAO Toolkit 4.0, Yolo v4 not working with Deepstream nvinfer - parseModel: Failed to parse ONNX model

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 61
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only) 510
• Issue Type( questions, new requirements, bugs) bug
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing) Export yolo_v4 etlt model using tao toolkit and try to run with deepstream nvinfer
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

I have trained a yolo_v4 model using TAO toolkit 4.0. The TAO version was installed with pip install nvidia-tao and is the latest version. I exported the model in etlt format and tried to use it in a deepstream application. The NvInfer plugin is supposed to generate an engine file from etlt and run but it throws a ONNX parse error.

WARNING: [TRT]: onnx2trt_utils.cpp:369: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
WARNING: [TRT]: onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped

WARNING: [TRT]: onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
WARNING: [TRT]: onnx2trt_utils.cpp:395: One or more weights outside the range of INT32 was clamped
parseModel: Failed to parse ONNX model
ERROR: tlt/tlt_decode.cpp:389 Failed to build network, error in model parsing.
ERROR: …/nvdsinfer/nvdsinfer_model_builder.cpp:723 Failed to create network using custom network creation function
ERROR: …/nvdsinfer/nvdsinfer_model_builder.cpp:789 Failed to get cuda engine from custom library API
0:00:02.631375039 4115 0x55d7c4af1a30 ERROR nvinfer gstnvinfer.cpp:640:gst_nvinfer_logger: NvDsInferContext[UID 4]: Error in NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1934> [UID = 4]: build engine file failed
ERROR: [TRT]: 2: [logging.cpp::decRefCount::61] Error Code 2: Internal Error (Assertion mRefCount > 0 failed. )
corrupted size vs. prev_size while consolidating

My nvinfer config looks like so
ppev1_3_nvinfer_config.txt (1.3 KB)

The spec file of my TAO model looks like so
yolo_v4_retrain_mobilenet_v1_kitti_seq_ppe_set_3.txt (2.4 KB)

Why is TAO model not working in deepstream?

I also tried to use a separate tao-deploy container to generate the trt engine and pass that to nvinfer, but the engine generated in that container doesn’t work with serialization version mismatch error.

ERROR: [TRT]: 1: [stdArchiveReader.cpp::StdArchiveReader::40] Error Code 1: Serialization (Serialization assertion stdVersionRead == serializationVersion failed.Version tag does not match. Note: Current Version: 213, Serialized Engine Version: 232)
ERROR: [TRT]: 4: [runtime.cpp::deserializeCudaEngine::50] Error Code 4: Internal Error (Engine deserialization failed.)

Sorry for delay! Will check and get back to you ASAP

Moving into TAO forum.

Is the key correct? Can you share your command?

This is a common log when the trt version under inference environment is not the same as the trt version under the environment where to generate the trt engine.

The original issue was TAO 4.0 toolkit’s etlt models were not being read by nvinfer plugins. So I had to generate engines using tao deploy containers. I checked the tensorrt version in tao deploy container and it was slightly higher from the deepstream version.

I had to update nvinfer to version TENSORRT_VERSION=8.5.2-1+cuda11.8 . It then started working. The key was correct.

export TENSORRT_VERSION=8.5.2-1+cuda11.8

apt-get install -y /tmp/nv-tensorrt-repo-ubuntu2004-cuda11.4-trt8.2.5.1-ga-20220505_1-1_amd64.deb
&& apt-key add /var/nv-tensorrt-repo-ubuntu2004-cuda11.4-trt8.2.5.1-ga-20220505/82307095.pub
&& apt-get update && apt-get install -y --no-install-recommends libcudnn8=${CUDNN_VERSION}
libnvinfer-plugin-dev=${TENSORRT_VERSION} \

It works now.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.