ONNX to TensorRT serialized engine with dynamic shapes

Hello,

I’m trying to realize a standard way to convert ONNX models to tensorRT serialized engine.

I saw several ways as follows,

1- Using trtexec (I could generate engine).
2- ONNX2trt Github repo (didn’t work for me).
3- Using Deepstream to create the engine directly.

At first when I flashed the JETPACK 4.4 and installed deepstream, I could create engines when running configuration files using deepstream but now it’s not working and I get the following error,

*** DeepStream: Launched RTSP Streaming at rtsp://localhost:8554/ds-test ***

Opening in BLOCKING MODE
Opening in BLOCKING MODE

Using winsys: x11
0:00:35.360371681 18885 0xae49350 INFO nvinfer gstnvinfer.cpp:602:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1577> [UID = 1]: deserialized trt engine from :/home/mqabbari/Workspace/Git/Deep-Stream-ONNX/tiny_yolov2.engine
INFO: [Implicit Engine Info]: layers num: 2
0 INPUT kFLOAT image 3x416x416
1 OUTPUT kFLOAT grid 125x13x13

0:00:35.708241629 18885 0xae49350 WARN nvinfer gstnvinfer.cpp:599:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:1518> [UID = 1]: Backend has maxBatchSize 1 whereas 8 has been requested
0:00:35.708286161 18885 0xae49350 WARN nvinfer gstnvinfer.cpp:599:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1689> [UID = 1]: deserialized backend context :/home/mqabbari/Workspace/Git/Deep-Stream-ONNX/tiny_yolov2.engine failed to match config params, trying rebuild
0:00:36.496712223 18885 0xae49350 INFO nvinfer gstnvinfer.cpp:602:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1591> [UID = 1]: Trying to create engine from model files

Input filename: /home/mqabbari/Workspace/Git/Deep-Stream-ONNX/tiny_yolov2.onnx
ONNX IR version: 0.0.5
Opset version: 8
Producer name: OnnxMLTools
Producer version: 1.5.2
Domain: onnxconverter-common
Model version: 0
Doc string: The Tiny YOLO network from the paper ‘YOLO9000: Better, Faster, Stronger’ (2016), arXiv:1612.08242

ERROR: [TRT]: Network has dynamic or shape inputs, but no optimization profile has been defined.
ERROR: [TRT]: Network validation failed.
ERROR: Build engine failed from config file
ERROR: failed to build trt engine.
0:00:38.201802273 18885 0xae49350 ERROR nvinfer gstnvinfer.cpp:596:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1611> [UID = 1]: build engine file failed
0:00:38.212847743 18885 0xae49350 ERROR nvinfer gstnvinfer.cpp:596:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1697> [UID = 1]: build backend context failed
0:00:38.212954411 18885 0xae49350 ERROR nvinfer gstnvinfer.cpp:596:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1024> [UID = 1]: generate backend failed, check config file settings
0:00:38.245487791 18885 0xae49350 WARN nvinfer gstnvinfer.cpp:781:gst_nvinfer_start:<primary_gie> error: Failed to create NvDsInferContext instance
0:00:38.245535708 18885 0xae49350 WARN nvinfer gstnvinfer.cpp:781:gst_nvinfer_start:<primary_gie> error: Config file path: /home/mqabbari/Workspace/Git/Deep-Stream-ONNX/config/config_infer_custom_yolo.txt, NvDsInfer Error: NVDSINFER_CONFIG_FAILED
** ERROR: main:651: Failed to set pipeline to PAUSED
Quitting
ERROR from primary_gie: Failed to create NvDsInferContext instance
Debug info: /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp(781): gst_nvinfer_start (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInfer:primary_gie:
Config file path: /home/mqabbari/Workspace/Git/Deep-Stream-ONNX/config/config_infer_custom_yolo.txt, NvDsInfer Error: NVDSINFER_CONFIG_FAILED
App run failed

I tried many things to resolve the issue of dynamic shapes as directed by tensorrt documentation but didn’t get it to work with deepstream.

I’m using,

Jetpack 4.4(Tensorrt 7.1.3)
Deepstream 5

Onnx Model:

Tiny_yolov2 from model zoo.
Input (-1,3,416,416) so is dynamic for batch size

Thanks in advance…

Hi,

May I know how do you install Deepstream SDK to the JetPack4.4 GA first?
Do you use the package released for JetPack4.4 DP?

Thanks.

Hi,

Thanks for your fast response.

I downloaded the tar file for deepstream5 from the getting started page “https://developer.nvidia.com/deepstream-getting-started” and installed directly from there.

Hi,

Sorry for the confusing.

Currently, our Deepstream package is built on JetPack4.4 DP.
The Deepstream 5.0 GA version which builit on JetPack4.4 GA will be released shortly (in days).
Please pay attention to our announcement and update your package into GA version.

Thanks.

1 Like

What is the difference between DP and GA versions?

Hi,

The package version are different.

For example, JetPack4.4 DP use TensorRT 7.1.0 while JetPack4.4 GA uses version 7.1.3.
To use the library built on different package will lead to some compatibility issue, especially on TensorRT.

Thanks.

But then will I be able to migrate to the new jetpack without lossing everything I installed to the current version?

Hi,

You will need to reflash and install the package for the new JetPack release.
Thanks.

Hello again,

So I managed to solve the issue with the dynamic shapes using optimization profile.

The issue last time when I used optimization profiles that I didn’t create the profile with the network input tensor name.

Now I could do it using following python commands,

profile = builder.create_optimization_profile()
profile.set_shape(network.get_input(0).name, (1, 3, 416, 416), (1, 3, 416, 416), (8, 3, 416, 416))
config.add_optimization_profile(profile)

However I face now another issue that when I do inference with the engine created using tensorrt python API, I could do inference using python inference API and I get correct model results.

On the other hand, when I use the same engine with the deepstream, I get different inference output and the model results is not correct at all.

I wonder if it could be the same issue you mentioned earlier about tensorrt different versions that causes deepstream to parse the engine differently!

What do you think?

I see that the Jetpack release with tensorrt 7.1.3 already arrived. Will I be able to upgrade to this version like mentioned in section 1.3.2 here,

https://docs.nvidia.com/jetson/jetpack/install-jetpack/index.html

or this option is only available starting the new Jetpack release?

If you are staying with Jetpack4.4 GA better just wait a bit for the corresponding deepstream 5 release coming soon

1 Like

Hi,

Was deepstream GA version released? Where to find old versions?

Hi mqabbari,

You should be able to find the archived version for Jetson platform from https://developer.nvidia.com/deepstream-getting-started

Hello @mqabbari,

Can you please share your python code in which you defined optimization profile ?

I want to know where exactly the below lines fitted in your code