Resnet50.onnx failed to parse using tensorrt trtexec to be used in Deepstream

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

Resnet50.onnx failed to parse using tensorrt trtexec to be used in Deepstream

./trtexec --onnx=/home/ubuntu/Desktop/tensorflow-onnx/resnet_v1_50_inference.onnx
&&&& RUNNING TensorRT.trtexec # ./trtexec --onnx=/home/ubuntu/Desktop/tensorflow-onnx/resnet_v1_50_inference.onnx
[07/13/2020-07:12:21] [I] === Model Options ===
[07/13/2020-07:12:21] [I] Format: ONNX
[07/13/2020-07:12:21] [I] Model: /home/ubuntu/Desktop/tensorflow-onnx/resnet_v1_50_inference.onnx
[07/13/2020-07:12:21] [I] Output:
[07/13/2020-07:12:21] [I] === Build Options ===
[07/13/2020-07:12:21] [I] Max batch: 1
[07/13/2020-07:12:21] [I] Workspace: 16 MB
[07/13/2020-07:12:21] [I] minTiming: 1
[07/13/2020-07:12:21] [I] avgTiming: 8
[07/13/2020-07:12:21] [I] Precision: FP32
[07/13/2020-07:12:21] [I] Calibration:
[07/13/2020-07:12:21] [I] Safe mode: Disabled
[07/13/2020-07:12:21] [I] Save engine:
[07/13/2020-07:12:21] [I] Load engine:
[07/13/2020-07:12:21] [I] Inputs format: fp32:CHW
[07/13/2020-07:12:21] [I] Outputs format: fp32:CHW
[07/13/2020-07:12:21] [I] Input build shapes: model
[07/13/2020-07:12:21] [I] === System Options ===
[07/13/2020-07:12:21] [I] Device: 0
[07/13/2020-07:12:21] [I] DLACore:
[07/13/2020-07:12:21] [I] Plugins:
[07/13/2020-07:12:21] [I] === Inference Options ===
[07/13/2020-07:12:21] [I] Batch: 1
[07/13/2020-07:12:21] [I] Iterations: 10
[07/13/2020-07:12:21] [I] Duration: 3s (+ 200ms warm up)
[07/13/2020-07:12:21] [I] Sleep time: 0ms
[07/13/2020-07:12:21] [I] Streams: 1
[07/13/2020-07:12:21] [I] ExposeDMA: Disabled
[07/13/2020-07:12:21] [I] Spin-wait: Disabled
[07/13/2020-07:12:21] [I] Multithreading: Disabled
[07/13/2020-07:12:21] [I] CUDA Graph: Disabled
[07/13/2020-07:12:21] [I] Skip inference: Disabled
[07/13/2020-07:12:21] [I] Input inference shapes: model
[07/13/2020-07:12:21] [I] Inputs:
[07/13/2020-07:12:21] [I] === Reporting Options ===
[07/13/2020-07:12:21] [I] Verbose: Disabled
[07/13/2020-07:12:21] [I] Averages: 10 inferences
[07/13/2020-07:12:21] [I] Percentile: 99
[07/13/2020-07:12:21] [I] Dump output: Disabled
[07/13/2020-07:12:21] [I] Profile: Disabled
[07/13/2020-07:12:21] [I] Export timing to JSON file:
[07/13/2020-07:12:21] [I] Export output to JSON file:
[07/13/2020-07:12:21] [I] Export profile to JSON file:
[07/13/2020-07:12:21] [I]

Input filename: /home/ubuntu/Desktop/tensorflow-onnx/resnet_v1_50_inference.onnx
ONNX IR version: 0.0.4
Opset version: 8
Producer name: tf2onnx
Producer version: 1.7.0
Domain:
Model version: 0
Doc string:

[07/13/2020-07:12:22] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[07/13/2020-07:12:22] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[07/13/2020-07:12:22] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[07/13/2020-07:12:22] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[07/13/2020-07:12:22] [E] [TRT] Network has dynamic or shape inputs, but no optimization profile has been defined.
[07/13/2020-07:12:22] [E] [TRT] Network validation failed.
[07/13/2020-07:12:22] [E] Engine creation failed
[07/13/2020-07:12:22] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec # ./trtexec --onnx=/home/ubuntu/Desktop/tensorflow-onnx/resnet_v1_50_inference.onnx

Environment

TensorRT Version:7.0
GPU Type: T4
Nvidia Driver Version: 440.33.01
CUDA Version: 10.2
CUDNN Version:
Operating System + Version: 18.04
Python Version (if applicable): 3.6.9
TensorFlow Version (if applicable): 1.14
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

2

Hi @GalibaSashi,
Can you please share your onnx model, so that we can try reproducing the issue.
Thanks!

3

Is there any other additional arguments I should be giving?

4

Hi @GalibaSashi,
Please try the command,

trtexec --onnx=yourModel --verbose --explicitBatch --shapes=inputShapes

Thanks!

6

TRT >= 7 requires EXPLICIT_BATCH for ONNX, for fixed-shape model, the batch size is fixed.

Please refer to the below link.

Also, here you need to provide the shape of your model, something like input: [ batch_size,1,224,224 ]

Thanks!

7

input: [ batch_size,1,224,224 ]
I had not specified in the above post so how did it work then.

2)Also to save I should use --save-engine right

8

Because you have not specified, hence its running on the default value.

Yes, you can use trtexec with the --saveEngine argument to save the Engine, and can use it later by passing --loadEngine argument.
Thanks!

9

Hi @AakankshaS,
1)So even if I donot give > input: [ batch_size,1,224,224 ] it will give the same trt engine file and same performance right?
2)Also where can I find the input: [ *batch_size,1,224,224 details as I only have a .pb file and a .onnx file
Thanks in advance.

10

Hi @GalibaSashi,

No, there will be performance impact as the engine will be optimized for the default shape.
Hence it is recommended to specify one or more optimization profiles at build time that specify the permitted range of dimensions for inputs with runtime dimensions, and the dimensions for which the auto-tuner should optimize.
Please refer to the below link to understand this better.

You can use Netron to view your onnx model, and there you can find the input shapes.
Thanks!