Trtexec onnx inception_v3

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

Every example I’ve found shows using tensorflow 1.x.

I have trained an inception_v3 model (with my own classes) using tensorflow 2.3.0. I have tried keras2onnx, but get errors when try trtexe to save the engine.

Environment

TensorRT Version: 6
GPU Type: Quadro P3200
Nvidia Driver Version: 460.32.03
CUDA Version: 10.1
TensorFlow Version (if applicable): 2.3.0
Baremetal or Container (if container which image + tag): nvcr.io/nvidia/tensorrt:19.10-py3

Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)

Steps To Reproduce

Install in container
pip3 install tensorflow==2.3.0
pip3 install git+https://github.com/microsoft/onnxconverter-common
pip 3install git+https://github.com/onnx/keras-onnx

create onnx
import tensorflow as tf
import onnx
import keras2onnx
model = tf.keras.models.load_model(‘mymodel.h5’)
onnx_model = keras2onnx.convert_keras(model, model.name, target_opset=8)
file = open(“mymodel.onnx”, “wb”)
file.write(onnx_model.SerializeToString())
file.close()

onnx → engine
trtexec --onnx=./mymodel.onnx --saveEngine=my.engine

Input filename: ./mymodel.onnx
ONNX IR version: 0.0.3
Opset version: 8
Producer name: keras2onnx
Producer version: 1.8.0
Domain: onnxmltools
Model version: 0
Doc string:

[01/20/2021-20:48:50] [E] [TRT] (Unnamed Layer* 39) [Concatenation]: all concat input tensors must have the same dimensions except on the concatenation axis (0), but dimensions mismatched at input 3 at index 1. Input 0 shape: [64,35,35], Input 3 shape: [32,33,33]
While parsing node number 40 [Conv]:
ERROR: builtin_op_importers.cpp:788 In function importConv:
[8] Assertion failed: (nbSpatialDims == 2 && kernel_weights.shape.nbDims == 4) || (nbSpatialDims == 3 && kernel_weights.shape.nbDims == 5)
[01/20/2021-20:48:50] [E] Failed to parse onnx file
[01/20/2021-20:48:50] [E] Parsing model failed
[01/20/2021-20:48:50] [E] Engine could not be created
&&&& FAILED TensorRT.trtexec # trtexec --onnx=./mymodel.onnx --saveEngine=my.engine

2

Hi, Request you to share the ONNX model and the script so that we can assist you better.

Alongside you can try validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).

Alternatively, you can try running your model with trtexec command.
https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec

Thanks!

4

The instructions for attaching files are outdated. How do I attach my model?

5

Hi @mhunsake,
Could you please try using latest trt release. And share complete verbose logs if you still face issue.
Also please validate your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).

Thank you.

6

I am able to use TRT7, but now I have another issue. when I try to use the serialized engine from the TensorRT CPP, I the input shape is not right.

I use this to read the engine

    std::ifstream ifs(model_file, std::ios::binary | std::ios::ate);
    std::ifstream::pos_type len = ifs.tellg();

    std::vector<char> blob(len);

    ifs.seekg(0, std::ios::beg);
    ifs.read(&blob[0], len);

    LOG(INFO) << "... Importing TensorRT engine";
    IRuntime* runtime = createInferRuntime(gLogger);
    engine_ =
        runtime->deserializeCudaEngine(&blob[0], blob.size(), nullptr);
    runtime->destroy();

But, when I try to allocate memory for input, there is issue with dimensions

  ICudaEngine* engine = engine_->Get();

  context_ = engine->createExecutionContext();
  CHECK(context_) << "Failed to create execution context.";

  int input_index = engine->getBindingIndex(input_name.c_str());
  input_dim_ = static_cast<DimsCHW&&>(engine->getBindingDimensions(input_index));
 OG(INFO) << "input_dim " << input_dim_.c() << "x"<< input_dim_.h()<< "x"<<input_dim_.w();
  input_cv_size_ = Size(input_dim_.w(), input_dim_.h());
  size_t input_size = input_dim_.c() * input_dim_.h() * input_dim_.w() * sizeof(float);
  cudaError_t st = cudaMalloc(&input_layer_, input_size);
  CHECK_EQ(st, cudaSuccess) << "Could not allocate input layer.";

Instead of 3,299,299 for input dimensions, the deserialized engine shows input_dim -1x3x299

7

Hi @mhunsake,

Could you please check input shape in the onnx file using Netron (model visualizer).
For the dims which have -1, are axis with dynamic shape. We need to use --minShape, --optShapes, --maxShapes when build the engine using trtexec.

For your reference,
https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec#tensorrt-command-line-wrapper-trtexec

Thank you.

8

@spolisetty

YES it has dynamic input with name inception_v3_input and dimensions [-1, 3, 299, 299]

If I use same approach as /opt/tensorrt/samples/python/yolov3_onnx/onnx_to_tensorrt.py get_engine to convert the onnx to .trt, it has correct shape. This code is explicitly setting the input shape after parsing onnx and before building engine.

(Pdb) engine.get_binding_shape('inception_v3_input')
(1, 3, 299, 299)

LOOKING TO DO THE SAME THING WITH TRTEXEC USING

trtexec --onnx=./best_model_loss.onnx --minShapes=inception_v3_input:1x3x299x299 --maxShapes=inception_v3_input:1x3x299x299 --optShapes=inception_v3_input:1x3x299x299 --shapes=inception_v3_input:1x3x299x299 --saveEngine=best_model_loss.trt --buildOnly

BUT, WHEN I DE-SERIALIZE THE ENGINE, IT STILL HAS SHAPE -1

(Pdb) engine.get_binding_shape('inception_v3_input')
(-1, 3, 299, 299)

Based on /opt/tensorrt/samples/sampleSSD/ /opt/tensorrt/samples/sampleOnnxMNIST/ I have created my own sample to try the c++ code to parse onnx and create engine. It fails with same type of error

[E] [TRT] Network has dynamic or shape inputs, but no optimization profile has been defined
9

For the moment, I have used tensorfow.onnx (GitHub - onnx/tensorflow-onnx: Convert TensorFlow, Keras, Tensorflow.js and Tflite models to ONNX) in stead of keras2onnx and am overriding the input size there.

python -m tf2onnx.convert --saved-model ./saved_model  --opset 13 --output my-model.onnx --inputs inception_v3_input:0[1,3,299,299]
trtexec --onnx=./my-model.onnx --saveEngine=my-model.engine --buildOnly

I AM STILL INTERESTED IN FINDING OUT HOW I CAN DO tf.keras → onnx → tensorrt IN C++ WITH ONNX MODEL THAT HAS DYNAMIC INPUT SHAPE

10

Hi @mhunsake,

Could you please provide output of following command.
trtexec --onnx=./best_model_loss.onnx --minShapes=inception_v3_input:1x3x299x299 --maxShapes=inception_v3_input:1x3x299x299 --optShapes=inception_v3_input:1x3x299x299 --shapes=inception_v3_input:1x3x299x299 --verbose

Thank you.