ONNX to TRT Serialization Error

AI & Data Science Deep Learning (Training & Inference) TensorRT
,
1

Loading TensorRT Engine from File Failed

I have a neural network that originally wrote in Keras
I converted it to Onnx, and then to TensorRT

######################################################

FIRST - Keras to Onnx conversion with → GitHub - onnx/keras-onnx: Convert tf.keras/Keras models to ONNX
Installed with pip, the conversion succeeded.

SECOND - Onnx to trt conversion with → GitHub - onnx/onnx-tensorrt: ONNX-TensorRT: TensorRT backend for ONNX
Cloned the repo
To create the engine:

onnx2trt my_model.onnx -o my_engine.trt

To create the txt file: (link to it attached at the end)

onnx2trt my_model.onnx -t my_engine.trt

When trying to load the trt engine I get:

[E] [TRT] (31) - Serialization Error in verifyHeader: 0 (Magic tag does not match)
[E] [TRT] INVALID_STATE: std::exception[E] [TRT] INVALID_CONFIG: Deserialize the cuda engine failed.

The code to load the trt engine is attached below (link to drive)

######################################################
I updated the keras to onnx converter - installed it from source

pip install -U git+https://github.com/microsoft/onnxconverter-common
pip install -U git+https://github.com/onnx/keras-onnx

FIRST: Convert the Keras model to onnx successfully.

SECOND: Convert Onnx to TRT (engine txt file attached in the link below)
The conversion failed with output:

Input filename: /home/shira/Documents/Projects/EmotionsTrtInfer/models/emotion_model_new.onnx
ONNX IR version: 0.0.6
Opset version: 11
Producer name: keras2onnx
Producer version: 1.7.0
Domain: onnxmltools
Model version: 0
Doc string:

WARNING: ONNX model has a newer ir_version (0.0.6) than this parser was built against (0.0.3).
Writing ONNX model (without weights) as text to /home/shira/Documents/Projects/EmotionsTrtInfer/models/emotion_model_new.txt
Parsing model
While parsing node number 1 [Conv → “convolution_output22”]:
ERROR: /home/shira/Documents/Git/onnx-tensorrt/ModelImporter.cpp:537 In function importModel:
[5] Assertion failed: tensors.count(input_name)

######################################################

Environment

TensorRT Version 7.0.0:
GPU Type GeForce RTX 2080 Ti:
Nvidia Driver Version 440.33.01:
CUDA Version 10.2:
CUDNN Version 7:
Linux 18.04:

Link contain:

  1. Load_trt_engine.txt → C++ code to load the engine
  2. emotion_model_old.txt → engine from first onnx to trt conversion
  3. emotion_model_new.txt → engine from second onnx to trt conversion
    emotions - Google Drive
2

Can you try few things:

  1. Check ONNX model using checker function and see if it passes?
    import onnx

    model = onnx.load(“model.onnx”)

    onnx.checker.check_model(model)

  2. Please try trtexec commands to generate TRT model
    https://github.com/NVIDIA/TensorRT/blob/master/samples/opensource/trtexec/README.md
    If issue persist, could you please share the ONNX model so we can better help.

Thanks

3

Thank you for answer.
I checked the ONNX model, also did inference with it and it is ok.

Used trtexec and it worked! thanks a lot!

4

I got same problem.
My scenario is that.

A: PyTorch Model → B: Onnx Model → C: Tensorrt .TRT file

Python Tensorrt inference with B is done successfully.
onnx checking is passed.

but with C is failed with exactly same error
and TRTEXEC also failed with same error

It’s not because of version-matches. because All these evens were made in the single environment.

I uses TensorRT 7.0.0