Keras CRNN model conversion to tensorrt engine error

AI & Data Science Deep Learning (Training & Inference) TensorRT
, ,
#1

Description

tensorflow to onnx to trt engine conversion failure for crnn model with bidirectional layer with ReverseSequence Op on Jetson Xavier with trt version 8.0.1.6

Environment

TensorRT Version: ‘8.0.1.6’
GPU Type: Jetson Xavier
Nvidia Driver Version:
CUDA Version: 4.6-b197
CUDNN Version: 4.6-b197
Operating System + Version: Ubuntu 18.04
Python Version (if applicable): Python 3.6.9
TensorFlow Version (if applicable): 1.15.2
PyTorch Version (if applicable): -
Baremetal or Container (if container which image + tag): -

Relevant Files

Package: nvidia-jetpack
Version: 4.6-b197
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-cuda (= 4.6-b197), nvidia-opencv (= 4.6-b197), nvidia-cudnn8 (= 4.6-b197), nvidia-tensorrt (= 4.6-b197), nvidia-visionworks (= 4.6-b197), nvidia-container (= 4.6-b197), nvidia-vpi (= 4.6-b197), nvidia-l4t-jetson-multimedia-api (>> 32.6-0), nvidia-l4t-jetson-multimedia-api (<< 32.7-0)
Homepage: Autonomous Machines | NVIDIA Developer
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_4.6-b197_arm64.deb

keras/tensorflow crnn model with bidirectional lstm layer

Steps To Reproduce

tensorrt engine conversion from onnx fails at ReverseSequence Op
[TensorRT] VERBOSE: lstm_10_back_reverse_seq_rev_seq [ReverseSequence] inputs: [lstm_10_back_transpose → (-1, 50, 128)[FLOAT]], [lstm_10_back_reverse_seq_expand0 → (-1)[INT32]],
[TensorRT] ERROR: 4: [network.cpp::validate::2411] Error Code 4: Internal Error (Network must have at least one output)
engine conversion failed
Even though Trt ‘8.0.1.6’ version supports ReverseSequence Op

Onnx model node names:
permute
permute/transpose:0_cropping
Transpose28
conv_1
Relu7
conv_2
Relu6
conv_3
Relu5
bn_3
bn_3/FusedBatchNormV3:0_pooling
conv_4
Relu4
conv_5
Relu3
bn_5
bn_5/FusedBatchNormV3:0_pooling
conv_6
Relu2
conv_7
Relu1
bn_7
Transpose9
reshape
fc_9
Add2
Relu
Transpose4
Transpose6
lstm_10
lstm_10_back
Squeeze2
Squeeze3
Transpose5
Transpose7
Shape1
Gather2
Gather3
lstm_10_back_reverse_seq_expand
lstm_10_back_reverse_seq_rev_seq
Add1
Transpose
Transpose2
lstm_11
lstm_11_back
Squeeze
Squeeze1
Transpose1
Transpose3
Shape
Gather
Gather1
lstm_11_back_reverse_seq_expand
lstm_11_back_reverse_seq_rev_seq
concatenate
fc_12
Add
Softmax
fc_12/truediv:0_cropping

#2

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.

In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

#5

Thanks Nves for quick reply, I did run the 2 options you told me to perform.

  1. onnx.checker.check_model(model). - did not return any o/p
  2. ./trtexec failed with this error
    [04/06/2022-17:41:23] [E] [TRT] ModelImporter.cpp:720: While parsing node number 39 [ReverseSequence → “lstm_10_back_reverse_seq_rev_seq0”]:
    [04/06/2022-17:41:23] [E] [TRT] ModelImporter.cpp:721: — Begin node —
    [04/06/2022-17:41:23] [E] [TRT] ModelImporter.cpp:722: input: “lstm_10_back_transpose”
    input: “lstm_10_back_reverse_seq_expand0”
    output: “lstm_10_back_reverse_seq_rev_seq0”
    name: “lstm_10_back_reverse_seq_rev_seq”
    op_type: “ReverseSequence”
    attribute {
    name: “batch_axis”
    i: 0
    type: INT
    }
    attribute {
    name: “time_axis”
    i: 1
    type: INT
    }
    domain: “”

[04/06/2022-17:41:23] [E] [TRT] ModelImporter.cpp:723: — End node —
[04/06/2022-17:41:23] [E] [TRT] ModelImporter.cpp:726: ERROR: builtin_op_importers.cpp:3496 In function importReverseSequence:
[8] Assertion failed: size != -1 && “This version of TensorRT does not support dynamic ReverseSequence lengths!”
[04/06/2022-17:41:23] [E] Failed to parse onnx file
[04/06/2022-17:41:23] [I] Finish parsing network model
[04/06/2022-17:41:23] [E] Parsing model failed
[04/06/2022-17:41:23] [E] Engine creation failed
[04/06/2022-17:41:23] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8001] # ./trtexec --onnx=/home/nxdev8/vision-research-pranav/crnn_pretrained/crnn_pretrained_nx.onnx --verbose=True

Attaching onnx file
crnn_pretrained_nx.onnx (30.0 MB)

#6

Hi,

Currently ReverseSequence Op, which as we state in the support document, does not yet support dynamic length inputs (i.e., lengths of the sequences in a batch, sequence_lens, cannot be dynamic). Maybe we can try for a static batch.

image
image2642×111 12.7 KB

Thank you.