InvalidArgumentError when converting tensorflow model into tensorRT model

[elhe]

I am trying to convert my tensorflow model into an tensorRT model in order to use it with nvinfer of deepstream.
Unfortunately I always get the following error in the building step:

2021-08-04 13:12:25.059598: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at transpose_op.cc:157 : Invalid argument: transpose expects a vector of size 4. But input(1) is a vector of size 5
2021-08-04 13:12:25.060068: W tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:588] Running native segment forStatefulPartitionedCall/sequential/conv_lst_m2d/TRTEngineOp_0_0 due to failure in verifying input shapes: Input shapes do not match input partial shapes stored in graph, for StatefulPartitionedCall/sequential/conv_lst_m2d/TRTEngineOp_0_0: [[16,128,128,1]] != [[?,16,1,128,128]]
2021-08-04 13:12:25.060151: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at trt_engine_op.cc:401 : Cancelled: Function was cancelled before it was started
Traceback (most recent call last):
  File "savedmodel_to_trt.py", line 25, in <module>
    converter.build(input_fn=my_input_fn)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/compiler/tensorrt/trt_convert.py", line 1174, in build
    func(*map(ops.convert_to_tensor, inp))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1655, in __call__
    return self._call_impl(args, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/wrap_function.py", line 247, in _call_impl
    args, kwargs, cancellation_manager)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1673, in _call_impl
    return self._call_with_flat_signature(args, kwargs, cancellation_manager)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1722, in _call_with_flat_signature
    return self._call_flat(args, self.captured_inputs, cancellation_manager)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1924, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 550, in call
    ctx=ctx)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError:  transpose expects a vector of size 4. But input(1) is a vector of size 5
         [[node StatefulPartitionedCall/sequential/conv_lst_m2d/transpose_2 (defined at savedmodel_to_trt.py:18) ]] [Op:__inference_pruned_38898]

Function call stack:
pruned

And this is the code I am using to convert it:

import tensorflow as tf
from tensorflow.python.compiler.tensorrt import trt_convert as trt
import numpy as np

input_saved_model_dir="savedmodel/"
output_saved_model_dir="trtmodel/"

conversion_params = trt.DEFAULT_TRT_CONVERSION_PARAMS
conversion_params = conversion_params._replace(
    max_workspace_size_bytes=(1<<32))
conversion_params = conversion_params._replace(precision_mode="FP16")
conversion_params = conversion_params._replace(
    maximum_cached_engines=100)

converter = trt.TrtGraphConverterV2(
    input_saved_model_dir=input_saved_model_dir,
    conversion_params=conversion_params)
converter.convert()
def my_input_fn():

  inp1 = np.random.normal(size=(16, 128, 128,1)).astype(np.float32)
  yield [inp1]


converter.build(input_fn=my_input_fn)
converter.save(output_saved_model_dir)

Any tips on what I am doing wrong?

[AastaLLL]

Hi,

Please note that you are using TF-TRT, the integrated version of TensorRT.
If you want to deploy it with Deepstream (since you say nvinfer here).
You will need a pure TensorRT engine rather than an integrated one.

In general, you can convert your model into an ONNX format.
And parse it into Deepstream to trigger the ONNX parser directly.

Thanks.