Is it possible to convert OpenSeq2Seq Jasper pb file into trt engine file?

I have freeze pb file with OpenSeq2Seq Jasper graph (2 inputs, 1 output).
When I’ve tried to convert this graph to uff, I have got a lot of warnings, such as:

Warning: No conversion function registered for layer: FusedBatchNormV2 yet.
Converting ForwardPass/w2l_encoder/conv33/bn/FusedBatchNormV2 as custom op: FusedBatchNormV2

Does anyone know, or have experience preprocessing graph to convert into engine using TensorRT 7.0?

Hi,

Try converting your model to ONNX instead using tf2onnx and then convert to TensorRT using ONNX parser. Any layer that are not supported needs to be replaced by custom plugin.
https://github.com/onnx/tensorflow-onnx
https://github.com/onnx/onnx-tensorrt/blob/master/operators.md

Another alternative approach, you can convert the model to TRT using TF-TRT and serialize it to a .plan file. Then deserialize the .plan file using the C++ API (TensorRT’s C++ API or through the TensorRT Inference Server).
See:
https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#usage-example

Thanks

Thank you for your answer, but there are troubles with the ways you have suggested.

  1. Converting to onnx fails because of troubles with shapes:
ERROR - tf2onnx.tfonnx: Failed to convert node ForwardPass/w2l_encoder/conv121/SpaceToBatchND
OP=SpaceToDepth
Name=ForwardPass/w2l_encoder/conv121/SpaceToBatchND
Inputs:
        ForwardPass/w2l_encoder/Relu_50:0=Relu, [10, -1, 768], 10
        ForwardPass/w2l_encoder/conv121/SpaceToBatchND/block_shape:0=Const, [1], 6
        ForwardPass/w2l_encoder/conv121/concat:0=Identity, [1, 2], 6
Outpus:
        ForwardPass/w2l_encoder/conv121/SpaceToBatchND:0=[20, -1, 768], 10
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx/tfonnx.py", line 354, in tensorflow_onnx_mapping
    func(g, node, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx/onnx_opset/tensor.py", line 1283, in version_1
    utils.make_sure(len(ctx.get_shape(input_tensor.output[0])) == 4, "only supports 4D for now")
  File "/usr/local/lib/python3.6/dist-packages/tf2onnx/utils.py", line 291, in make_sure
    raise ValueError("make_sure failure: " + error_msg % args)
ValueError: make_sure failure: only supports 4D for now
  1. Converting using tf-trt doesn’t work (nodes didn’t converted into TRTEngineOp).
  1. It is due to the limitation of tf2onnx API. Please refer below link:
    https://github.com/onnx/tensorflow-onnx/blob/master/tf2onnx/onnx_opset/tensor.py#L1281

  2. Try setting minimum_segment_size value to 2. Default value is 3 in TF-TRT https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#tf-trt-api

Also, please refer to supported ops in TRT:
https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/tensorrt-700/tensorrt-support-matrix/index.html#supported-ops

Thanks

Thanks for your answer. Minimum segment size didn’t help. Trying to replace current layers to supported layers. I will post results here.