Uff to trt8

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Hello guys, I am trying to convert a ssd_inception model that we trained on our own dataset consisting of 1 class(excluding background) to trt. When trying to convert it to trt8 from uff we receive the following error messages depending on the uff config used for the conversion of .pb to .uff.

Code used for conversation is taken from TensorRT/samples/python/uff_ssd at main · NVIDIA/TensorRT · GitHub

Using old UFF config that was valid for trt6

TensorRT inference engine settings:
  * Inference precision - DataType.HALF
  * Max batch size - 1

[TensorRT] INTERNAL ERROR: Assertion failed: mConcatAxisID == 1 || mConcatAxisID == 2 || mConcatAxisID == 3
flattenConcat.cpp:40```

Or using the newest uff config located in TensorRT/model.py at main · NVIDIA/TensorRT · GitHub


TensorRT inference engine settings:
  * Inference precision - DataType.HALF
  * Max batch size - 1

[TensorRT] ERROR: 3: [layers.h::setAxis::562] Error Code 3: Internal Error (Parameter check failed at: /_src/build/cuda-10.2/8.2/aarch64/release/optimizer/api/layers.h::setAxis::562, condition: axis >= 0 && axis < Dims::MAX_DIMS
)
[TensorRT] ERROR: 4: [layers.cpp::estimateOutputDims::1856] Error Code 4: Internal Error (Concatenate/concat: all concat input tensors must have the same dimensions except on the concatenation axis (0), but dimensions mismatched at index 1. Input 0 shape: [2,7668,1], Input 1 shape: [2,4332,1])
[TensorRT] ERROR: 4: [layers.cpp::estimateOutputDims::1856] Error Code 4: Internal Error (Concatenate/concat: all concat input tensors must have the same dimensions except on the concatenation axis (0), but dimensions mismatched at index 1. Input 0 shape: [2,7668,1], Input 1 shape: [2,4332,1])
[TensorRT] ERROR: 4: [layers.cpp::estimateOutputDims::1856] Error Code 4: Internal Error (Concatenate/concat: all concat input tensors must have the same dimensions except on the concatenation axis (0), but dimensions mismatched at index 1. Input 0 shape: [2,7668,1], Input 1 shape: [2,4332,1])
[TensorRT] ERROR: 4: [layers.cpp::estimateOutputDims::1856] Error Code 4: Internal Error (Concatenate/concat: all concat input tensors must have the same dimensions except on the concatenation axis (0), but dimensions mismatched at index 1. Input 0 shape: [2,7668,1], Input 1 shape: [2,4332,1])
[TensorRT] ERROR: 4: [layers.cpp::estimateOutputDims::1856] Error Code 4: Internal Error (Concatenate/concat: all concat input tensors must have the same dimensions except on the concatenation axis (0), but dimensions mismatched at index 1. Input 0 shape: [2,7668,1], Input 1 shape: [2,4332,1])
[TensorRT] ERROR: UffParser: Parser error: BoxPredictor_0/ClassPredictor/BiasAdd: The input to the Scale Layer is required to have a minimum of 3 dimensions.
Building TensorRT engine. This may take few minutes.
[TensorRT] ERROR: 4: [network.cpp::validate::2411] Error Code 4: Internal Error (Network must have at least one output)
[TensorRT] ERROR: 2: [builder.cpp::buildSerializedNetwork::417] Error Code 2: Internal Error (Assertion enginePtr != nullptr failed.)
Traceback (most recent call last):
  File "detect_objects.py", line 193, in <module>
    main()
  File "detect_objects.py", line 167, in main
    batch_size=args.max_batch_size)
  File "/home/jetson/convert_uff/uff_ssd/utils/inference_trt.py", line 82, in __init__
    batch_size=batch_size)
  File "/home/jetson/convert_uff/uff_ssd/utils/engine.py", line 98, in build_engine
    return runtime.deserialize_cuda_engine(plan)
TypeError: deserialize_cuda_engine(): incompatible function arguments. The following argument types are supported:
    1. (self: tensorrt.tensorrt.Runtime, serialized_engine: buffer) -> tensorrt.tensorrt.ICudaEngine

Invoked with: <tensorrt.tensorrt.Runtime object at 0x7f7cf7dab0>, None

If you need the .pb and .uff files to debug this further please let me know and I will provide you them privately.
Help with getting the model converted to a trt8 compatible engine would be much appreciated!

2

Hi , UFF and Caffe Parser have been deprecated from TensorRT 7 onwards, hence request you to try ONNX parser.
Please check the below link for the same.

Thanks!

4

Good to know they have been deprecated. So then the next question becomes how to convert the .pb file of ssd_inception to onnx?

5

Hi,

Hope the following may help you.

Thank you.

6

I converted the frozen graph to onnx but it fails to convert to a trt engine.

In the attachement are 2 onnx models, one with fp32 inputs(standard result from using tf2onnx) and one where I changed the input data type to uint8. Both fail to convert. Any tips to convert them to trt would be much appreciated.

updated_uint8_model.onnx (50.5 MB)
model.onnx (50.5 MB)

7

@NVES Could you give any pointers on how to successfully convert the onnx into a trt engine? Thanks again for the help!

8

Hi,

TensorRT currently doesn’t support UINT8 datatype.
TensorRT supports the following ONNX data types: DOUBLE, FLOAT32, FLOAT16, INT8, and BOOL

You may need to cast data type.

Thank you.