Bad shape (TRT 7)

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

I created an ONNX sample file having output shape Nx100.
When I parsed the ONNX with trtexec I got (-1,-1) for output shape. Why not (-1, 100)?
Please help.

Environment

TensorRT Version: 7.0.0.11
GPU Type: Quadro P1000
Nvidia Driver Version: 441.22
CUDA Version: 10.2.89
CUDNN Version: 7.6.5.32
Operating System + Version: Windows 10 Enterprise 64bit
Python Version (if applicable): 3.6.7
TensorFlow Version (if applicable): 1.13.1
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Steps To Reproduce

model = tf.keras.models.Sequential([tf.keras.layers.InputLayer(name=‘input’,input_shape=(32,32,3)),
tf.keras.layers.Conv2D(16, (3,3), activation=‘relu’),
tf.keras.layers.MaxPool2D(),
tf.keras.layers.Conv2D(32, (3,3), activation=‘relu’),
tf.keras.layers.MaxPool2D(),
tf.keras.layers.Conv2D(64, (3,3), activation=‘relu’),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(1000, activation=‘relu’),
tf.keras.layers.Dense(100, activation=‘softmax’, name=‘output’)])
model.summary()

Layer (type) Output Shape Param #

conv2d (Conv2D) (None, 30, 30, 16) 448

max_pooling2d (MaxPooling2D) (None, 15, 15, 16) 0

conv2d_1 (Conv2D) (None, 13, 13, 32) 4640

max_pooling2d_1 (MaxPooling2 (None, 6, 6, 32) 0

conv2d_2 (Conv2D) (None, 4, 4, 64) 18496

flatten (Flatten) (None, 1024) 0

dense (Dense) (None, 1000) 1025000

output (Dense) (None, 100) 100100

=================================================================
Total params: 1,148,684
Trainable params: 1,148,684
Non-trainable params: 0

trtexec --verbose --explicitBatch --workspace=128 --onnx=cifar100.onnx

output:
[03/31/2020-17:14:05] [V] [TRT] ModelImporter.cpp:123: Searching for input: output/bias:0
[03/31/2020-17:14:05] [V] [TRT] ModelImporter.cpp:129: Add [Add] inputs: [transformed_tensor → (-1, 100)], [output/bias:0 → (100)],
[03/31/2020-17:14:05] [V] [TRT] ImporterContext.hpp:122: Registering layer: Add for ONNX node: Add
[03/31/2020-17:14:05] [V] [TRT] ImporterContext.hpp:97: Registering tensor: biased_tensor_name for ONNX tensor: biased_tensor_name
[03/31/2020-17:14:05] [V] [TRT] ModelImporter.cpp:180: Add [Add] outputs: [biased_tensor_name → (-1, 100)],
[03/31/2020-17:14:05] [V] [TRT] ModelImporter.cpp:107: Parsing node: Softmax [Softmax]
[03/31/2020-17:14:05] [V] [TRT] ModelImporter.cpp:123: Searching for input: biased_tensor_name
[03/31/2020-17:14:05] [V] [TRT] ModelImporter.cpp:129: Softmax [Softmax] inputs: [biased_tensor_name → (-1, 100)],
[03/31/2020-17:14:05] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[03/31/2020-17:14:05] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[03/31/2020-17:14:05] [V] [TRT] ImporterContext.hpp:122: Registering layer: Softmax for ONNX node: Softmax
[03/31/2020-17:14:05] [V] [TRT] ImporterContext.hpp:97: Registering tensor: output/Softmax:01 for ONNX tensor: output/Softmax:01
[03/31/2020-17:14:05] [V] [TRT] ModelImporter.cpp:180: Softmax [Softmax] outputs: [output/Softmax:01 → (-1, -1)],
[03/31/2020-17:14:05] [V] [TRT] ModelImporter.cpp:107: Parsing node: Identity5 [Identity]
[03/31/2020-17:14:05] [V] [TRT] ModelImporter.cpp:123: Searching for input: output/Softmax:01
[03/31/2020-17:14:05] [V] [TRT] ModelImporter.cpp:129: Identity5 [Identity] inputs: [output/Softmax:01 → (-1, -1)],
[03/31/2020-17:14:05] [V] [TRT] ImporterContext.hpp:122: Registering layer: Identity5 for ONNX node: Identity5
[03/31/2020-17:14:05] [V] [TRT] ImporterContext.hpp:97: Registering tensor: output_1 for ONNX tensor: output
[03/31/2020-17:14:05] [V] [TRT] ModelImporter.cpp:180: Identity5 [Identity] outputs: [output → (-1, -1)],
[03/31/2020-17:14:05] [V] [TRT] ModelImporter.cpp:494: Marking output_1 as output: output
----- Parsing of ONNX model cifar100.onnx is Done ----

2

Hi,

Can you try few things:

  1. Check ONNX model using checker function and see if it passes?
    import onnx
    model = onnx.load(“model.onnx”)
    onnx.checker.check_model(model)

  2. If (1) passes, please try one of below trtexec commands to generate TRT model:
    ./trtexec --onnx=model.onnx --shapes=input:32x3x244x244
    ./trtexec --onnx=model.onnx --minShapes=input:1x3x244x244 --optShapes=input:16x3x244x244 --maxShapes=input:32x3x244x244 --shapes=input:5x3x244x244

Thanks

3

Thanks for your reply.
The test(1) passed and I had the same wrong shape with your suggested trtexec params as well.
I have the desired output shape (-1,100), if I replace the last layer ‘softmax’ to ‘relu’.
I noticed some differences of trt’s outputs of ‘softmax’ and ‘relu’.
The ‘softmax’ complained about INT64 weights but ‘relu’ didn’t:

‘softmax’:

[04/01/2020-09:22:00] [V] [TRT] ModelImporter.cpp:107: Parsing node: Softmax [Softmax]
[04/01/2020-09:22:00] [V] [TRT] ModelImporter.cpp:123: Searching for input: biased_tensor_name
[04/01/2020-09:22:00] [V] [TRT] ModelImporter.cpp:129: Softmax [Softmax] inputs: [biased_tensor_name → (-1, 100)],
[04/01/2020-09:22:00] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[04/01/2020-09:22:00] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[04/01/2020-09:22:00] [V] [TRT] ImporterContext.hpp:122: Registering layer: Softmax for ONNX node: Softmax
[04/01/2020-09:22:00] [V] [TRT] ImporterContext.hpp:97: Registering tensor: output_8/Softmax:01 for ONNX tensor: output_8/Softmax:01
[04/01/2020-09:22:00] [V] [TRT] ModelImporter.cpp:180: Softmax [Softmax] outputs: [output_8/Softmax:01 → (-1, -1)],
[04/01/2020-09:22:00] [V] [TRT] ModelImporter.cpp:107: Parsing node: Identity5 [Identity]
[04/01/2020-09:22:00] [V] [TRT] ModelImporter.cpp:123: Searching for input: output_8/Softmax:01
[04/01/2020-09:22:00] [V] [TRT] ModelImporter.cpp:129: Identity5 [Identity] inputs: [output_8/Softmax:01 → (-1, -1)],
[04/01/2020-09:22:00] [V] [TRT] ImporterContext.hpp:122: Registering layer: Identity5 for ONNX node: Identity5
[04/01/2020-09:22:00] [V] [TRT] ImporterContext.hpp:97: Registering tensor: output_1 for ONNX tensor: output
[04/01/2020-09:22:00] [V] [TRT] ModelImporter.cpp:180: Identity5 [Identity] outputs: [output → (-1, -1)],
[04/01/2020-09:22:00] [V] [TRT] ModelImporter.cpp:494: Marking output_1 as output: output
----- Parsing of ONNX model my_model2.onnx is Done ----

‘relu’:
[04/01/2020-09:12:12] [V] [TRT] ModelImporter.cpp:107: Parsing node: Relu [Relu]
[04/01/2020-09:12:12] [V] [TRT] ModelImporter.cpp:123: Searching for input: biased_tensor_name
[04/01/2020-09:12:12] [V] [TRT] ModelImporter.cpp:129: Relu [Relu] inputs: [biased_tensor_name → (-1, 100)],
[04/01/2020-09:12:12] [V] [TRT] ImporterContext.hpp:122: Registering layer: Relu for ONNX node: Relu
[04/01/2020-09:12:12] [V] [TRT] ImporterContext.hpp:97: Registering tensor: output_6/Relu:01 for ONNX tensor: output_6/Relu:01
[04/01/2020-09:12:12] [V] [TRT] ModelImporter.cpp:180: Relu [Relu] outputs: [output_6/Relu:01 → (-1, 100)],
[04/01/2020-09:12:12] [V] [TRT] ModelImporter.cpp:107: Parsing node: Identity5 [Identity]
[04/01/2020-09:12:12] [V] [TRT] ModelImporter.cpp:123: Searching for input: output_6/Relu:01
[04/01/2020-09:12:12] [V] [TRT] ModelImporter.cpp:129: Identity5 [Identity] inputs: [output_6/Relu:01 → (-1, 100)],
[04/01/2020-09:12:12] [V] [TRT] ImporterContext.hpp:122: Registering layer: Identity5 for ONNX node: Identity5
[04/01/2020-09:12:12] [V] [TRT] ImporterContext.hpp:97: Registering tensor: output_1 for ONNX tensor: output
[04/01/2020-09:12:12] [V] [TRT] ModelImporter.cpp:180: Identity5 [Identity] outputs: [output → (-1, 100)],
[04/01/2020-09:12:12] [V] [TRT] ModelImporter.cpp:494: Marking output_1 as output: output
----- Parsing of ONNX model my_model3.onnx is Done ----

4

Hi,

It seems to be a expected behavior.
The implementation of softmax op ONNX parser is such that dimension cannot be directly calculated during parser. But it should not lead to any problem building the engine.
https://github.com/onnx/onnx-tensorrt/blob/97c97f19a18494498192bf3834003b954b1877b1/builtin_op_importers.cpp#L2939

Please do let us know if you are getting performance issues due to softmax dim during parser stage.

Thanks

5

Thank you for your quick support.
OK I will inform you about any kind of issue of it.