Bad shape (TRT 7)

csaba.janos · March 31, 2020, 4:33pm

Description

I created an ONNX sample file having output shape Nx100.
When I parsed the ONNX with trtexec I got (-1,-1) for output shape. Why not (-1, 100)?
Please help.

Environment

TensorRT Version: 7.0.0.11
GPU Type: Quadro P1000
Nvidia Driver Version: 441.22
CUDA Version: 10.2.89
CUDNN Version: 7.6.5.32
Operating System + Version: Windows 10 Enterprise 64bit
Python Version (if applicable): 3.6.7
TensorFlow Version (if applicable): 1.13.1
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Steps To Reproduce

model = tf.keras.models.Sequential([tf.keras.layers.InputLayer(name=‘input’,input_shape=(32,32,3)),
tf.keras.layers.Conv2D(16, (3,3), activation=‘relu’),
tf.keras.layers.MaxPool2D(),
tf.keras.layers.Conv2D(32, (3,3), activation=‘relu’),
tf.keras.layers.MaxPool2D(),
tf.keras.layers.Conv2D(64, (3,3), activation=‘relu’),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(1000, activation=‘relu’),
tf.keras.layers.Dense(100, activation=‘softmax’, name=‘output’)])
model.summary()

Layer (type) Output Shape Param #

conv2d (Conv2D) (None, 30, 30, 16) 448

max_pooling2d (MaxPooling2D) (None, 15, 15, 16) 0

conv2d_1 (Conv2D) (None, 13, 13, 32) 4640

max_pooling2d_1 (MaxPooling2 (None, 6, 6, 32) 0

conv2d_2 (Conv2D) (None, 4, 4, 64) 18496

flatten (Flatten) (None, 1024) 0

dense (Dense) (None, 1000) 1025000

output (Dense) (None, 100) 100100

=================================================================
Total params: 1,148,684
Trainable params: 1,148,684
Non-trainable params: 0

trtexec --verbose --explicitBatch --workspace=128 --onnx=cifar100.onnx

output:
[03/31/2020-17:14:05] [V] [TRT] ModelImporter.cpp:123: Searching for input: output/bias:0
[03/31/2020-17:14:05] [V] [TRT] ModelImporter.cpp:129: Add [Add] inputs: [transformed_tensor -> (-1, 100)], [output/bias:0 -> (100)],
[03/31/2020-17:14:05] [V] [TRT] ImporterContext.hpp:122: Registering layer: Add for ONNX node: Add
[03/31/2020-17:14:05] [V] [TRT] ImporterContext.hpp:97: Registering tensor: biased_tensor_name for ONNX tensor: biased_tensor_name
[03/31/2020-17:14:05] [V] [TRT] ModelImporter.cpp:180: Add [Add] outputs: [biased_tensor_name -> (-1, 100)],
[03/31/2020-17:14:05] [V] [TRT] ModelImporter.cpp:107: Parsing node: Softmax [Softmax]
[03/31/2020-17:14:05] [V] [TRT] ModelImporter.cpp:123: Searching for input: biased_tensor_name
[03/31/2020-17:14:05] [V] [TRT] ModelImporter.cpp:129: Softmax [Softmax] inputs: [biased_tensor_name -> (-1, 100)],
[03/31/2020-17:14:05] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[03/31/2020-17:14:05] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[03/31/2020-17:14:05] [V] [TRT] ImporterContext.hpp:122: Registering layer: Softmax for ONNX node: Softmax
[03/31/2020-17:14:05] [V] [TRT] ImporterContext.hpp:97: Registering tensor: output/Softmax:01 for ONNX tensor: output/Softmax:01
[03/31/2020-17:14:05] [V] [TRT] ModelImporter.cpp:180: Softmax [Softmax] outputs: [output/Softmax:01 -> (-1, -1)],
[03/31/2020-17:14:05] [V] [TRT] ModelImporter.cpp:107: Parsing node: Identity5 [Identity]
[03/31/2020-17:14:05] [V] [TRT] ModelImporter.cpp:123: Searching for input: output/Softmax:01
[03/31/2020-17:14:05] [V] [TRT] ModelImporter.cpp:129: Identity5 [Identity] inputs: [output/Softmax:01 -> (-1, -1)],
[03/31/2020-17:14:05] [V] [TRT] ImporterContext.hpp:122: Registering layer: Identity5 for ONNX node: Identity5
[03/31/2020-17:14:05] [V] [TRT] ImporterContext.hpp:97: Registering tensor: output_1 for ONNX tensor: output
[03/31/2020-17:14:05] [V] [TRT] ModelImporter.cpp:180: Identity5 [Identity] outputs: [output -> (-1, -1)],
[03/31/2020-17:14:05] [V] [TRT] ModelImporter.cpp:494: Marking output_1 as output: output
----- Parsing of ONNX model cifar100.onnx is Done ----

SunilJB · April 1, 2020, 6:35am

Hi,

Can you try few things:

Check ONNX model using checker function and see if it passes?
import onnx
model = onnx.load(“model.onnx”)
onnx.checker.check_model(model)
If (1) passes, please try one of below trtexec commands to generate TRT model:
./trtexec --onnx=model.onnx --shapes=input:32x3x244x244
./trtexec --onnx=model.onnx --minShapes=input:1x3x244x244 --optShapes=input:16x3x244x244 --maxShapes=input:32x3x244x244 --shapes=input:5x3x244x244

Thanks

csaba.janos · April 1, 2020, 7:30am

Thanks for your reply.
The test(1) passed and I had the same wrong shape with your suggested trtexec params as well.
I have the desired output shape (-1,100), if I replace the last layer ‘softmax’ to ‘relu’.
I noticed some differences of trt’s outputs of ‘softmax’ and ‘relu’.
The ‘softmax’ complained about INT64 weights but ‘relu’ didn’t:

‘softmax’:

[04/01/2020-09:22:00] [V] [TRT] ModelImporter.cpp:107: Parsing node: Softmax [Softmax]
[04/01/2020-09:22:00] [V] [TRT] ModelImporter.cpp:123: Searching for input: biased_tensor_name
[04/01/2020-09:22:00] [V] [TRT] ModelImporter.cpp:129: Softmax [Softmax] inputs: [biased_tensor_name -> (-1, 100)],
[04/01/2020-09:22:00] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[04/01/2020-09:22:00] [W] [TRT] onnx2trt_utils.cpp:198: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[04/01/2020-09:22:00] [V] [TRT] ImporterContext.hpp:122: Registering layer: Softmax for ONNX node: Softmax
[04/01/2020-09:22:00] [V] [TRT] ImporterContext.hpp:97: Registering tensor: output_8/Softmax:01 for ONNX tensor: output_8/Softmax:01
[04/01/2020-09:22:00] [V] [TRT] ModelImporter.cpp:180: Softmax [Softmax] outputs: [output_8/Softmax:01 -> (-1, -1)],
[04/01/2020-09:22:00] [V] [TRT] ModelImporter.cpp:107: Parsing node: Identity5 [Identity]
[04/01/2020-09:22:00] [V] [TRT] ModelImporter.cpp:123: Searching for input: output_8/Softmax:01
[04/01/2020-09:22:00] [V] [TRT] ModelImporter.cpp:129: Identity5 [Identity] inputs: [output_8/Softmax:01 -> (-1, -1)],
[04/01/2020-09:22:00] [V] [TRT] ImporterContext.hpp:122: Registering layer: Identity5 for ONNX node: Identity5
[04/01/2020-09:22:00] [V] [TRT] ImporterContext.hpp:97: Registering tensor: output_1 for ONNX tensor: output
[04/01/2020-09:22:00] [V] [TRT] ModelImporter.cpp:180: Identity5 [Identity] outputs: [output -> (-1, -1)],
[04/01/2020-09:22:00] [V] [TRT] ModelImporter.cpp:494: Marking output_1 as output: output
----- Parsing of ONNX model my_model2.onnx is Done ----

‘relu’:
[04/01/2020-09:12:12] [V] [TRT] ModelImporter.cpp:107: Parsing node: Relu [Relu]
[04/01/2020-09:12:12] [V] [TRT] ModelImporter.cpp:123: Searching for input: biased_tensor_name
[04/01/2020-09:12:12] [V] [TRT] ModelImporter.cpp:129: Relu [Relu] inputs: [biased_tensor_name -> (-1, 100)],
[04/01/2020-09:12:12] [V] [TRT] ImporterContext.hpp:122: Registering layer: Relu for ONNX node: Relu
[04/01/2020-09:12:12] [V] [TRT] ImporterContext.hpp:97: Registering tensor: output_6/Relu:01 for ONNX tensor: output_6/Relu:01
[04/01/2020-09:12:12] [V] [TRT] ModelImporter.cpp:180: Relu [Relu] outputs: [output_6/Relu:01 -> (-1, 100)],
[04/01/2020-09:12:12] [V] [TRT] ModelImporter.cpp:107: Parsing node: Identity5 [Identity]
[04/01/2020-09:12:12] [V] [TRT] ModelImporter.cpp:123: Searching for input: output_6/Relu:01
[04/01/2020-09:12:12] [V] [TRT] ModelImporter.cpp:129: Identity5 [Identity] inputs: [output_6/Relu:01 -> (-1, 100)],
[04/01/2020-09:12:12] [V] [TRT] ImporterContext.hpp:122: Registering layer: Identity5 for ONNX node: Identity5
[04/01/2020-09:12:12] [V] [TRT] ImporterContext.hpp:97: Registering tensor: output_1 for ONNX tensor: output
[04/01/2020-09:12:12] [V] [TRT] ModelImporter.cpp:180: Identity5 [Identity] outputs: [output -> (-1, 100)],
[04/01/2020-09:12:12] [V] [TRT] ModelImporter.cpp:494: Marking output_1 as output: output
----- Parsing of ONNX model my_model3.onnx is Done ----

SunilJB · April 2, 2020, 8:45am

Hi,

It seems to be a expected behavior.
The implementation of softmax op ONNX parser is such that dimension cannot be directly calculated during parser. But it should not lead to any problem building the engine.

github.com

onnx/onnx-tensorrt/blob/97c97f19a18494498192bf3834003b954b1877b1/builtin_op_importers.cpp#L2939


    ends = decodeOnnxIndices(ctx, ends, dims);


    // TensorRT uses sizes of the output dimensions instead of ends.
    const ShapeTensor sizes = computeSizes(ctx, starts, ends, steps, dims);


    nvinfer1::ISliceLayer* slice = addSlice(ctx, data, starts, sizes, steps);


    RETURN_FIRST_OUTPUT(slice);
}


DEFINE_BUILTIN_OP_IMPORTER(Softmax)
{
    OnnxAttrs attrs(node, ctx);
    // "input : T"
    nvinfer1::ITensor& input = convertToTensor(inputs.at(0), ctx);
    const ShapeTensor dims = shapeOf(ctx, input);


    // "axis : int (default is 1)"
    int axis = attrs.get("axis", 1);


    // "Negative value means counting dimensions from the back.

Please do let us know if you are getting performance issues due to softmax dim during parser stage.

Thanks

csaba.janos · April 2, 2020, 3:53pm

Thank you for your quick support.
OK I will inform you about any kind of issue of it.