Error ExpandDims output when parse Tensorflow model to TensorRT

lingchao.zhu · April 3, 2020, 6:50am

Description

I was working with TensorRT and met some probelms about parsing Tensorflow network to TensorRT model.
I use expand_dims but get error shape in TensorRT.

The details is as below.

Environment

1. GPU 2070 super
2. Driver 440.31
3. CUDA 10.0.130
4. CUDNN 7.6.5
5. Python 3.6
6. Tensorflow 1.13.1
7. TensorRT 6.0.1.5
8. OS Ubuntu18.04
9. TensorRT 6.0

Flow code

    #
    # Model Code
    #

    sess.run(tf.initialize_all_variables())

    frozen_graph_def = tf.graph_util.convert_variables_to_constants(
            sess,
            tf.get_default_graph().as_graph_def(),
            output_node_names = ["output"]
    )

    dyn_graph = gs.DynamicGraph(frozen_graph_def)

    uff_model = uff.from_tensorflow(
        dyn_graph.as_graph_def(),
        output_nodes=["output"],
        output_filename="./checkpoint/model_test.uff",
        txt=True
    )

    input_name = "input"
    input_shape = (1, 32, 32)
    output_name = "output"
    model_file = "./checkpoint/model_test.uff"

    logger = trt.Logger(trt.Logger.VERBOSE)

    builder = trt.Builder(logger)
    network = builder.create_network()
    parser = trt.UffParser()

    builder.max_workspace_size = (1 << 30)
    # Parse the Uff Network
    parser.register_input(input_name, input_shape)
    parser.register_output(output_name)
    parser.parse(model_file, network)

NCHW

model

input_chw = tf.placeholder(tf.float32, [None, 1, 32, 32],   name="input")

b = tf.constant(np.zeros([32, 32], np.float32), name="b")
b = tf.expand_dims(b, dim=0, name="expand_dim")
input_chw = input_chw + b

input_hwc = tf.transpose(input_chw, [0, 2, 3, 1])

conv1_weight = tf.get_variable("conv1_weight", shape=[3, 3, 1, 16])
conv1_bias   = tf.get_variable("conv1_bias", shape=[16])

conv1 = tf.nn.conv2d(input_hwc, conv1_weight, strides=[1, 1, 1, 1], padding="SAME")
conv1 = tf.add(conv1, conv1_bias, name="output")

result

[TensorRT] VERBOSE: UFFParser: Parsing input[Op: Input].
[TensorRT] VERBOSE: UFFParser: input -> [1,32,32]
[TensorRT] VERBOSE: UFFParser: Applying order forwarding to: input
[TensorRT] VERBOSE: UFFParser: Parsing b[Op: Const].
[TensorRT] VERBOSE: UFFParser: b -> [32,32]
[TensorRT] VERBOSE: UFFParser: Applying order forwarding to: b
[TensorRT] VERBOSE: UFFParser: Parsing expand_dim[Op: ExpandDims]. Inputs: b
[TensorRT] VERBOSE: UFFParser: expand_dim -> [1,32,32]
[TensorRT] VERBOSE: UFFParser: Applying order forwarding to: expand_dim
[TensorRT] VERBOSE: UFFParser: Parsing add[Op: Binary]. Inputs: input, expand_dim
[TensorRT] VERBOSE: UFFParser: add -> [1,32,32]
[TensorRT] VERBOSE: UFFParser: Applying order forwarding to: add
[TensorRT] VERBOSE: UFFParser: Parsing transpose[Op: Transpose]. Inputs: add
[TensorRT] VERBOSE: UFFParser: transpose -> [1,32,32]
[TensorRT] VERBOSE: UFFParser: Applying order forwarding to: transpose
[TensorRT] VERBOSE: UFFParser: Parsing conv1_weight[Op: Const].
[TensorRT] VERBOSE: UFFParser: conv1_weight -> [3,3,1,16]
[TensorRT] VERBOSE: UFFParser: Applying order forwarding to: conv1_weight
[TensorRT] VERBOSE: UFFParser: Parsing Conv2D[Op: Conv]. Inputs: transpose, conv1_weight
[TensorRT] VERBOSE: UFFParser: Conv2D -> [16,32,32]
[TensorRT] VERBOSE: UFFParser: Applying order forwarding to: Conv2D
[TensorRT] VERBOSE: UFFParser: Parsing conv1_bias[Op: Const].
[TensorRT] VERBOSE: UFFParser: conv1_bias -> [16]
[TensorRT] VERBOSE: UFFParser: Applying order forwarding to: conv1_bias
[TensorRT] VERBOSE: UFFParser: Parsing output[Op: Binary]. Inputs: Conv2D, conv1_bias
[TensorRT] VERBOSE: UFFParser: output -> [16,32,32]
[TensorRT] VERBOSE: UFFParser: Applying order forwarding to: output
[TensorRT] VERBOSE: UFFParser: Parsing MarkOutput_0[Op: MarkOutput]. Inputs: output
[TensorRT] VERBOSE: UFFParser: Applying order forwarding to: MarkOutput_0

HWC Format

model

input_chw = tf.placeholder(tf.float32, [None, 1, 32, 32], name="input")
input_hwc = tf.transpose(input_chw, [0, 2, 3, 1])

b = tf.constant(np.zeros([32, 32], np.float32), name="b")
b = tf.expand_dims(b, dim=2, name="expand_dim")
input_hwc = input_hwc + b

# input_hwc = tf.transpose(input_chw, [0, 2, 3, 1])

conv1_weight = tf.get_variable("conv1_weight", shape=[3, 3, 1, 16])
conv1_bias   = tf.get_variable("conv1_bias", shape=[16])

conv1 = tf.nn.conv2d(input_hwc, conv1_weight, strides=[1, 1, 1, 1], padding="SAME")
conv1 = tf.add(conv1, conv1_bias, name="output")

result

[TensorRT] VERBOSE: UFFParser: Parsing input[Op: Input].
[TensorRT] VERBOSE: UFFParser: input -> [1,32,32]
[TensorRT] VERBOSE: UFFParser: Applying order forwarding to: input
[TensorRT] VERBOSE: UFFParser: Parsing transpose[Op: Transpose]. Inputs: input
[TensorRT] VERBOSE: UFFParser: transpose -> [1,32,32]
[TensorRT] VERBOSE: UFFParser: Applying order forwarding to: transpose
[TensorRT] VERBOSE: UFFParser: Parsing b[Op: Const].
[TensorRT] VERBOSE: UFFParser: b -> [32,32]
[TensorRT] VERBOSE: UFFParser: Applying order forwarding to: b
[TensorRT] VERBOSE: UFFParser: Parsing expand_dim[Op: ExpandDims]. Inputs: b
[TensorRT] VERBOSE: UFFParser: expand_dim -> [32,1,32]
[TensorRT] VERBOSE: UFFParser: Applying order forwarding to: expand_dim
[TensorRT] VERBOSE: UFFParser: Parsing add[Op: Binary]. Inputs: transpose, expand_dim
[TensorRT] VERBOSE: UFFParser: add -> [32,32,32]
[TensorRT] VERBOSE: UFFParser: Applying order forwarding to: add
[TensorRT] VERBOSE: UFFParser: Parsing conv1_weight[Op: Const].
[TensorRT] VERBOSE: UFFParser: conv1_weight -> [3,3,1,16]
[TensorRT] VERBOSE: UFFParser: Applying order forwarding to: conv1_weight
[TensorRT] VERBOSE: UFFParser: Parsing Conv2D[Op: Conv]. Inputs: add, conv1_weight
[TensorRT] ERROR: Conv2D: kernel weights has count 144 but 4608 was expected
[TensorRT] ERROR: Conv2D: count of 144 weights in kernel, but kernel dimensions (3,3) with 32 input channels, 16 output channels and 1 groups were specified. Expected Weights count is 32 * 3*3 * 16 / 1 = 4608
[TensorRT] VERBOSE: UFFParser: Conv2D -> []
[TensorRT] VERBOSE: UFFParser: Applying order forwarding to: Conv2D
[TensorRT] VERBOSE: UFFParser: Parsing conv1_bias[Op: Const].
[TensorRT] VERBOSE: UFFParser: conv1_bias -> [16]
[TensorRT] VERBOSE: UFFParser: Applying order forwarding to: conv1_bias
[TensorRT] VERBOSE: UFFParser: Parsing output[Op: Binary]. Inputs: Conv2D, conv1_bias
[TensorRT] ERROR: UffParser: Parser error: output: The input to the Scale Layer is required to have a minimum of 3 dimensions.

Question
1. Why input_hwc = tf.transpose(input_chw, [0, 2, 3, 1]) seems no influence as below,
  
  [TensorRT] VERBOSE: UFFParser: Parsing input[Op: Input].
  [TensorRT] VERBOSE: UFFParser: input → [1,32,32]
  [TensorRT] VERBOSE: UFFParser: Applying order forwarding to: input
  [TensorRT] VERBOSE: UFFParser: Parsing transpose[Op: Transpose]. Inputs: input
  [TensorRT] VERBOSE: UFFParser: transpose → [1,32,32]
2. Why with nhwc format, the b = tf.expand_dims(b, dim=2, name="expand_dim") will lead to error output as below.
  
  [TensorRT] VERBOSE: UFFParser: Parsing b[Op: Const].
  [TensorRT] VERBOSE: UFFParser: b → [32,32]
  [TensorRT] VERBOSE: UFFParser: Applying order forwarding to: b
  [TensorRT] VERBOSE: UFFParser: Parsing expand_dim[Op: ExpandDims]. Inputs: b
  [TensorRT] VERBOSE: UFFParser: expand_dim → [32,1,32]

SunilJB · April 7, 2020, 8:01am

Hi,

I will recommend you to try tf2ONNX and ONNX parser for TRT engine generation. Please refer below link:

github.com

onnx/onnx/blob/main/docs/Operators.md

<!--- SPDX-License-Identifier: Apache-2.0 -->
## Operator Schemas
*This file is automatically generated from the
            [def files](/onnx/defs) via [this script](/onnx/defs/gen_doc.py).
            Do not modify directly and instead edit operator definitions.*

For an operator input/output's differentiability, it can be differentiable,
            non-differentiable, or undefined. If a variable's differentiability
            is not specified, that variable has undefined differentiability.

### ai.onnx (default)
|**Operator**|**Since version**|
|-|-|
|<a href="#Abs">Abs</a>|<a href="Changelog.md#Abs-13">13</a>, <a href="Changelog.md#Abs-6">6</a>, <a href="Changelog.md#Abs-1">1</a>|
|<a href="#Acos">Acos</a>|<a href="Changelog.md#Acos-7">7</a>|
|<a href="#Acosh">Acosh</a>|<a href="Changelog.md#Acosh-9">9</a>|
|<a href="#Add">Add</a>|<a href="Changelog.md#Add-14">14</a>, <a href="Changelog.md#Add-13">13</a>, <a href="Changelog.md#Add-7">7</a>, <a href="Changelog.md#Add-6">6</a>, <a href="Changelog.md#Add-1">1</a>|
|<a href="#And">And</a>|<a href="Changelog.md#And-7">7</a>, <a href="Changelog.md#And-1">1</a>|
|<a href="#ArgMax">ArgMax</a>|<a href="Changelog.md#ArgMax-13">13</a>, <a href="Changelog.md#ArgMax-12">12</a>, <a href="Changelog.md#ArgMax-11">11</a>, <a href="Changelog.md#ArgMax-1">1</a>|
|<a href="#ArgMin">ArgMin</a>|<a href="Changelog.md#ArgMin-13">13</a>, <a href="Changelog.md#ArgMin-12">12</a>, <a href="Changelog.md#ArgMin-11">11</a>, <a href="Changelog.md#ArgMin-1">1</a>|

This file has been truncated. show original

Thanks

lingchao.zhu · April 7, 2020, 11:54am

Thank you,

I will try it.