Error ExpandDims output when parse Tensorflow model to TensorRT

Description

I was working with TensorRT and met some probelms about parsing Tensorflow network to TensorRT model.
I use expand_dims but get error shape in TensorRT.

The details is as below.

Environment

1. GPU 2070 super
2. Driver 440.31
3. CUDA 10.0.130
4. CUDNN 7.6.5
5. Python 3.6
6. Tensorflow 1.13.1
7. TensorRT 6.0.1.5
8. OS Ubuntu18.04
9. TensorRT 6.0

Flow code

    #
    # Model Code
    #

    sess.run(tf.initialize_all_variables())

    frozen_graph_def = tf.graph_util.convert_variables_to_constants(
            sess,
            tf.get_default_graph().as_graph_def(),
            output_node_names = ["output"]
    )

    dyn_graph = gs.DynamicGraph(frozen_graph_def)

    uff_model = uff.from_tensorflow(
        dyn_graph.as_graph_def(),
        output_nodes=["output"],
        output_filename="./checkpoint/model_test.uff",
        txt=True
    )

    input_name = "input"
    input_shape = (1, 32, 32)
    output_name = "output"
    model_file = "./checkpoint/model_test.uff"

    logger = trt.Logger(trt.Logger.VERBOSE)

    builder = trt.Builder(logger)
    network = builder.create_network()
    parser = trt.UffParser()

    builder.max_workspace_size = (1 << 30)
    # Parse the Uff Network
    parser.register_input(input_name, input_shape)
    parser.register_output(output_name)
    parser.parse(model_file, network)
  1. NCHW
  • model

    input_chw = tf.placeholder(tf.float32, [None, 1, 32, 32],   name="input")
    
    b = tf.constant(np.zeros([32, 32], np.float32), name="b")
    b = tf.expand_dims(b, dim=0, name="expand_dim")
    input_chw = input_chw + b
    
    input_hwc = tf.transpose(input_chw, [0, 2, 3, 1])
    
    conv1_weight = tf.get_variable("conv1_weight", shape=[3, 3, 1, 16])
    conv1_bias   = tf.get_variable("conv1_bias", shape=[16])
    
    conv1 = tf.nn.conv2d(input_hwc, conv1_weight, strides=[1, 1, 1, 1], padding="SAME")
    conv1 = tf.add(conv1, conv1_bias, name="output")
    
  • result

    [TensorRT] VERBOSE: UFFParser: Parsing input[Op: Input].
    [TensorRT] VERBOSE: UFFParser: input -> [1,32,32]
    [TensorRT] VERBOSE: UFFParser: Applying order forwarding to: input
    [TensorRT] VERBOSE: UFFParser: Parsing b[Op: Const].
    [TensorRT] VERBOSE: UFFParser: b -> [32,32]
    [TensorRT] VERBOSE: UFFParser: Applying order forwarding to: b
    [TensorRT] VERBOSE: UFFParser: Parsing expand_dim[Op: ExpandDims]. Inputs: b
    [TensorRT] VERBOSE: UFFParser: expand_dim -> [1,32,32]
    [TensorRT] VERBOSE: UFFParser: Applying order forwarding to: expand_dim
    [TensorRT] VERBOSE: UFFParser: Parsing add[Op: Binary]. Inputs: input, expand_dim
    [TensorRT] VERBOSE: UFFParser: add -> [1,32,32]
    [TensorRT] VERBOSE: UFFParser: Applying order forwarding to: add
    [TensorRT] VERBOSE: UFFParser: Parsing transpose[Op: Transpose]. Inputs: add
    [TensorRT] VERBOSE: UFFParser: transpose -> [1,32,32]
    [TensorRT] VERBOSE: UFFParser: Applying order forwarding to: transpose
    [TensorRT] VERBOSE: UFFParser: Parsing conv1_weight[Op: Const].
    [TensorRT] VERBOSE: UFFParser: conv1_weight -> [3,3,1,16]
    [TensorRT] VERBOSE: UFFParser: Applying order forwarding to: conv1_weight
    [TensorRT] VERBOSE: UFFParser: Parsing Conv2D[Op: Conv]. Inputs: transpose, conv1_weight
    [TensorRT] VERBOSE: UFFParser: Conv2D -> [16,32,32]
    [TensorRT] VERBOSE: UFFParser: Applying order forwarding to: Conv2D
    [TensorRT] VERBOSE: UFFParser: Parsing conv1_bias[Op: Const].
    [TensorRT] VERBOSE: UFFParser: conv1_bias -> [16]
    [TensorRT] VERBOSE: UFFParser: Applying order forwarding to: conv1_bias
    [TensorRT] VERBOSE: UFFParser: Parsing output[Op: Binary]. Inputs: Conv2D, conv1_bias
    [TensorRT] VERBOSE: UFFParser: output -> [16,32,32]
    [TensorRT] VERBOSE: UFFParser: Applying order forwarding to: output
    [TensorRT] VERBOSE: UFFParser: Parsing MarkOutput_0[Op: MarkOutput]. Inputs: output
    [TensorRT] VERBOSE: UFFParser: Applying order forwarding to: MarkOutput_0
    
  1. HWC Format
  • model

    input_chw = tf.placeholder(tf.float32, [None, 1, 32, 32], name="input")
    input_hwc = tf.transpose(input_chw, [0, 2, 3, 1])
    
    b = tf.constant(np.zeros([32, 32], np.float32), name="b")
    b = tf.expand_dims(b, dim=2, name="expand_dim")
    input_hwc = input_hwc + b
    
    # input_hwc = tf.transpose(input_chw, [0, 2, 3, 1])
    
    conv1_weight = tf.get_variable("conv1_weight", shape=[3, 3, 1, 16])
    conv1_bias   = tf.get_variable("conv1_bias", shape=[16])
    
    conv1 = tf.nn.conv2d(input_hwc, conv1_weight, strides=[1, 1, 1, 1], padding="SAME")
    conv1 = tf.add(conv1, conv1_bias, name="output")
    
  • result

    [TensorRT] VERBOSE: UFFParser: Parsing input[Op: Input].
    [TensorRT] VERBOSE: UFFParser: input -> [1,32,32]
    [TensorRT] VERBOSE: UFFParser: Applying order forwarding to: input
    [TensorRT] VERBOSE: UFFParser: Parsing transpose[Op: Transpose]. Inputs: input
    [TensorRT] VERBOSE: UFFParser: transpose -> [1,32,32]
    [TensorRT] VERBOSE: UFFParser: Applying order forwarding to: transpose
    [TensorRT] VERBOSE: UFFParser: Parsing b[Op: Const].
    [TensorRT] VERBOSE: UFFParser: b -> [32,32]
    [TensorRT] VERBOSE: UFFParser: Applying order forwarding to: b
    [TensorRT] VERBOSE: UFFParser: Parsing expand_dim[Op: ExpandDims]. Inputs: b
    [TensorRT] VERBOSE: UFFParser: expand_dim -> [32,1,32]
    [TensorRT] VERBOSE: UFFParser: Applying order forwarding to: expand_dim
    [TensorRT] VERBOSE: UFFParser: Parsing add[Op: Binary]. Inputs: transpose, expand_dim
    [TensorRT] VERBOSE: UFFParser: add -> [32,32,32]
    [TensorRT] VERBOSE: UFFParser: Applying order forwarding to: add
    [TensorRT] VERBOSE: UFFParser: Parsing conv1_weight[Op: Const].
    [TensorRT] VERBOSE: UFFParser: conv1_weight -> [3,3,1,16]
    [TensorRT] VERBOSE: UFFParser: Applying order forwarding to: conv1_weight
    [TensorRT] VERBOSE: UFFParser: Parsing Conv2D[Op: Conv]. Inputs: add, conv1_weight
    [TensorRT] ERROR: Conv2D: kernel weights has count 144 but 4608 was expected
    [TensorRT] ERROR: Conv2D: count of 144 weights in kernel, but kernel dimensions (3,3) with 32 input channels, 16 output channels and 1 groups were specified. Expected Weights count is 32 * 3*3 * 16 / 1 = 4608
    [TensorRT] VERBOSE: UFFParser: Conv2D -> []
    [TensorRT] VERBOSE: UFFParser: Applying order forwarding to: Conv2D
    [TensorRT] VERBOSE: UFFParser: Parsing conv1_bias[Op: Const].
    [TensorRT] VERBOSE: UFFParser: conv1_bias -> [16]
    [TensorRT] VERBOSE: UFFParser: Applying order forwarding to: conv1_bias
    [TensorRT] VERBOSE: UFFParser: Parsing output[Op: Binary]. Inputs: Conv2D, conv1_bias
    [TensorRT] ERROR: UffParser: Parser error: output: The input to the Scale Layer is required to have a minimum of 3 dimensions.
    
  1. Question
    1. Why input_hwc = tf.transpose(input_chw, [0, 2, 3, 1]) seems no influence as below,

      [TensorRT] VERBOSE: UFFParser: Parsing input[Op: Input].
      [TensorRT] VERBOSE: UFFParser: input -> [1,32,32]
      [TensorRT] VERBOSE: UFFParser: Applying order forwarding to: input
      [TensorRT] VERBOSE: UFFParser: Parsing transpose[Op: Transpose]. Inputs: input
      [TensorRT] VERBOSE: UFFParser: transpose -> [1,32,32]

    2. Why with nhwc format, the b = tf.expand_dims(b, dim=2, name="expand_dim") will lead to error output as below.

      [TensorRT] VERBOSE: UFFParser: Parsing b[Op: Const].
      [TensorRT] VERBOSE: UFFParser: b -> [32,32]
      [TensorRT] VERBOSE: UFFParser: Applying order forwarding to: b
      [TensorRT] VERBOSE: UFFParser: Parsing expand_dim[Op: ExpandDims]. Inputs: b
      [TensorRT] VERBOSE: UFFParser: expand_dim -> [32,1,32]

Hi,

I will recommend you to try tf2ONNX and ONNX parser for TRT engine generation. Please refer below link:


Thanks

Thank you,

I will try it.