Broadcast Support for Add Operation with UFFParser

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

Hello, I have a custom keras layer. For simplicity, I input the image directly into the custom layer. Here is my keras code:

        input_img = keras.layers.Input( shape=(60, 80, 256 ) )
        out = CustomLayer(num_clusters = 16)( input_img )
        model = keras.models.Model( inputs=input_img, outputs=out )

Code inside the custom layer:

# K: tf.keras.backend 
# Shape of self.C is 1x1x1x256x16
print 'x.shape=', x.shape # x.shape=(?, 60, 80, 256)
# v = K.expand_dims(x, -1) + self.C #original code <--- 
v = K.reshape( x, [ K.shape(x)[0],  K.shape(x)[1], K.shape(x)[2], K.shape(x)[3], 1 ] ) + self.C
print 'v.shape=', v.shape #v.shape=(?, ?, ?, 256, 16)
return v

The original code with expand_dims seem to work fine on my desktop with tensorflow/keras. However, as TensorRT doesnot support expand_dims I am bypassing it with reshape.

However the +self.C to x causes the UFFParser to complain while loading the UFF file.
I was under impression that broadcast should work in this case as noted in https://docs.nvidia.com/deeplearning/sdk/tensorrt-support-matrix/index.html#layers-matrix

[TensorRT] ERROR: UffParser: Parser error: net_vlad_layer_1/add_1: Invalid scale mode, nbWeights: 4096

However, if I do not add self.C to the reshaped x, the UFFParser doesn’t seem to throw the error.
Could help me sort this out please?

C was initialized like this in the keras code:

        self.C = self.add_weight( name='cluster_centers',
                                shape=[1,1,1,256, 16],
                                initializer='uniform',
                                trainable=True)
2

Hi,

To give a further suggestion, would you mind to share a complete sample with us?
We want to reproduce this on our side first.

Thanks.

3

Try this:

Clone the repo and do python main.py

tensorflow.version ‘1.12.0’
keras.version ‘2.2.4’
tensorrt.version ‘5.1.5.0’

4

Hi,

Thanks for the feedback.
We will test this internally and update more information with you later.

5

Just as a heads up my custom layer contains the softmax operator. From this thread https://devtalk.nvidia.com/default/topic/1045262/tensorrt/uffparser-error-unary-not-supported-for-other-non-constant-node/ I understand that unary operations are not yet supported on C++ API. Any updates on this as well ?

I tried a standard VGG16 keras model (imagenet) which has a softmax layer, which resulted in this error:

[loadModelAndCreateEngine] Parse UFF File: uff_ready/vgg16_imagenet.uff
ERROR: UFFParser: Parser error: act_softmax/Exp: Unary not supported for other non-constant node
ERROR: sample_uff_mnist: Fail to parse

However, it seem to work alright on my x86 desktop with python tensorrt. Both (my TX2 and desktop) have tensorRT5.

6

Hi,

May I know how do you install the TensorFlow package.
It looks like you are using python2 but we only release python3 TensorFlow package from JetPack4.2.

Would you mind to reproduce this issue with JetPack4.2 and install the TensorFlow package from here:
[url]https://devtalk.nvidia.com/default/topic/1038957/jetson-tx2/tensorflow-for-jetson-tx2-/[/url]

Thanks.

7

I github link I sent you https://github.com/mpkuse/demo_keras_to_nvidia-uff/blob/master/main.py was run entirely on an x86 desktop, so uses tensorrt-python.

On the desktop the uff parser throws the error.

8

Hi,

Sorry for the late update.
Does this issue also occur on Jetson or x86-limited?

Thanks.

9

On x86 the python wrapper just crashes with a core dump. On Jetson when I load the UFF with C++, it gives segfault.

10

Hi,

I can convert your model from .pb to uff with this sample.
[url]https://github.com/NVIDIA-AI-IOT/tf_to_trt_image_classification[/url]

But due to the API update, the sample for uff to TensorRT is not working with TensorRT v5.0.
Would you mind to take a took first.

Thanks.

11

Hi,
Using your script I could convert my frozen_graph to uff. Particularly the script https://github.com/NVIDIA-AI-IOT/tf_to_trt_image_classification/blob/master/scripts/convert_plan.py

However, the execution of uff_to_plan.cpp (from the script convert_plan.py) still causes core dump.

I tried this on my x86 machine.

12

Hi. I encountered same problems. Any update on this?

13

Hi,

Here is something may cause this issue:

node {
  name: "net_vlad_layer_1/Reshape_1"
  op: "Reshape"
  input: "input_1"
  input: "net_vlad_layer_1/Reshape_1/shape"
  attr {
    key: "T"
    value {
      type: DT_FLOAT
    }
  }
  attr {
    key: "Tshape"
    value {
      type: DT_INT32
    }
  }
}

Is it possible to fix all the type to DT_FLOAT and try it again?

Thanks.

14

I have no idea where this DT_INT32 comes from? Where do you suggest the fix lies? How should I accomplish this?

15

Hi,

Could you check the follow input:

input: "input_1"
  input: "net_vlad_layer_1/Reshape_1/shape"

If there are input tensor, please update the data format into DT_FLOAT.
Thanks.

16

In the frozen graph tensorflow .pb I searched DT_INT32, looks like for some reason padding has it.
Could this be the source of the problem?

node {
  name: "conv1_pad/Pad/paddings"
  op: "Const"
  attr {
    key: "dtype"
    value {
      type: DT_INT32
    }
  }
  attr {
    key: "value"
    value {
      tensor {
        dtype: DT_INT32
        tensor_shape {
          dim {
            size: 4
          }
          dim {
            size: 2
          }
        }
        tensor_content: "\000\000\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\000\000\000\000"
      }
    }
  }
}

About the input_1, I see DT_FLOAT as the data type.

  nodes {
    id: "input_1"
    operation: "Input"
    fields {
      key: "dtype"
      value {
        dtype: DT_FLOAT32
      }
    }
    fields {
      key: "shape"
      value {
        i_list {
          val: -1
          val: 240
          val: 320
          val: 3
        }
      }
    }
  }
17

Hi,

Sorry for the late update.

It looks like the DT_INT32 comes from the input layer:

input_img = keras.layers.Input( shape=(60, 80, 256 ) )

Could you try to add an offline converter before the keras model from INT32 to FLOAT?
And then fix the input layer type into float.

Thanks.