TensorRT concatenate layer doesn't work

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Hi,
I’m trying to run TensorRT on a Keras model developed for High Energy Physics stuff.
When I compile, I get no errors from TensorRT but the parse fails at runtime.
Apparently, there’s a problem when concatenating one Input Layer (infos) to a Flatten one (flat), I tried with a Dense after infos, but got no solution.

Using TensorRT 4 C++ API, with cuda 9.0.

Here’s the function to create the model with Keras:

def model_build(img_size = 16, n_channels=20, labels=67):
hit_shapes = Input(shape=(n_channels,img_size,img_size), name=‘hit_shape_input’)
infos = Input(shape=(labels,), name=‘info_input’)

conv = Conv2D(64, (4, 4), activation='relu', padding='same', data_format="channels_first", name='conv1')(hit_shapes)
conv = Conv2D(64, (3, 3), activation='relu', padding='same', data_format="channels_first", name='conv2')(conv)
pool = MaxPooling2D(pool_size=(2, 2), padding='same', data_format="channels_first", name='pool1')(conv)

conv = Conv2D(128, (3, 3), activation='relu', padding='same', data_format="channels_first", name='conv3')(pool)
conv = Conv2D(128, (3, 3), activation='relu', padding='same', data_format="channels_first", name='conv4')(conv)
pool = MaxPooling2D(pool_size=(2, 2), padding='same', data_format="channels_first", name='pool2')(conv)

conv = Conv2D(64, (3, 3), activation='relu', padding='same', data_format="channels_first", name='conv5')(pool)
pool = MaxPooling2D(pool_size=(2, 2), padding='same', data_format="channels_first", name='pool3')(conv)

flat = Flatten(data_format='channels_first')(pool)
concat = Concatenate()([flat, infos])

dense = Dense(128, activation='sigmoid', kernel_constraint=max_norm(1.0), name='dense1')(concat)
dense = Dense(64, activation='sigmoid', kernel_constraint=max_norm(1.0), name='dense2')(dense)

pred = Dense(2, activation='softmax', kernel_constraint=max_norm(1.0), name='output')(dense)

model = Model(inputs=[hit_shapes, infos], outputs=pred)
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
return model

And here’s the error I get:

ERROR: concatenate_1/concat: all concat input tensors must have the same number of dimensions
ERROR: UFFParser: Parser error: dense1/BiasAdd: The input to the Scale Layer is required to have a minimum of 3 dimensions.

2

Hello,

Please make sure ‘flat’ and ‘infos’ have the same dimension.

3

Here’s the model summary:

Layer (type) Output Shape Param # Connected to

hit_shape_input (InputLayer) (None, 20, 16, 16) 0

conv1 (Conv2D) (None, 64, 16, 16) 32064 hit_shape_input[0][0]

conv2 (Conv2D) (None, 64, 16, 16) 36928 conv1[0][0]

pool1 (MaxPooling2D) (None, 64, 8, 8) 0 conv2[0][0]

conv3 (Conv2D) (None, 128, 8, 8) 73856 pool1[0][0]

conv4 (Conv2D) (None, 128, 8, 8) 147584 conv3[0][0]

pool2 (MaxPooling2D) (None, 128, 4, 4) 0 conv4[0][0]

conv5 (Conv2D) (None, 128, 4, 4) 147584 pool2[0][0]

pool3 (MaxPooling2D) (None, 128, 2, 2) 0 conv5[0][0]

flatten_1 (Flatten) (None, 512) 0 pool3[0][0]

info_input (InputLayer) (None, 67) 0

concatenate_1 (Concatenate) (None, 579) 0 flatten_1[0][0]
info_input[0][0]

dense1 (Dense) (None, 256) 148480 concatenate_1[0][0]

dense2 (Dense) (None, 128) 32896 dense1[0][0]

dense3 (Dense) (None, 64) 8256 dense2[0][0]

output (Dense) (None, 2) 130 dense3[0][0]

Total params: 627,778
Trainable params: 627,778
Non-trainable params: 0

As you can see, the shape is the same (Respectively (None, 512) and (None, 67)) and the Concatenate works on Keras, but it seems it gives some issue on TensorRT.

4

Thanks. Can you share the .pb for us to repro? you can DM me if you’d like.

5

Here it is:

[url]https://drive.google.com/file/d/1f8WMfp63XPwNp-RCADWjvwjakBE_KqmJ/view?usp=sharing[/url]

6

Still no updates?

7

Hello,

Per engineering, this seems like a limitation of the UFF parser. Basically, UFF can’t handle tensors of rank < 3, so when it flattens, it’s actually creating a 4D tensor instead of a 2D one and the subsequent concat fails.

While we work on fixing this for future releases, one work around for now is: try using a 4D input tensor instead of the current 2D one, and replace flatten with a reshape.

regards,
NVIDIA Enterprise Support

8

Hi,

Actually I already tried to Reshape the other Input Tensor into a 4D Tensor (more precisely, since it couldn’t be reshaped properly because of the number of the input features, I first connected it to a Dense Layer of 64 neurons and then Reshaped that into a 4D and tried to Concatenate it to the last MaxPooling Layer). Afterwards, I used a Flatten Layer.

What you are suggesting me, then, is to use a Reshape also instead of the Flatten Layer (after Concatenating the 4D Tensors).

I will test this solution in a few days, and let you know if it works.

Anyway, if I developed the same model with Caffe, is it supposed to work properly? I’ve never used Caffe, but I’m considering if I should move to it in order to work with TensorRT.

In the meantime, I DM you about other problems I should solve to import this model in TensorRT.

Thank you for the support,
Tony

9

Here’s the new model, with a dense layer for the input_info, a Reshape, Concatenate and then Reshape into 2D tensor for the last dense and softmax layers:

https://drive.google.com/file/d/1PkW7b6FIGz_M0Y5bvsA0dU5LFAxNqTq1/view?usp=sharing

Still not working, it gives an error with the UffParser. Any other suggestions?

In the meantime, can you check the DM I sent you as well?

Thank you for the support,
Tony

10

what UffParser error are you seeing? can you upload the full log?

11

Here’s the error:

ERROR: dense1/MatMul: at least three non-batch dimensions are required for input
ERROR: UFFParser: Parser error: dense1/BiasAdd: The input to the Scale Layer is required to have a minimum of 3 dimensions.

I used the work around you suggested me (so I’m not using a Flatten Layer but a normal Reshape, and I’m Concatenating 4D tensors.

12

Hi,

Any update on this issue?

I am facing the same error.

Regards,
Miguel

13

Hello, we’ve deprecateed UFF and caffe, and recommend our customers move to ONNX. Please see: Release Notes :: NVIDIA Deep Learning TensorRT Documentation

"Deprecated Features

The following features are deprecated in TensorRT 7.0.0:
…
Deprecation of Caffe Parser and UFF Parser - We are deprecating Caffe Parser and UFF Parser in TensorRT 7. They will be tested and functional in the next major release of TensorRT 8, but we plan to remove the support in the subsequent major release. Plan to migrate your workflow to use tf2onnx, keras2onnx or TensorFlow-TensorRT (TF-TRT) for deployment."