Concat error with uff parser for TensorRT4

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Hi, i have a keras model and convert it to a uff model.
All ops in the model are supported by uffparser.
But there is a concat error when parsing the model with uff parser. I am confused.

[TensorRT] INFO: UFFParser: parsing batch_normalization_20/batchnorm/mul_1
[TensorRT] INFO: UFFParser: parsing batch_normalization_20/beta
[TensorRT] INFO: UFFParser: parsing batch_normalization_20/moving_mean
[TensorRT] INFO: UFFParser: parsing batch_normalization_20/batchnorm/mul_2
[TensorRT] INFO: UFFParser: parsing batch_normalization_20/batchnorm/sub
[TensorRT] INFO: UFFParser: parsing batch_normalization_20/batchnorm/add_1
[TensorRT] INFO: UFFParser: parsing leakyrelu_20/LeakyRelu/mul
[TensorRT] INFO: UFFParser: parsing leakyrelu_20/LeakyRelu/Maximum
[TensorRT] INFO: UFFParser: parsing leakyrelu_21/LeakyRelu/alpha
[TensorRT] INFO: UFFParser: parsing conv2d_21/kernel
[TensorRT] INFO: UFFParser: parsing conv2d_21/convolution
[TensorRT] INFO: UFFParser: parsing batch_normalization_21/moving_variance
[TensorRT] INFO: UFFParser: parsing batch_normalization_21/batchnorm/add/y
[TensorRT] INFO: UFFParser: parsing batch_normalization_21/batchnorm/add
[TensorRT] INFO: UFFParser: parsing batch_normalization_21/batchnorm/Rsqrt
[TensorRT] INFO: UFFParser: parsing batch_normalization_21/gamma
[TensorRT] INFO: UFFParser: parsing batch_normalization_21/batchnorm/mul
[TensorRT] INFO: UFFParser: parsing batch_normalization_21/batchnorm/mul_1
[TensorRT] INFO: UFFParser: parsing batch_normalization_21/beta
[TensorRT] INFO: UFFParser: parsing batch_normalization_21/moving_mean
[TensorRT] INFO: UFFParser: parsing batch_normalization_21/batchnorm/mul_2
[TensorRT] INFO: UFFParser: parsing batch_normalization_21/batchnorm/sub
[TensorRT] INFO: UFFParser: parsing batch_normalization_21/batchnorm/add_1
[TensorRT] INFO: UFFParser: parsing leakyrelu_21/LeakyRelu/mul
[TensorRT] INFO: UFFParser: parsing leakyrelu_21/LeakyRelu/Maximum
[TensorRT] INFO: UFFParser: parsing reshape_1/Shape
[TensorRT] INFO: UFFParser: parsing reshape_1/strided_slice/stack
[TensorRT] INFO: UFFParser: parsing reshape_1/strided_slice/stack_1
[TensorRT] INFO: UFFParser: parsing reshape_1/strided_slice/stack_2
[TensorRT] INFO: UFFParser: parsing reshape_1/strided_slice
[TensorRT] INFO: UFFParser: parsing reshape_1/Reshape/shape/1
[TensorRT] INFO: UFFParser: parsing reshape_1/Reshape/shape/2
[TensorRT] INFO: UFFParser: parsing reshape_1/Reshape/shape/3
[TensorRT] INFO: UFFParser: parsing reshape_1/Reshape/shape
[TensorRT] INFO: UFFParser: parsing reshape_1/Reshape
[TensorRT] INFO: UFFParser: parsing concatenate_1/concat
[TensorRT] ERROR: UFFParser: Parser error: concatenate_1/concat: dim order not consistant with previous layers
[TensorRT] ERROR: Failed to parse UFF File ‘yolo.uff’

The layer operations in keras are:
tmp2 = Reshape([13,13,256])(tmp2)
tmp = Concatenate(axis=-1)([tmp, tmp2])
where original tmp2 is 26x26x64, original tmp is 13x13x1024, after reshaping, tmp2 is 13x13x256, and concat with tmp to 13x13x1280.

Is there any solution? thanks.

2

Hello, can you provide details on the platforms you are using?

Linux distro and version
GPU type
nvidia driver version
CUDA version
CUDNN version
Python version [if using python]
Tensorflow version
TensorRT version

Any usage/source file you can provide will help us debug too.

3

Hi, the platform details:
Ubuntu 16.04.5 LTS (GNU/Linux 4.15.0-24-generic x86_64)
Tesla V100
Driver Version: 396.26
cuda-9.0
cudnn-7.1.4
Python 3.5.0
tensorflow-1.8.0
TensorRT-4.0.1.6

The parser log and tensor details are listed above.

The corresponding operations and layer:

   [b]"""
   convolutional_19
   batch_normalize=1
   filters=1024
   size=3
   stride=1
   pad=1
   activation=leaky
   """[/b]

   #read weights from yolo.weights
   #weights_file.seek(count)
   bias = np.ndarray(shape=(1024,),dtype='float32',buffer=weights_file.read(1024*4))
   bn_list = np.ndarray(shape=(3, 1024),dtype='float32',buffer=weights_file.read(1024*3*4))
   bn_weights = [bn_list[0], bias, bn_list[1], bn_list[2]]
   weights = np.ndarray(shape=(1024,1024,3,3),dtype='float32',buffer=weights_file.read(1024*1024*3*3*4))
   weights = np.transpose(weights,(2,3,1,0))

   #read for convolution
   tmp = Convolution2D(1024,3,3,
                 subsample=(1,1),
                 border_mode='same',
                 activation=None, 
                 weights=[weights], 
                 bias=False,
                 W_regularizer=l2(0.0005),
                 name='conv2d_20')(tmp)

   #batchnormalization
   tmp = BatchNormalization(weights=bn_weights,name='batch_normalization_20')(tmp)
                        
   #activation
   tmp = LeakyReLU(alpha=0.1,name='leakyrelu_20')(tmp)

   #help to go back.   
   count += (1024 + 1024*3 + 3*3*1024*1024)*4
   print "file read to",count


    

   [b]"""
   convolutional_20
   batch_normalize=1
   filters=64
   size=3
   stride=1
   pad=1
   activation=leaky
   """[/b]

   #read weights from yolo.weights
   #weights_file.seek(count)
   bias = np.ndarray(shape=(64,),dtype='float32',buffer=weights_file.read(64*4))
   bn_list = np.ndarray(shape=(3, 64),dtype='float32',buffer=weights_file.read(64*3*4))
   bn_weights = [bn_list[0], bias, bn_list[1], bn_list[2]]
   weights = np.ndarray(shape=(64,512,1,1),dtype='float32',buffer=weights_file.read(64*512*1*1*4))
   weights = np.transpose(weights,(2,3,1,0))

   #read for convolution
   tmp2 = Convolution2D(64,1,1,
                 subsample=(1,1),
                 border_mode='same',
                 activation=None, 
                 weights=[weights], 
                 bias=False,
                 W_regularizer=l2(0.0005),
                 name='conv2d_21')(image_tmp_output)

   #batchnormalization
   tmp2 = BatchNormalization(weights=bn_weights,name='batch_normalization_21')(tmp2)
                        
   #activation
   tmp2 = LeakyReLU(alpha=0.1,name='leakyrelu_21')(tmp2)

   #help to go back.   
   count += (64 + 64*3 + 1*1*64*512)*4
   print "file read to",count

   [b]tmp2 = Reshape([13,13,256])(tmp2)
   tmp = Concatenate()([tmp2, tmp])

[/b]

   [b]"""
   convolutional_21
   batch_normalize=1
   filters=1024
   size=3
   stride=1
   pad=1
   activation=leaky
   """[/b]

   #read weights from yolo.weights
   #weights_file.seek(count)
   bias = np.ndarray(shape=(1024,),dtype='float32',buffer=weights_file.read(1024*4))
   bn_list = np.ndarray(shape=(3, 1024),dtype='float32',buffer=weights_file.read(1024*3*4))
   bn_weights = [bn_list[0], bias, bn_list[1], bn_list[2]]
   weights = np.ndarray(shape=(1024,1280,3,3),dtype='float32',buffer=weights_file.read(1024*1280*3*3*4))
   weights = np.transpose(weights,(2,3,1,0))

   #read for convolution
   tmp = Convolution2D(1024,3,3,
                 subsample=(1,1),
                 border_mode='same',
                 activation=None, 
                 weights=[weights], 
                 bias=False,
                 W_regularizer=l2(0.0005),
                 name='conv2d_22')(tmp)

   #batchnormalization
   tmp = BatchNormalization(weights=bn_weights,name='batch_normalization_22')(tmp)
                        
   #activation
   tmp = LeakyReLU(alpha=0.1,name='leakyrelu_22')(tmp)

   #help to go back.   
   count += (1024 + 1024*3 + 3*3*1024*1280)*4
   print "file read to",count
4

Hello jianxiangm,

What is format is network being run in? The reshape operation sets the order of the reshape layer to linear order. So if the concat operation has non-linear order (i.e. if any transposes have been inserted) then this issue may arise.

One way to check if order is the issue, can you run the network in NCHW format. So that no transposes are inserted in the network.

Thanks,
NVIDIA Enterprise Support

5

Hi, NVES.
All tensors in my network (Tensorflow) is NHWC, do you mean i need to convert all of them to NCHW to uff parser?
If so, it’s so complex.

I used resnet tf model for TensorRT, which is also NHWC format, but it is good when using uff parser.

By the way, i have no transpose in network, just like the source file:
tmp2 = Reshape([13,13,256])(tmp2)
tmp = Concatenate()([tmp2, tmp])
tmp2(HWC):26x26x64->13x13X256
tmp(HWC):13X13X1024->13X13X1280

Thanks.

6

Hi,

A few additional requests:

Is it possible to run your network in NCHW and see if it works? Some operations in TensorRT are run in NCHW (like convolution) so if the input network is in NHWC, the parser will insert transposes to convert it to NCHW.

We’d like to try a local repro of the issue for a more deeper debug. Can you please share your .pb file and the generated .uff and .uff text files ? (upload to github and PM me the url if you’d like).

7

Hello,

Looks like a bug in Uff Parser code. The fix will be available in a future version of TensorRT. In the meantime, we recommend to modify the network to run in NCHW format (rather than NHWC). That should get rid of the bug.

The issue is due to some transposes inserted in the parser due to NHWC format

8

ok，thanks a lot.

9

Hello

I am trying to get a Tensorflow/Keras model (Yolov2) which has a concat layer and a few permute_dimensions layer into a TesnorRT engine running in DrivePX2 (TensorRT 4) on C++.

I would find the same error as jianxiangm when trying to parse the UFF into a TensorRT engine in C++:

UFFParser: parsing model_3/Concat_1_tf/concat
Parameter check failed at: Network.cpp::addConcatenation::152, condition: first->getDimensions().d[j] == dims.d[j] && "All non-channel dimensions must match across tensors."

I tried to solve it by including a couple of permute_dimensions before and after the concat operation, so that the Concat occurs on the 1st axis: as if NCHW but keeping the rest of the network NHWC. It did not work, still same error when parsing the UFF into a C++ TensortRT.
I tried changing on the the input format: so that input is in NCHW format but putting a permute_dimensions after input, so that I can keep the rest of the network in NHWC format. It did not work.

I decided to translate the entire network (In Keras/Tensorflow) to NCHW: This would still have a permute_dimensions and concat would occur in the 1st channel. I encountered another error: On the Batchnorm layers, it seems Keras introduces Reshape operations getting the following problem when parsing the UFF:

UFFParser: parsing norm_1_1/Reshape_7
UFFParser: Parser error: norm_1_1/Reshape_7: You cannot reshape with a different volume
Failed to parse UFF

Which I think it is a bug of the UFF parser.

I am using TensorRT 4 on Host and on DrivePX2.

Any idea why this is happening? What can I do?

I understand some of these issues will be fix or are fix in current versions of TensorRT, but I see there has not been an update on the TensorRT version for DrivePX2. So, I need to work with TensorRT 4

Thank you and regards

10

@ dtorres1 : I am having the same issue with TensorRT 5.1.5 as well. Keras/Tensorflow backend puts reshape while using BatchNormalizaation in NCHW format and UFF parser throws error same as mentioned in your post.
I have tried replacing reshape with squeeze and still UFF parser throws error as follows :

[I] [TRT] UFFParser: Applying order forwarding to: batch_normalization_1/Reshape_3
[I] [TRT] UFFParser: parsing batch_normalization_1/Squeeze_3
trtexec: uff/UffParser.cpp:1542: std::shared_ptr<ParserLayer> UffParser::parseSqueeze(const uff::Node&, const Fields&, NodesMap&): Assertion `inputs.size() == 1' failed.

I want to use Convolution with BatchNormalization in channel first format(NCHW) as TensorRT performs convolution in channel first (NCHW) format.