Gather layer‘s output is all 0 in trt model

Description

I’ve converted a pytorch detection model to a trt model. The backbone is shufflenet, the code below is channel shuffle operation in which i need divide a tensor into two parts.

def channel_shuffle(self, x):
    batchsize, num_channels, height, width = x.data.size()

    x = x.reshape(batchsize, num_channels // 2, 2, height * width)
    x = x.permute(0, 2, 1, 3)
    x = x.reshape(batchsize, 2, num_channels // 2, height, width)

    x_0 = x.index_select(1, torch.LongTensor([0]).to(x.device)).squeeze(1)
    x_1 = x.index_select(1, torch.LongTensor([1]).to(x.device)).squeeze(1)

        return x_0, x_1

index_select op in onnx is as follow:
‘’’
%635 : Float(1, 2!, 58!, 40, 40) = onnx::Reshape(%633, %634)
%636 : Long(1) = onnx::Constant[value={0}]()
%637 : Float(1, 1, 58, 40, 40) = onnx::Gather[axis=1](%635, %636)
%638 : Float(1, 58, 40, 40) = onnx::Squeeze[axes=[1]](%637)
%639 : Long(1) = onnx::Constant[value={1}]()
%640 : Float(1, 1, 58, 40, 40) = onnx::Gather[axis=1](%635, %639)
%641 : Float(1, 58, 40, 40) = onnx::Squeeze[axes=[1]](%640)
‘’’

This op’s output in pytorch is correct, however, it‘s all 0 in trt inference. I find that it has nothing to do with reshape and permute, just because index_select.
Why is it? Could someone help on this issue? Thank you.

Environment

TensorRT Version: 5.1.5
GPU Type: V100
Nvidia Driver Version: NVIDIA UNIX x86_64 Kernel Module 440.33.01
CUDA Version: 9.0.176
CUDNN Version: 7.4.1
Operating System + Version:CentOS Linux release 7.4.1708
**Python Version **: 3.6
PyTorch Version: 1.0.1

Hi,

Can you try few things:

  1. Check ONNX model using checker function and see if it passes?
    import onnx
    model = onnx.load(“model.onnx”)
    onnx.checker.check_model(model)

  2. If (1) passes, try comparing the ONNX output with pt ops output

  3. If (2) passes, please try trtexec commands to generate TRT model:
    https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec#example-4-running-an-onnx-model-with-full-dimensions-and-dynamic-shapes

  4. If possible, try latest TRT version.

Thanks

Hi, thanks for reply.
When I tried trtexec, an error occurred:

[E] [TRT] (Unnamed Layer* 388) [Concatenation]: all concat input tensors must have the same dimensions except on the concatenation axis
While parsing node number 423 [Reshape]:
ERROR: builtin_op_importers.cpp:1579 In function importReshape:
[8] Assertion failed: get_shape_size(new_shape) == get_shape_size(tensor.getDimensions())
[E] failed to parse onnx file
[E] Engine could not be created
[E] Engine could not be created

It looks like that something wrong with shape in Reshape. And I find that node number 423 corresponds to %992.
The onnx describes as follow:

%954 : Float(1, 128, 10, 10) = onnx::Concat[axis=1](%945, %953)
...
%972 : Tensor = onnx::Constant[value= 1  1  2  2 [ CPUFloatType{4} ]]()
%973 : Float(1, 128, 20, 20) = onnx::Upsample[mode="nearest"](%954, %972)
...
%980 : Float(1, 64, 20, 20) = onnx::Gather[axis=1](%978, %979)
... 
%988 : Float(1, 64, 20, 20) = onnx::Relu(%987)
%989 : Float(1, 128, 20, 20) = onnx::Concat[axis=1](%980, %988)
%990 : Float(1, 256, 20, 20) = onnx::Concat[axis=1](%989, %973)
%991 : Tensor = onnx::Constant[value=   1  128    2  400 [ CPULongType{4} ]]()
%992 : Float(1, 128, 2, 400) = onnx::Reshape(%990, %991)
%993 : Float(1!, 2!, 128!, 400) = onnx::Transpose[perm=[0, 2, 1, 3]](%992)
%994 : Tensor = onnx::Constant[value=   1    2  128   20   20 [ CPULongType{5} ]]()
%995 : Float(1, 2!, 128!, 20, 20) = onnx::Reshape(%993, %994)

1*256*20*20 == 1*128*2*400, new_shape size and tensor shape size are same.

However ,when I use ‘–verbose’, I find upsample’s output(%973) size is (256, 20, 0) which should be (128, 20, 20)

[V] [TRT] 954:Concat -> (128, 10, 10)
...
[V] [TRT] 972:Constant -> (4)
[V] [TRT] 973:Upsample -> (256, 20, 0)

It seems that the scales are (1, 2, 2, 0)

The pytorch code is :

F.interpolate(last_inner, scale_factor=2, mode="nearest")

Why is it? How to fix it?

Hi,

Could you please share the onnx model so we can help better?
I am not sure, but as per error it seems to be issue at concat layer. “All concat input tensors must have the same dimensions except on the concatenation axis”

Thanks

Hi,
Thanks for your reply. Here is the onnx model:

Hi,

I tried to generate trt model on latest TRT 7 release and it seems to be working fine.
PASSED TensorRT.trtexec # trtexec --onnx=net.onnx --explicitBatch --verbose --saveEngine=net.trt

Could you please try to upgrade to latest TRT version and retry to generate the engine file?

Thanks