The inference time of Deconvolution in tensorrt is slower than pytorch

Hi All,
I want to convert one model trained by pytorch1.1.0 to tensorrt6.0.1 to speed up the inferece. The inference time in tensorrt is slower than pytorch. I find the problem comes from the Deconvolution operation. Thus I try a model with only deconv operation that :

class Test_net(nn.Module):
    def __init__(self, num_classes=3):
        super(Test_net, self).__init__()
        self.deconv4 = nn.ConvTranspose2d(2048, 1024, 3, stride=2, output_padding=1, padding=1)
        self.deconv3 = nn.ConvTranspose2d(1024, 1024, 3, stride=2, output_padding=1, padding=1)
        self.deconv2 = nn.ConvTranspose2d(1024, 1024, 3, stride=2, output_padding=1, padding=1)
        self.deconv1 = nn.ConvTranspose2d(1024, num_classes, 3, stride=2, output_padding=1, padding=1)

    def forward(self, x):
        tmp = self.deconv4(x)
        tmp = self.deconv3(tmp)
        tmp = self.deconv2(tmp)
        x = self.deconv1(tmp)

        return x

pytorch speed: 5.95s/1000
tensorrt speed: 9.11s/1000

My env details are:

  1. Ubuntu18.04
  2. TITAN Xp
  3. Driver Version: 418.87.00
  4. cuda vesion: 10.1
  5. cudnn version: 7.6
  6. python version: 3.6.8
  7. pytorch version 1.1.0
  8. tensorrt version: 6.0.1
  9. onnx version 1.6.0