Deconvolution Layer Accuracy

Please provide the following info (check/uncheck the boxes after creating this topic):
Software Version
DRIVE OS Linux 5.2.6
DRIVE OS Linux 5.2.6 and DriveWorks 4.0
DRIVE OS Linux 5.2.0
[*] DRIVE OS Linux 5.2.0 and DriveWorks 3.5
NVIDIA DRIVE™ Software 10.0 (Linux)
NVIDIA DRIVE™ Software 9.0 (Linux)
other DRIVE OS version

Target Operating System
[*] Linux

Hardware Platform
[*] NVIDIA DRIVE™ AGX Xavier DevKit (E3550)
NVIDIA DRIVE™ AGX Pegasus DevKit (E3550)

SDK Manager Version
[*] other

Host Machine Version
native Ubuntu 18.04
[*] other

Hi. I noticed that I was using Deconvolution Layer as part of model development, but it seems that the output results have changed between the PC and the DRIVE AGX Xavier environment. I tried the following environment, can you tell me if such a thing can happen?

Host PC environment (GTX1660)

  • TensorRT 6.0.1 (use official github)
  • TensorRT 6.3.1 (use DRIVE OS 5.2.0 + DriveWorks3.5)

Xaver environment

  • TensorRT 6.3.1 (use DRIVE OS 5.2.0 + DriveWorks3.5)

Within the same PC, there was no difference depending on the version of TensorRT, but the results of PC and Xavier are different. It would be helpful if you could give me some advice.

Dear @tatsuya_suzuki,
Could you confirm if you are using TRT 6.3 on both host and target? You may check the linked libs in executable.

but it seems that the output results have changed between the PC and the DRIVE AGX Xavier environment

Have you just used single deconv layer in your model for verification or you notice output difference just after deconv layer in your DL model?

Note that each DL layer have multiple CUDA kernel implementations and when TensorRT engine is build, it chooses the best CUDA kernel in that run. So it is possible that two different TensorRT engine build runs may have picked up different set of CUDA kernel pipeline. Alsom TensorRT engine may have different set of CUDA kernels on host and target based on the underlying GPU features. But the output difference among layer outputs is not expected to be so huge.

If you notice, huge difference among layer outputs, please share reproducible steps for verification at our end.