TensorRT is slower than pth

1462555784 · May 27, 2023, 7:51am

Description

When I convert the PTH model of U2Net to the TensorRT model, the TensorRT fp16 model is 10 times slower than the PTH inference model. The ONNX structure of the model does not have a special network layer. According to the profiler, the time consumption of some convolutional layer and resizing layers is abnormal.

Environment

TensorRT Version: tensorRT7.1.3
GPU Type: Tesla T4
Nvidia Driver Version: 460.106.00
CUDA Version: 11.2
CUDNN Version: 7.2.1
Operating System + Version: centos8
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable): 1.12
Baremetal or Container (if container which image + tag):

Relevant Files

Quantization and Profiler Logs：
[05/27/2023-15:22:51] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [05/27/2023-15:22:51] [I] [I] === Profile (27 iterations ) ===
Layer Time (ms) Avg. Time (ms) Time %
Conv_33 + Relu_34 input reformatter 0 6.77 0.25 0.2
Conv_33 + Relu_34 67.00 2.48 1.9
Conv_35 + Relu_36 122.07 4.52 3.5
MaxPool_37 15.03 0.56 0.4
Conv_38 + Relu_39 16.38 0.61 0.5
MaxPool_40 3.87 0.14 0.1
Conv_41 + Relu_42 4.50 0.17 0.1
MaxPool_43 1.14 0.04 0.0
Conv_44 + Relu_45 1.51 0.06 0.0
MaxPool_46 0.24 0.01 0.0
Conv_47 + Relu_48 0.64 0.02 0.0
MaxPool_49 0.17 0.01 0.0
Conv_50 + Relu_51 0.58 0.02 0.0
Conv_52 + Relu_53 0.48 0.02 0.0
onnx::Conv_824 copy 0.16 0.01 0.0
Conv_55 + Relu_56 0.48 0.02 0.0
Resize_65 input reformatter 0 0.16 0.01 0.0
Resize_65 0.55 0.02 0.0
onnx::Concat_846 copy 0.23 0.01 0.0
Conv_67 + Relu_68 0.71 0.03 0.0
Resize_77 input reformatter 0 0.19 0.01 0.0
Resize_77 1.73 0.06 0.0
onnx::Concat_865 copy 0.48 0.02 0.0
Conv_79 + Relu_80 2.08 0.08 0.1
Resize_89 input reformatter 0 0.33 0.01 0.0
Resize_89 6.08 0.23 0.2
onnx::Concat_884 copy 1.83 0.07 0.1
Conv_91 + Relu_92 7.24 0.27 0.2
Resize_101 input reformatter 0 1.51 0.06 0.0
Resize_101 23.22 0.86 0.7
onnx::Concat_903 copy 6.79 0.25 0.2
Conv_103 + Relu_104 30.16 1.12 0.9
Resize_113 input reformatter 0 6.07 0.22 0.2
Resize_113 91.54 3.39 2.6
onnx::Concat_922 copy 26.47 0.98 0.8
Conv_115 + Relu_116 127.60 4.73 3.6
Add_117 67.54 2.50 1.9
MaxPool_118 28.71 1.06 0.8
Conv_119 + Relu_120 49.48 1.83 1.4
Conv_121 + Relu_122 55.16 2.04 1.6
MaxPool_123 3.83 0.14 0.1
Conv_124 + Relu_125 4.42 0.16 0.1
MaxPool_126 1.14 0.04 0.0
Conv_127 + Relu_128 1.49 0.06 0.0
MaxPool_129 0.24 0.01 0.0
Conv_130 + Relu_131 0.63 0.02 0.0
MaxPool_132 0.17 0.01 0.0
Conv_133 + Relu_134 0.57 0.02 0.0
Conv_135 + Relu_136 0.48 0.02 0.0
onnx::Conv_950 copy 0.16 0.01 0.0
Conv_138 + Relu_139 0.48 0.02 0.0
Resize_148 input reformatter 0 0.15 0.01 0.0
Resize_148 0.56 0.02 0.0
onnx::Concat_972 copy 0.22 0.01 0.0
Conv_150 + Relu_151 0.69 0.03 0.0
Resize_160 input reformatter 0 0.18 0.01 0.0
Resize_160 1.69 0.06 0.0
onnx::Concat_991 copy 0.47 0.02 0.0
Conv_162 + Relu_163 2.04 0.08 0.1
Resize_172 input reformatter 0 0.33 0.01 0.0
Resize_172 5.97 0.22 0.2
onnx::Concat_1010 copy 1.82 0.07 0.1
Conv_174 + Relu_175 7.11 0.26 0.2
Resize_184 input reformatter 0 1.51 0.06 0.0
Resize_184 22.91 0.85 0.7
onnx::Concat_1029 copy 6.74 0.25 0.2
Conv_186 + Relu_187 49.31 1.83 1.4
Add_188 33.69 1.25 1.0
MaxPool_189 14.30 0.53 0.4
Conv_190 + Relu_191 46.14 1.71 1.3
Conv_192 + Relu_193 27.09 1.00 0.8
MaxPool_194 1.96 0.07 0.1
Conv_195 + Relu_196 2.80 0.10 0.1
MaxPool_197 0.42 0.02 0.0
Conv_198 + Relu_199 0.95 0.04 0.0
MaxPool_200 0.18 0.01 0.0
Conv_201 + Relu_202 0.62 0.02 0.0
Conv_203 + Relu_204 0.56 0.02 0.0
onnx::Conv_1053 copy 0.19 0.01 0.0
Conv_206 + Relu_207 0.84 0.03 0.0
Resize_216 input reformatter 0 0.17 0.01 0.0
Resize_216 0.96 0.04 0.0
onnx::Concat_1075 copy 0.33 0.01 0.0
Conv_218 + Relu_219 1.52 0.06 0.0
Resize_228 input reformatter 0 0.26 0.01 0.0
Resize_228 3.23 0.12 0.1
onnx::Concat_1094 copy 1.02 0.04 0.0
Conv_230 + Relu_231 4.62 0.17 0.1
Resize_240 input reformatter 0 0.85 0.03 0.0
Resize_240 11.74 0.43 0.3
onnx::Concat_1113 copy 3.91 0.14 0.1
Conv_242 + Relu_243 46.52 1.72 1.3
Add_244 16.60 0.61 0.5
MaxPool_245 7.24 0.27 0.2
Conv_246 + Relu_247 43.88 1.63 1.2
Conv_248 + Relu_249 22.78 0.84 0.6
MaxPool_250 1.02 0.04 0.0
Conv_251 + Relu_252 2.52 0.09 0.1
MaxPool_253 0.22 0.01 0.0
Conv_254 + Relu_255 1.16 0.04 0.0
Conv_256 + Relu_257 1.18 0.04 0.0
onnx::Conv_1133 copy 0.24 0.01 0.0
Conv_259 + Relu_260 1.94 0.07 0.1
Resize_269 input reformatter 0 0.22 0.01 0.0
Resize_269 1.76 0.07 0.1
onnx::Concat_1155 copy 0.48 0.02 0.0
Conv_271 + Relu_272 4.60 0.17 0.1
Resize_281 input reformatter 0 0.37 0.01 0.0
Resize_281 6.17 0.23 0.2
onnx::Concat_1174 copy 2.08 0.08 0.1
Conv_283 + Relu_284 44.42 1.65 1.3
Add_285 8.36 0.31 0.2
MaxPool_286 3.59 0.13 0.1
Conv_287 + Relu_288 22.85 0.85 0.6
Conv_289 + Relu_290 13.60 0.50 0.4
Conv_291 + Relu_292 7.14 0.26 0.2
Conv_293 + Relu_294 7.13 0.26 0.2
Conv_295 + Relu_296 7.38 0.27 0.2
onnx::Conv_1192 copy 0.98 0.04 0.0
Conv_298 + Relu_299 13.65 0.51 0.4
Conv_301 + Relu_302 13.66 0.51 0.4
Conv_304 + Relu_305 23.03 0.85 0.7
Add_306 2.16 0.08 0.1
MaxPool_307 1.05 0.04 0.0
Conv_308 + Relu_309 8.94 0.33 0.3
Conv_310 + Relu_311 4.95 0.18 0.1
Conv_312 + Relu_313 2.67 0.10 0.1
Conv_314 + Relu_315 2.66 0.10 0.1
Conv_316 + Relu_317 2.80 0.10 0.1
onnx::Conv_1221 copy 0.34 0.01 0.0
Conv_319 + Relu_320 4.97 0.18 0.1
Conv_322 + Relu_323 4.97 0.18 0.1
Conv_325 + Relu_326 8.96 0.33 0.3
Add_327 0.56 0.02 0.0
Resize_336 input reformatter 0 0.39 0.01 0.0
Resize_336 6.31 0.23 0.2
Conv_689 1.96 0.07 0.1
Resize_698 input reformatter 0 0.15 0.01 0.0
Resize_698 2.97 0.11 0.1
onnx::Concat_1252 copy 2.12 0.08 0.1
Conv_338 + Relu_339 46.50 1.72 1.3
Conv_340 + Relu_341 13.84 0.51 0.4
Conv_342 + Relu_343 7.24 0.27 0.2
Conv_344 + Relu_345 7.23 0.27 0.2
Conv_346 + Relu_347 7.43 0.28 0.2
onnx::Conv_1265 copy 0.99 0.04 0.0
Conv_349 + Relu_350 13.90 0.51 0.4
Conv_352 + Relu_353 13.90 0.51 0.4
Conv_355 + Relu_356 23.55 0.87 0.7
Add_357 2.12 0.08 0.1
Resize_366 input reformatter 0 1.79 0.07 0.1
Resize_366 24.80 0.92 0.7
Conv_679 3.65 0.14 0.1
Resize_688 input reformatter 0 0.16 0.01 0.0
Resize_688 3.38 0.13 0.1
onnx::Concat_1296 copy 8.07 0.30 0.2
Conv_368 + Relu_369 88.55 3.28 2.5
Conv_370 + Relu_371 12.66 0.47 0.4
MaxPool_372 1.09 0.04 0.0
Conv_373 + Relu_374 2.68 0.10 0.1
MaxPool_375 0.24 0.01 0.0
Conv_376 + Relu_377 1.22 0.05 0.0
Conv_378 + Relu_379 1.26 0.05 0.0
onnx::Conv_1311 copy 0.25 0.01 0.0
Conv_381 + Relu_382 2.06 0.08 0.1
Resize_391 input reformatter 0 0.22 0.01 0.0
Resize_391 1.87 0.07 0.1
onnx::Concat_1333 copy 0.50 0.02 0.0
Conv_393 + Relu_394 4.89 0.18 0.1
Resize_403 input reformatter 0 0.40 0.01 0.0
Resize_403 6.50 0.24 0.2
onnx::Concat_1352 copy 2.17 0.08 0.1
Conv_405 + Relu_406 24.12 0.89 0.7
Add_407 4.23 0.16 0.1
Resize_416 input reformatter 0 3.23 0.12 0.1
Resize_416 50.20 1.86 1.4
Conv_669 6.68 0.25 0.2
Resize_678 input reformatter 0 0.20 0.01 0.0
Resize_678 3.60 0.13 0.1
onnx::Concat_1372 copy 16.29 0.60 0.5
Conv_418 + Relu_419 92.65 3.43 2.6
Conv_420 + Relu_421 16.30 0.60 0.5
MaxPool_422 1.99 0.07 0.1
Conv_423 + Relu_424 3.01 0.11 0.1
MaxPool_425 0.43 0.02 0.0
Conv_426 + Relu_427 1.03 0.04 0.0
MaxPool_428 0.19 0.01 0.0
Conv_429 + Relu_430 0.65 0.02 0.0
Conv_431 + Relu_432 0.62 0.02 0.0
onnx::Conv_1391 copy 0.20 0.01 0.0
Conv_434 + Relu_435 0.91 0.03 0.0
Resize_444 input reformatter 0 0.18 0.01 0.0
Resize_444 1.04 0.04 0.0
onnx::Concat_1413 copy 0.35 0.01 0.0
Conv_446 + Relu_447 1.67 0.06 0.0
Resize_456 input reformatter 0 0.28 0.01 0.0
Resize_456 3.54 0.13 0.1
onnx::Concat_1432 copy 1.10 0.04 0.0
Conv_458 + Relu_459 4.94 0.18 0.1
Resize_468 input reformatter 0 1.34 0.05 0.0
Resize_468 13.50 0.50 0.4
onnx::Concat_1451 copy 4.67 0.17 0.1
Conv_470 + Relu_471 25.71 0.95 0.7
Add_472 8.24 0.31 0.2
Resize_481 input reformatter 0 5.99 0.22 0.2
Resize_481 99.84 3.70 2.8
Conv_659 13.06 0.48 0.4
Resize_668 input reformatter 0 0.32 0.01 0.0
Resize_668 3.55 0.13 0.1
onnx::Concat_1471 copy 32.51 1.20 0.9
Conv_483 + Relu_484 113.52 4.20 3.2
Conv_485 + Relu_486 30.13 1.12 0.9
MaxPool_487 3.88 0.14 0.1
Conv_488 + Relu_489 4.84 0.18 0.1
MaxPool_490 1.14 0.04 0.0
Conv_491 + Relu_492 input reformatter 0 0.39 0.01 0.0
Conv_491 + Relu_492 1.65 0.06 0.0
MaxPool_493 0.31 0.01 0.0
Conv_494 + Relu_495 0.69 0.03 0.0
MaxPool_496 0.19 0.01 0.0
Conv_497 + Relu_498 0.38 0.01 0.0
Conv_499 + Relu_500 input reformatter 0 0.17 0.01 0.0
Conv_499 + Relu_500 0.62 0.02 0.0
onnx::Conv_1494 copy 0.16 0.01 0.0
Conv_502 + Relu_503 0.51 0.02 0.0
Resize_512 input reformatter 0 0.16 0.01 0.0
Resize_512 0.59 0.02 0.0
onnx::Concat_1516 copy 0.26 0.01 0.0
Conv_514 + Relu_515 1.03 0.04 0.0
Resize_524 input reformatter 0 0.24 0.01 0.0
Resize_524 1.78 0.07 0.1
onnx::Concat_1535 copy 0.76 0.03 0.0
Conv_526 + Relu_527 2.45 0.09 0.1
Resize_536 input reformatter 0 0.70 0.03 0.0
Resize_536 6.57 0.24 0.2
onnx::Concat_1554 copy 1.86 0.07 0.1
Conv_538 + Relu_539 7.79 0.29 0.2
Resize_548 input reformatter 0 1.50 0.06 0.0
Resize_548 24.81 0.92 0.7
onnx::Concat_1573 copy 6.94 0.26 0.2
Conv_550 + Relu_551 35.40 1.31 1.0
Add_552 16.39 0.61 0.5
Resize_561 input reformatter 0 11.44 0.42 0.3
Resize_561 195.07 7.22 5.5
Conv_649 26.61 0.99 0.8
Resize_658 input reformatter 0 0.90 0.03 0.0
Resize_658 3.19 0.12 0.1
onnx::Concat_1593 copy 61.75 2.29 1.8
Conv_563 + Relu_564 233.66 8.65 6.6
Conv_565 + Relu_566 116.17 4.30 3.3
MaxPool_567 13.05 0.48 0.4
Conv_568 + Relu_569 16.59 0.61 0.5
MaxPool_570 3.42 0.13 0.1
Conv_571 + Relu_572 input reformatter 0 0.68 0.03 0.0
Conv_571 + Relu_572 3.66 0.14 0.1
MaxPool_573 0.44 0.02 0.0
Conv_574 + Relu_575 1.18 0.04 0.0
MaxPool_576 0.20 0.01 0.0
Conv_577 + Relu_578 0.52 0.02 0.0
MaxPool_579 0.16 0.01 0.0
Conv_580 + Relu_581 0.32 0.01 0.0
Conv_582 + Relu_583 0.59 0.02 0.0
onnx::Conv_1620 copy 0.14 0.01 0.0
Conv_585 + Relu_586 0.37 0.01 0.0
Resize_595 input reformatter 0 0.15 0.01 0.0
Resize_595 0.37 0.01 0.0
onnx::Concat_1642 copy 0.20 0.01 0.0
Conv_597 + Relu_598 0.69 0.03 0.0
Resize_607 input reformatter 0 0.19 0.01 0.0
Resize_607 0.94 0.03 0.0
onnx::Concat_1661 copy 0.35 0.01 0.0
Conv_609 + Relu_610 1.55 0.06 0.0
Resize_619 input reformatter 0 0.33 0.01 0.0
Resize_619 3.36 0.12 0.1
onnx::Concat_1680 copy 1.29 0.05 0.0
Conv_621 + Relu_622 5.91 0.22 0.2
Resize_631 input reformatter 0 1.10 0.04 0.0
Resize_631 12.05 0.45 0.3
onnx::Concat_1699 copy 3.42 0.13 0.1
Conv_633 + Relu_634 16.50 0.61 0.5
Resize_643 input reformatter 0 2.91 0.11 0.1
Resize_643 47.31 1.75 1.3
onnx::Concat_1718 copy 12.82 0.47 0.4
Conv_645 + Relu_646 68.96 2.55 2.0
Add_647 65.34 2.42 1.9
Conv_648 102.34 3.79 2.9
Conv_648 output reformatter 0 3.11 0.12 0.1
onnx::Concat_1740 copy 0.70 0.03 0.0
onnx::Concat_1756 copy 0.86 0.03 0.0
onnx::Concat_1772 copy 0.87 0.03 0.0
onnx::Concat_1788 copy 0.85 0.03 0.0
onnx::Concat_1804 copy 0.87 0.03 0.0
Conv_700 14.26 0.53 0.4
Sigmoid_701 1.11 0.04 0.0
Sigmoid_701 output reformatter 0 1.59 0.06 0.0
Total 3519.88 130.37 100.0

AakankshaS · May 29, 2023, 5:33am

Hi,

Request you to share the model, script, profiler, and performance output if not shared already so that we can help you better.

Alternatively, you can try running your model with trtexec command.

While measuring the model performance, make sure you consider the latency and throughput of the network inference, excluding the data pre and post-processing overhead.
Please refer to the below links for more details:

Thanks!