Weights are outside of fp16 range - Re-training SSD-Mobilenet

Autonomous Machines Jetson & Embedded Systems Jetson Nano
, ,
1

Hi,

I’m following the guide for Re-training SSD-Mobilenet and when I’m running the model with detectnet I’ve got this error:

[TRT]    ../builder/cudnnBuilderWeightConverters.cpp (541) - Misc Error in operator(): 1 (Weights are outside of fp16 range. A possible fix is to retrain the model with regularization to bring the magnitude of the weights down.)
[TRT]    Builder timing cache: created 65 entries, 20 hit(s)
[TRT]    ../builder/cudnnBuilderWeightConverters.cpp (541) - Misc Error in operator(): 1 (Weights are outside of fp16 range. A possible fix is to retrain the model with regularization to bring the magnitude of the weights down.)
[TRT]    device GPU, failed to build CUDA engine
[TRT]    device GPU, failed to load models/mask/ssd-mobilenet.onnx
[TRT]    detectNet -- failed to initialize.
detectnet:  failed to load detectNet model

I trainned the model with +800 images and 3 classes with for 30 epochs and the last .pth is:
mb1-ssd-Epoch-29-Loss-12491.12704326923.pth

Seems that the errors are a bit high but I wanted to try it anyway, Should I retrain it?

3

Hi,

Could you try the following command and share the output log with us first?

/usr/src/tensorrt/bin/trtexec --onnx=[your/model] --verbose

Thanks.

4

Sure @AastaLLL

Here is the last part:

[12/08/2020-09:51:25] [I] [TRT] Detected 1 inputs and 4 output network tensors.
[12/08/2020-09:51:25] [V] [TRT] Conv_0 + Relu_2 (scudnn) Set Tactic Name: maxwell_scudnn_128x32_relu_small_nn_v1
[12/08/2020-09:51:25] [V] [TRT] Conv_6 + Relu_8 (scudnn) Set Tactic Name: maxwell_scudnn_128x64_relu_interior_nn_v1
[12/08/2020-09:51:25] [V] [TRT] Conv_12 + Relu_14 (scudnn) Set Tactic Name: maxwell_scudnn_128x64_relu_interior_nn_v1
[12/08/2020-09:51:25] [V] [TRT] Conv_18 + Relu_20 (scudnn) Set Tactic Name: maxwell_scudnn_128x64_relu_interior_nn_v1
[12/08/2020-09:51:25] [V] [TRT] Conv_24 + Relu_26 (scudnn) Set Tactic Name: maxwell_scudnn_128x64_relu_interior_nn_v1
[12/08/2020-09:51:25] [V] [TRT] Conv_30 + Relu_32 (scudnn) Set Tactic Name: maxwell_scudnn_128x64_relu_interior_nn_v1
[12/08/2020-09:51:25] [V] [TRT] Conv_36 + Relu_38 (scudnn) Set Tactic Name: maxwell_scudnn_128x64_relu_interior_nn_v1
[12/08/2020-09:51:25] [V] [TRT] Conv_42 + Relu_44 (scudnn) Set Tactic Name: maxwell_scudnn_128x128_relu_interior_nn_v1
[12/08/2020-09:51:25] [V] [TRT] Conv_48 + Relu_50 (scudnn) Set Tactic Name: maxwell_scudnn_128x128_relu_interior_nn_v1
[12/08/2020-09:51:25] [V] [TRT] Conv_54 + Relu_56 (scudnn) Set Tactic Name: maxwell_scudnn_128x128_relu_interior_nn_v1
[12/08/2020-09:51:25] [V] [TRT] Conv_60 + Relu_62 (scudnn) Set Tactic Name: maxwell_scudnn_128x128_relu_interior_nn_v1
[12/08/2020-09:51:25] [V] [TRT] Conv_66 + Relu_68 (scudnn) Set Tactic Name: maxwell_scudnn_128x128_relu_interior_nn_v1
[12/08/2020-09:51:25] [V] [TRT] Conv_77 || Conv_69 (scudnn) Set Tactic Name: maxwell_scudnn_128x64_relu_small_nn_v1
[12/08/2020-09:51:25] [V] [TRT] Conv_88 + Relu_90 (scudnn) Set Tactic Name: maxwell_scudnn_128x128_relu_interior_nn_v1
[12/08/2020-09:51:25] [V] [TRT] Conv_94 + Relu_96 (scudnn) Set Tactic Name: maxwell_scudnn_128x64_relu_interior_nn_v1
[12/08/2020-09:51:25] [V] [TRT] Conv_113 + Relu_114 (scudnn) Set Tactic Name: maxwell_scudnn_128x128_relu_interior_nn_v1
[12/08/2020-09:51:25] [V] [TRT] Conv_105 || Conv_97 (scudnn_winograd) Set Tactic Name: maxwell_scudnn_winograd_128x128_ldg1_ldg4_mobile_relu_tile148t_nt_v0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_0 + Relu_2 Weights: 0 HostPersistent: 1664 DevicePersistent: 138752
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_3 + Relu_5 Weights: 1152 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_6 + Relu_8 Weights: 0 HostPersistent: 3200 DevicePersistent: 143872
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_9 + Relu_11 Weights: 2304 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_12 + Relu_14 Weights: 0 HostPersistent: 3200 DevicePersistent: 67072
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_15 + Relu_17 Weights: 4608 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_18 + Relu_20 Weights: 0 HostPersistent: 3200 DevicePersistent: 99840
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_21 + Relu_23 Weights: 4608 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_24 + Relu_26 Weights: 0 HostPersistent: 3200 DevicePersistent: 140800
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_27 + Relu_29 Weights: 9216 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_30 + Relu_32 Weights: 0 HostPersistent: 3200 DevicePersistent: 271872
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_33 + Relu_35 Weights: 9216 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_36 + Relu_38 Weights: 0 HostPersistent: 3200 DevicePersistent: 528896
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_39 + Relu_41 Weights: 18432 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_42 + Relu_44 Weights: 0 HostPersistent: 3200 DevicePersistent: 1053184
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_45 + Relu_47 Weights: 18432 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_48 + Relu_50 Weights: 0 HostPersistent: 3200 DevicePersistent: 1053184
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_51 + Relu_53 Weights: 18432 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_54 + Relu_56 Weights: 0 HostPersistent: 3200 DevicePersistent: 1053184
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_57 + Relu_59 Weights: 18432 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_60 + Relu_62 Weights: 0 HostPersistent: 3200 DevicePersistent: 1053184
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_63 + Relu_65 Weights: 18432 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_66 + Relu_68 Weights: 0 HostPersistent: 3200 DevicePersistent: 1053184
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_85 + Relu_87 Weights: 18432 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_77 || Conv_69 Weights: 0 HostPersistent: 1664 DevicePersistent: 887296
[12/08/2020-09:51:25] [V] [TRT] Layer: Transpose_78 + Reshape_84 Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Transpose_70 + Reshape_76 Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_88 + Relu_90 Weights: 0 HostPersistent: 3200 DevicePersistent: 2102272
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_91 + Relu_93 Weights: 36864 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_94 + Relu_96 Weights: 0 HostPersistent: 3200 DevicePersistent: 4199424
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_113 + Relu_114 Weights: 0 HostPersistent: 3200 DevicePersistent: 1050624
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_105 || Conv_97 Weights: 0 HostPersistent: 512 DevicePersistent: 5964288
[12/08/2020-09:51:25] [V] [TRT] Layer: Transpose_106 + Reshape_112 Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Transpose_98 + Reshape_104 Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_115 + Relu_116 Weights: 4718592 HostPersistent: 8 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_133 + Relu_134 Weights: 262144 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_125 || Conv_117 Weights: 884736 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Transpose_126 + Reshape_132 Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Transpose_118 + Reshape_124 Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_135 + Relu_136 Weights: 1179648 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_153 + Relu_154 Weights: 131072 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_145 || Conv_137 Weights: 442368 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Transpose_146 + Reshape_152 Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Transpose_138 + Reshape_144 Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_155 + Relu_156 Weights: 1179648 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_173 + Relu_174 Weights: 131072 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_165 || Conv_157 Weights: 442368 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Transpose_166 + Reshape_172 Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Transpose_158 + Reshape_164 Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_175 + Relu_176 Weights: 1179648 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Conv_185 || Conv_177 Weights: 442368 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Transpose_186 + Reshape_192 Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: 295 copy Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: 331 copy Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: 359 copy Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: 387 copy Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: 415 copy Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: 443 copy Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Slice_203 Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Slice_196 Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Transpose_178 + Reshape_184 Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: 283 copy Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: 319 copy Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: 347 copy Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: 375 copy Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: 403 copy Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: 431 copy Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Softmax_195 Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: (Unnamed Layer* 230) [Shuffle] Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: (Unnamed Layer* 260) [Constant] Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: (Unnamed Layer* 293) [Constant] Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: PWN(PWN(PWN((Unnamed Layer* 289) [Constant] + (Unnamed Layer* 290) [Shuffle], Mul_205), Exp_206), Mul_208) Weights: 0 HostPersistent: 372 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: (Unnamed Layer* 262) [Constant] Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: PWN(PWN(PWN((Unnamed Layer* 257) [Constant] + (Unnamed Layer* 258) [Shuffle], Mul_198), Mul_200), Add_202) Weights: 0 HostPersistent: 468 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: 453 copy Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: 459 copy Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Slice_216 Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Slice_215 Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Slice_211 Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: Slice_210 Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: PWN(PWN((Unnamed Layer* 400) [Constant] + (Unnamed Layer* 401) [Shuffle], Div_218), Add_219) Weights: 0 HostPersistent: 372 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: PWN(PWN((Unnamed Layer* 346) [Constant] + (Unnamed Layer* 347) [Shuffle], Div_213), Sub_214) Weights: 0 HostPersistent: 372 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: 465 copy Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Layer: 470 copy Weights: 0 HostPersistent: 0 DevicePersistent: 0
[12/08/2020-09:51:25] [V] [TRT] Total Host Persistent Memory: 50232
[12/08/2020-09:51:25] [V] [TRT] Total Device Persistent Memory: 20860928
[12/08/2020-09:51:25] [V] [TRT] Total Weight Memory: 11172224
[12/08/2020-09:51:25] [V] [TRT] Builder timing cache: created 33 entries, 8 hit(s)
[12/08/2020-09:51:25] [V] [TRT] Engine generation completed in 176.2 seconds.
[12/08/2020-09:51:25] [V] [TRT] Engine Layer Information:
[12/08/2020-09:51:25] [V] [TRT] Layer(scudnn): Conv_0 + Relu_2, Tactic: -3456450830548107839, input_0[Float(3,300,300)] -> 205[Float(32,150,150)]
[12/08/2020-09:51:25] [V] [TRT] Layer(DepthwiseConvolution): Conv_3 + Relu_5, Tactic: -1, 205[Float(32,150,150)] -> 208[Float(32,150,150)]
[12/08/2020-09:51:25] [V] [TRT] Layer(scudnn): Conv_6 + Relu_8, Tactic: -37215280111360163, 208[Float(32,150,150)] -> 211[Float(64,150,150)]
[12/08/2020-09:51:25] [V] [TRT] Layer(DepthwiseConvolution): Conv_9 + Relu_11, Tactic: -1, 211[Float(64,150,150)] -> 214[Float(64,75,75)]
[12/08/2020-09:51:25] [V] [TRT] Layer(scudnn): Conv_12 + Relu_14, Tactic: -37215280111360163, 214[Float(64,75,75)] -> 217[Float(128,75,75)]
[12/08/2020-09:51:25] [V] [TRT] Layer(DepthwiseConvolution): Conv_15 + Relu_17, Tactic: -1, 217[Float(128,75,75)] -> 220[Float(128,75,75)]
[12/08/2020-09:51:25] [V] [TRT] Layer(scudnn): Conv_18 + Relu_20, Tactic: -37215280111360163, 220[Float(128,75,75)] -> 223[Float(128,75,75)]
[12/08/2020-09:51:25] [V] [TRT] Layer(DepthwiseConvolution): Conv_21 + Relu_23, Tactic: -1, 223[Float(128,75,75)] -> 226[Float(128,38,38)]
[12/08/2020-09:51:25] [V] [TRT] Layer(scudnn): Conv_24 + Relu_26, Tactic: -37215280111360163, 226[Float(128,38,38)] -> 229[Float(256,38,38)]
[12/08/2020-09:51:25] [V] [TRT] Layer(DepthwiseConvolution): Conv_27 + Relu_29, Tactic: -1, 229[Float(256,38,38)] -> 232[Float(256,38,38)]
[12/08/2020-09:51:25] [V] [TRT] Layer(scudnn): Conv_30 + Relu_32, Tactic: -37215280111360163, 232[Float(256,38,38)] -> 235[Float(256,38,38)]
[12/08/2020-09:51:25] [V] [TRT] Layer(DepthwiseConvolution): Conv_33 + Relu_35, Tactic: -1, 235[Float(256,38,38)] -> 238[Float(256,19,19)]
[12/08/2020-09:51:25] [V] [TRT] Layer(scudnn): Conv_36 + Relu_38, Tactic: -37215280111360163, 238[Float(256,19,19)] -> 241[Float(512,19,19)]
[12/08/2020-09:51:25] [V] [TRT] Layer(DepthwiseConvolution): Conv_39 + Relu_41, Tactic: -1, 241[Float(512,19,19)] -> 244[Float(512,19,19)]
[12/08/2020-09:51:25] [V] [TRT] Layer(scudnn): Conv_42 + Relu_44, Tactic: 5326823351883942011, 244[Float(512,19,19)] -> 247[Float(512,19,19)]
[12/08/2020-09:51:25] [V] [TRT] Layer(DepthwiseConvolution): Conv_45 + Relu_47, Tactic: -1, 247[Float(512,19,19)] -> 250[Float(512,19,19)]
[12/08/2020-09:51:25] [V] [TRT] Layer(scudnn): Conv_48 + Relu_50, Tactic: 5326823351883942011, 250[Float(512,19,19)] -> 253[Float(512,19,19)]
[12/08/2020-09:51:25] [V] [TRT] Layer(DepthwiseConvolution): Conv_51 + Relu_53, Tactic: -1, 253[Float(512,19,19)] -> 256[Float(512,19,19)]
[12/08/2020-09:51:25] [V] [TRT] Layer(scudnn): Conv_54 + Relu_56, Tactic: 5326823351883942011, 256[Float(512,19,19)] -> 259[Float(512,19,19)]
[12/08/2020-09:51:25] [V] [TRT] Layer(DepthwiseConvolution): Conv_57 + Relu_59, Tactic: -1, 259[Float(512,19,19)] -> 262[Float(512,19,19)]
[12/08/2020-09:51:25] [V] [TRT] Layer(scudnn): Conv_60 + Relu_62, Tactic: 5326823351883942011, 262[Float(512,19,19)] -> 265[Float(512,19,19)]
[12/08/2020-09:51:25] [V] [TRT] Layer(DepthwiseConvolution): Conv_63 + Relu_65, Tactic: -1, 265[Float(512,19,19)] -> 268[Float(512,19,19)]
[12/08/2020-09:51:25] [V] [TRT] Layer(scudnn): Conv_66 + Relu_68, Tactic: 5326823351883942011, 268[Float(512,19,19)] -> 271[Float(512,19,19)]
[12/08/2020-09:51:25] [V] [TRT] Layer(DepthwiseConvolution): Conv_85 + Relu_87, Tactic: -1, 271[Float(512,19,19)] -> 298[Float(512,10,10)]
[12/08/2020-09:51:25] [V] [TRT] Layer(scudnn): Conv_77 || Conv_69, Tactic: 5137655947464784826, 271[Float(512,19,19)] -> Conv_77 || Conv_69[Float(48,19,19)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Shuffle): Transpose_78 + Reshape_84, Tactic: 0, Conv_77 || Conv_69[Float(24,19,19)] -> 295[Float(2166,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Shuffle): Transpose_70 + Reshape_76, Tactic: 0, Conv_77 || Conv_69[Float(24,19,19)] -> 283[Float(2166,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(scudnn): Conv_88 + Relu_90, Tactic: 5326823351883942011, 298[Float(512,10,10)] -> 301[Float(1024,10,10)]
[12/08/2020-09:51:25] [V] [TRT] Layer(DepthwiseConvolution): Conv_91 + Relu_93, Tactic: -1, 301[Float(1024,10,10)] -> 304[Float(1024,10,10)]
[12/08/2020-09:51:25] [V] [TRT] Layer(scudnn): Conv_94 + Relu_96, Tactic: -37215280111360163, 304[Float(1024,10,10)] -> 307[Float(1024,10,10)]
[12/08/2020-09:51:25] [V] [TRT] Layer(scudnn): Conv_113 + Relu_114, Tactic: 5326823351883942011, 307[Float(1024,10,10)] -> 333[Float(256,10,10)]
[12/08/2020-09:51:25] [V] [TRT] Layer(scudnn_winograd): Conv_105 || Conv_97, Tactic: -1343271414618805657, 307[Float(1024,10,10)] -> Conv_105 || Conv_97[Float(48,10,10)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Shuffle): Transpose_106 + Reshape_112, Tactic: 0, Conv_105 || Conv_97[Float(24,10,10)] -> 331[Float(600,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Shuffle): Transpose_98 + Reshape_104, Tactic: 0, Conv_105 || Conv_97[Float(24,10,10)] -> 319[Float(600,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Convolution): Conv_115 + Relu_116, Tactic: 2, 333[Float(256,10,10)] -> 335[Float(512,5,5)]
[12/08/2020-09:51:25] [V] [TRT] Layer(FusedConvActDirect): Conv_133 + Relu_134, Tactic: 3997695, 335[Float(512,5,5)] -> 361[Float(128,5,5)]
[12/08/2020-09:51:25] [V] [TRT] Layer(FusedConvActDirect): Conv_125 || Conv_117, Tactic: 1769471, 335[Float(512,5,5)] -> Conv_125 || Conv_117[Float(48,5,5)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Shuffle): Transpose_126 + Reshape_132, Tactic: 0, Conv_125 || Conv_117[Float(24,5,5)] -> 359[Float(150,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Shuffle): Transpose_118 + Reshape_124, Tactic: 0, Conv_125 || Conv_117[Float(24,5,5)] -> 347[Float(150,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(FusedConvActDirect): Conv_135 + Relu_136, Tactic: 458751, 361[Float(128,5,5)] -> 363[Float(256,3,3)]
[12/08/2020-09:51:25] [V] [TRT] Layer(FusedConvActDirect): Conv_153 + Relu_154, Tactic: 5177343, 363[Float(256,3,3)] -> 389[Float(128,3,3)]
[12/08/2020-09:51:25] [V] [TRT] Layer(FusedConvActDirect): Conv_145 || Conv_137, Tactic: 7274495, 363[Float(256,3,3)] -> Conv_145 || Conv_137[Float(48,3,3)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Shuffle): Transpose_146 + Reshape_152, Tactic: 0, Conv_145 || Conv_137[Float(24,3,3)] -> 387[Float(54,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Shuffle): Transpose_138 + Reshape_144, Tactic: 0, Conv_145 || Conv_137[Float(24,3,3)] -> 375[Float(54,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(FusedConvActDirect): Conv_155 + Relu_156, Tactic: 458751, 389[Float(128,3,3)] -> 391[Float(256,2,2)]
[12/08/2020-09:51:25] [V] [TRT] Layer(FusedConvActDirect): Conv_173 + Relu_174, Tactic: 5177343, 391[Float(256,2,2)] -> 417[Float(128,2,2)]
[12/08/2020-09:51:25] [V] [TRT] Layer(FusedConvActDirect): Conv_165 || Conv_157, Tactic: 7274495, 391[Float(256,2,2)] -> Conv_165 || Conv_157[Float(48,2,2)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Shuffle): Transpose_166 + Reshape_172, Tactic: 0, Conv_165 || Conv_157[Float(24,2,2)] -> 415[Float(24,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Shuffle): Transpose_158 + Reshape_164, Tactic: 0, Conv_165 || Conv_157[Float(24,2,2)] -> 403[Float(24,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(FusedConvActDirect): Conv_175 + Relu_176, Tactic: 458751, 417[Float(128,2,2)] -> 419[Float(256,1,1)]
[12/08/2020-09:51:25] [V] [TRT] Layer(FusedConvActDirect): Conv_185 || Conv_177, Tactic: 7274495, 419[Float(256,1,1)] -> Conv_185 || Conv_177[Float(48,1,1)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Shuffle): Transpose_186 + Reshape_192, Tactic: 0, Conv_185 || Conv_177[Float(24,1,1)] -> 443[Float(6,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Reformat): 295 copy, Tactic: 1002, 295[Float(2166,4)] -> 445[Float(2166,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Reformat): 331 copy, Tactic: 1002, 331[Float(600,4)] -> 445[Float(600,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Reformat): 359 copy, Tactic: 0, 359[Float(150,4)] -> 445[Float(150,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Reformat): 387 copy, Tactic: 0, 387[Float(54,4)] -> 445[Float(54,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Reformat): 415 copy, Tactic: 0, 415[Float(24,4)] -> 445[Float(24,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Reformat): 443 copy, Tactic: 0, 443[Float(6,4)] -> 445[Float(6,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Reformat): Slice_203, Tactic: 0, 445[Float(3000,2)] -> 454[Float(3000,2)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Reformat): Slice_196, Tactic: 0, 445[Float(3000,2)] -> 447[Float(3000,2)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Shuffle): Transpose_178 + Reshape_184, Tactic: 0, Conv_185 || Conv_177[Float(24,1,1)] -> 431[Float(6,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Reformat): 283 copy, Tactic: 1002, 283[Float(2166,4)] -> 444[Float(2166,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Reformat): 319 copy, Tactic: 1002, 319[Float(600,4)] -> 444[Float(600,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Reformat): 347 copy, Tactic: 1002, 347[Float(150,4)] -> 444[Float(150,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Reformat): 375 copy, Tactic: 0, 375[Float(54,4)] -> 444[Float(54,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Reformat): 403 copy, Tactic: 0, 403[Float(24,4)] -> 444[Float(24,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Reformat): 431 copy, Tactic: 0, 431[Float(6,4)] -> 444[Float(6,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(ExtSoftMax): Softmax_195, Tactic: 0, (Unnamed Layer* 228) [Shuffle]_output[Float(4)] -> (Unnamed Layer* 229) [Softmax]_output[Float(4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Shuffle): (Unnamed Layer* 230) [Shuffle], Tactic: 0, (Unnamed Layer* 229) [Softmax]_output[Float(4)] -> scores[Float(3000,4)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Constant): (Unnamed Layer* 260) [Constant], Tactic: 0,  -> (Unnamed Layer* 260) [Constant]_output[Float(3000,2)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Constant): (Unnamed Layer* 293) [Constant], Tactic: 0,  -> (Unnamed Layer* 293) [Constant]_output[Float(3000,2)]
[12/08/2020-09:51:25] [V] [TRT] Layer(PointWiseV2): PWN(PWN(PWN((Unnamed Layer* 289) [Constant] + (Unnamed Layer* 290) [Shuffle], Mul_205), Exp_206), Mul_208), Tactic: 7, 454[Float(3000,2)], (Unnamed Layer* 293) [Constant]_output[Float(3000,2)] -> 459[Float(3000,2)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Constant): (Unnamed Layer* 262) [Constant], Tactic: 0,  -> (Unnamed Layer* 262) [Constant]_output[Float(3000,2)]
[12/08/2020-09:51:25] [V] [TRT] Layer(PointWiseV2): PWN(PWN(PWN((Unnamed Layer* 257) [Constant] + (Unnamed Layer* 258) [Shuffle], Mul_198), Mul_200), Add_202), Tactic: 8, 447[Float(3000,2)], (Unnamed Layer* 260) [Constant]_output[Float(3000,2)], (Unnamed Layer* 262) [Constant]_output[Float(3000,2)] -> 453[Float(3000,2)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Reformat): 453 copy, Tactic: 0, 453[Float(3000,2)] -> 460[Float(3000,2)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Reformat): 459 copy, Tactic: 0, 459[Float(3000,2)] -> 460[Float(3000,2)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Reformat): Slice_216, Tactic: 0, 460[Float(3000,2)] -> 467[Float(3000,2)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Reformat): Slice_215, Tactic: 0, 460[Float(3000,2)] -> 466[Float(3000,2)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Reformat): Slice_211, Tactic: 0, 460[Float(3000,2)] -> 462[Float(3000,2)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Reformat): Slice_210, Tactic: 0, 460[Float(3000,2)] -> 461[Float(3000,2)]
[12/08/2020-09:51:25] [V] [TRT] Layer(PointWiseV2): PWN(PWN((Unnamed Layer* 400) [Constant] + (Unnamed Layer* 401) [Shuffle], Div_218), Add_219), Tactic: 7, 467[Float(3000,2)], 466[Float(3000,2)] -> 470[Float(3000,2)]
[12/08/2020-09:51:25] [V] [TRT] Layer(PointWiseV2): PWN(PWN((Unnamed Layer* 346) [Constant] + (Unnamed Layer* 347) [Shuffle], Div_213), Sub_214), Tactic: 7, 462[Float(3000,2)], 461[Float(3000,2)] -> 465[Float(3000,2)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Reformat): 465 copy, Tactic: 0, 465[Float(3000,2)] -> boxes[Float(3000,2)]
[12/08/2020-09:51:25] [V] [TRT] Layer(Reformat): 470 copy, Tactic: 0, 470[Float(3000,2)] -> boxes[Float(3000,2)]
[12/08/2020-09:51:25] [I] Starting inference threads
[12/08/2020-09:51:29] [I] Warmup completed 3 queries over 200 ms
[12/08/2020-09:51:29] [I] Timing trace has 120 queries over 3.06717 s
[12/08/2020-09:51:29] [I] Trace averages of 10 runs:
[12/08/2020-09:51:29] [I] Average on 10 runs - GPU latency: 26.6585 ms - Host latency: 26.7821 ms (end to end 26.7952 ms, enqueue 8.15788 ms)
[12/08/2020-09:51:29] [I] Average on 10 runs - GPU latency: 25.306 ms - Host latency: 25.429 ms (end to end 25.442 ms, enqueue 7.00487 ms)
[12/08/2020-09:51:29] [I] Average on 10 runs - GPU latency: 25.3007 ms - Host latency: 25.4239 ms (end to end 25.4368 ms, enqueue 7.11261 ms)
[12/08/2020-09:51:29] [I] Average on 10 runs - GPU latency: 25.3029 ms - Host latency: 25.4263 ms (end to end 25.4393 ms, enqueue 7.21567 ms)
[12/08/2020-09:51:29] [I] Average on 10 runs - GPU latency: 25.3021 ms - Host latency: 25.4251 ms (end to end 25.4379 ms, enqueue 8.47745 ms)
[12/08/2020-09:51:29] [I] Average on 10 runs - GPU latency: 25.3211 ms - Host latency: 25.4446 ms (end to end 25.4575 ms, enqueue 7.81487 ms)
[12/08/2020-09:51:29] [I] Average on 10 runs - GPU latency: 25.3113 ms - Host latency: 25.4346 ms (end to end 25.4475 ms, enqueue 7.44973 ms)
[12/08/2020-09:51:29] [I] Average on 10 runs - GPU latency: 25.3116 ms - Host latency: 25.4343 ms (end to end 25.4473 ms, enqueue 7.08414 ms)
[12/08/2020-09:51:29] [I] Average on 10 runs - GPU latency: 25.3242 ms - Host latency: 25.448 ms (end to end 25.4612 ms, enqueue 6.47603 ms)
[12/08/2020-09:51:29] [I] Average on 10 runs - GPU latency: 25.3189 ms - Host latency: 25.442 ms (end to end 25.4549 ms, enqueue 6.84041 ms)
[12/08/2020-09:51:29] [I] Average on 10 runs - GPU latency: 25.3219 ms - Host latency: 25.446 ms (end to end 25.459 ms, enqueue 7.34497 ms)
[12/08/2020-09:51:29] [I] Average on 10 runs - GPU latency: 25.2956 ms - Host latency: 25.419 ms (end to end 25.4315 ms, enqueue 6.90645 ms)
[12/08/2020-09:51:29] [I] Host Latency
[12/08/2020-09:51:29] [I] min: 25.3474 ms (end to end 25.3599 ms)
[12/08/2020-09:51:29] [I] max: 29.9784 ms (end to end 29.9921 ms)
[12/08/2020-09:51:29] [I] mean: 25.5462 ms (end to end 25.5592 ms)
[12/08/2020-09:51:29] [I] median: 25.434 ms (end to end 25.4467 ms)
[12/08/2020-09:51:29] [I] percentile: 29.9427 ms at 99% (end to end 29.9566 ms at 99%)
[12/08/2020-09:51:29] [I] throughput: 39.124 qps
[12/08/2020-09:51:29] [I] walltime: 3.06717 s
[12/08/2020-09:51:29] [I] Enqueue Time
[12/08/2020-09:51:29] [I] min: 4.21875 ms
[12/08/2020-09:51:29] [I] max: 9.7052 ms
[12/08/2020-09:51:29] [I] median: 7.52998 ms
[12/08/2020-09:51:29] [I] GPU Compute
[12/08/2020-09:51:29] [I] min: 25.2242 ms
[12/08/2020-09:51:29] [I] max: 29.8511 ms
[12/08/2020-09:51:29] [I] mean: 25.4229 ms
[12/08/2020-09:51:29] [I] median: 25.3098 ms
[12/08/2020-09:51:29] [I] percentile: 29.8163 ms at 99%
[12/08/2020-09:51:29] [I] total compute time: 3.05075 s
&&&& PASSED TensorRT.trtexec # ./trtexec --onnx=ssd-mobilenet.onnx --verbose

Full log: https://pastebin.pl/view/raw/afe28e95

Regards

5

Hi,

Based on the log, your customized ssd-mobilenet.onnx model can run with TensorRT successfully.
This result indicates your model is good, and the error should occur in the detectnet app itself.

Since you have a customized model, please check if all the layer names, label files, and class numbers are updated accordingly.

Thanks.