Trtexec log problem and use DLA error on Jetson Xavier

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

Hi,

I have run my onnx model successfully using

./trtexec --onnx=/home/nvidia/APA_FS_Model.onnx --explicitBatch --verbose

[02/19/2021-11:22:18] [I] Starting inference threads
[02/19/2021-11:22:21] [I] Warmup completed 0 queries over 200 ms
[02/19/2021-11:22:21] [I] Timing trace has 0 queries over 3.00517 s
[02/19/2021-11:22:21] [I] Trace averages of 10 runs:
…
[02/19/2021-11:22:21] [I] Host Latency
[02/19/2021-11:22:21] [I] min: 1.92184 ms (end to end 1.92749 ms)
[02/19/2021-11:22:21] [I] max: 1.99609 ms (end to end 2.00391 ms)
[02/19/2021-11:22:21] [I] mean: 1.96004 ms (end to end 1.96797 ms)
[02/19/2021-11:22:21] [I] median: 1.9613 ms (end to end 1.96899 ms)
[02/19/2021-11:22:21] [I] percentile: 1.98669 ms at 99% (end to end 1.99561 ms at 99%)
[02/19/2021-11:22:21] [I] throughput: 0 qps
[02/19/2021-11:22:21] [I] walltime: 3.00517 s
[02/19/2021-11:22:21] [I] Enqueue Time
[02/19/2021-11:22:21] [I] min: 0.672119 ms
[02/19/2021-11:22:21] [I] max: 1.63806 ms
[02/19/2021-11:22:21] [I] median: 0.804443 ms
[02/19/2021-11:22:21] [I] GPU Compute
[02/19/2021-11:22:21] [I] min: 1.83517 ms
[02/19/2021-11:22:21] [I] max: 1.90869 ms
[02/19/2021-11:22:21] [I] mean: 1.87262 ms
[02/19/2021-11:22:21] [I] median: 1.87378 ms
[02/19/2021-11:22:21] [I] percentile: 1.89844 ms at 99%
[02/19/2021-11:22:21] [I] total compute time: 2.85949 s

What’s the difference between GPU Compute and Enqueue time ?
Which term is the running time for model execution?

Then I tried to use DLA to run trtexec using my onnx model with command below:

./trtexec --onnx=/home/nvidia/APA_FS_Model.onnx --explicitBatch --verbose --useDLACore=0 --allowGPUFallback --workspace=2048

It gives message :

[02/19/2021-11:10:13] [W] [TRT] Default DLA is enabled but layer (Unnamed Layer* 0) [Constant] is not supported on DLA, falling back to GPU.
[02/19/2021-11:10:13] [W] [TRT] Default DLA is enabled but layer (Unnamed Layer* 1) [Shuffle] is not supported on DLA, falling back to GPU.
[02/19/2021-11:10:13] [W] [TRT] Conv_26: DLA only supports prePadding < kernelSize. Detected: [4,4] < [3,3].
[02/19/2021-11:10:13] [W] [TRT] Conv_26: DLA only supports postPadding < kernelSize. Detected: [4,4] < [3,3].
[02/19/2021-11:10:13] [W] [TRT] Default DLA is enabled but layer Conv_26 is not supported on DLA, falling back to GPU.
[02/19/2021-11:10:13] [W] [TRT] Conv_34: DLA only supports prePadding < kernelSize. Detected: [4,4] < [3,3].
[02/19/2021-11:10:13] [W] [TRT] Conv_34: DLA only supports postPadding < kernelSize. Detected: [4,4] < [3,3].
[02/19/2021-11:10:13] [W] [TRT] Default DLA is enabled but layer Conv_34 is not supported on DLA, falling back to GPU.
[02/19/2021-11:10:13] [W] [TRT] ConvTranspose_49: DLA cores do not support more than 1 groups.
[02/19/2021-11:10:13] [W] [TRT] Default DLA is enabled but layer ConvTranspose_49 is not supported on DLA, falling back to GPU.
[02/19/2021-11:10:13] [W] [TRT] ConvTranspose_57: DLA cores do not support more than 1 groups.
[02/19/2021-11:10:13] [W] [TRT] Default DLA is enabled but layer ConvTranspose_57 is not supported on DLA, falling back to GPU.
[02/19/2021-11:10:13] [W] [TRT] ConvTranspose_60: DLA cores do not support more than 1 groups.
[02/19/2021-11:10:13] [W] [TRT] Default DLA is enabled but layer ConvTranspose_60 is not supported on DLA, falling back to GPU.
[02/19/2021-11:10:13] [V] [TRT] Applying generic optimizations to the graph for inference.
[02/19/2021-11:10:13] [V] [TRT] Original: 62 layers
[02/19/2021-11:10:13] [V] [TRT] After dead-layer removal: 62 layers
[02/19/2021-11:10:13] [W] [TRT] DLA allows only same dimensions inputs to Elementwise.
[02/19/2021-11:10:13] [W] [TRT] Internal DLA error for layer Add_1. Switching to GPU fallback.
[02/19/2021-11:10:13] [V] [TRT] After DLA optimization: 13 layers
[02/19/2021-11:10:13] [V] [TRT] Fusing (Unnamed Layer* 0) [Constant] with (Unnamed Layer* 1) [Shuffle]
[02/19/2021-11:10:13] [V] [TRT] After Myelin optimization: 12 layers
[02/19/2021-11:10:13] [V] [TRT] Fusing (Unnamed Layer* 0) [Constant] + (Unnamed Layer* 1) [Shuffle] with Add_1
[02/19/2021-11:10:13] [V] [TRT] After scale fusion: 11 layers
[02/19/2021-11:10:13] [V] [TRT] After vertical fusions: 11 layers
[02/19/2021-11:10:13] [V] [TRT] After final dead-layer removal: 11 layers
[02/19/2021-11:10:13] [V] [TRT] After tensor merging: 11 layers
[02/19/2021-11:10:13] [V] [TRT] After concat removal: 11 layers
[02/19/2021-11:10:13] [V] [TRT] Graph construction and optimization completed in 0.0485818 seconds.
[02/19/2021-11:10:13] [I] [TRT]
[02/19/2021-11:10:13] [I] [TRT] --------------- Layers running on DLA:
[02/19/2021-11:10:13] [I] [TRT] {Conv_2,Relu_3,Conv_4,Relu_5,Conv_6,Conv_7,Relu_8,Add_9,Conv_10,Relu_11,Conv_12,Conv_13,Relu_14,Add_15,Conv_16,Relu_17,Conv_18,Relu_19,Conv_20,Relu_21,Add_22,Conv_23,Relu_24,Add_25,Add_41,Add_42,Conv_50,Relu_51}, {Relu_27,Conv_28,Relu_29,Add_30,Conv_31,Relu_32,Add_33,Add_43,Add_44}, {Relu_35,Conv_36,Relu_37,Add_38,Conv_39,Relu_40,Add_45,Add_46,Conv_47,Relu_48}, {Add_52,Conv_53,Relu_54,Conv_55,Relu_56}, {Conv_58,Relu_59},
[02/19/2021-11:10:13] [I] [TRT] --------------- Layers running on GPU:
[02/19/2021-11:10:13] [I] [TRT] (Unnamed Layer* 0) [Constant] + (Unnamed Layer* 1) [Shuffle] + Add_1, Conv_26, Conv_34, ConvTranspose_49, ConvTranspose_57, ConvTranspose_60,
[02/19/2021-11:10:15] [V] [TRT] Constructing optimization profile number 0 [1/1].
[02/19/2021-11:10:15] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:15] [V] [TRT] Tactic: 1002 time 0.084996
[02/19/2021-11:10:15] [V] [TRT] Tactic: 0 time 0.124564
[02/19/2021-11:10:15] [V] [TRT] Fastest Tactic: 1002 Time: 0.084996
[02/19/2021-11:10:15] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:15] [V] [TRT] Tactic: 1002 time 0.90214
[02/19/2021-11:10:15] [V] [TRT] Tactic: 0 time 0.064912
[02/19/2021-11:10:15] [V] [TRT] Fastest Tactic: 0 Time: 0.064912
[02/19/2021-11:10:15] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:15] [V] [TRT] Tactic: 1002 time 1.11668
[02/19/2021-11:10:15] [V] [TRT] Tactic: 0 time 0.083608
[02/19/2021-11:10:15] [V] [TRT] Fastest Tactic: 0 Time: 0.083608
[02/19/2021-11:10:15] [V] [TRT] *************** Autotuning format combination: Float(1,256,49152,147456) → Float(1,256,49152,147456) ***************
[02/19/2021-11:10:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Constant] + (Unnamed Layer* 1) [Shuffle] + Add_1 (Scale)
[02/19/2021-11:10:15] [V] [TRT] Tactic: 0 time 0.058492
[02/19/2021-11:10:15] [V] [TRT] Fastest Tactic: 0 Time: 0.058492
[02/19/2021-11:10:15] [V] [TRT] *************** Autotuning format combination: Half(1,256,49152,147456) → Half(1,256,49152,147456) ***************
[02/19/2021-11:10:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Constant] + (Unnamed Layer* 1) [Shuffle] + Add_1 (Scale)
[02/19/2021-11:10:15] [V] [TRT] Tactic: 0 time 0.04428
[02/19/2021-11:10:15] [V] [TRT] Fastest Tactic: 0 Time: 0.04428
[02/19/2021-11:10:15] [V] [TRT] *************** Autotuning format combination: Half(1,256,49152:2,98304) → Half(1,256,49152:2,98304) ***************
[02/19/2021-11:10:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Constant] + (Unnamed Layer* 1) [Shuffle] + Add_1 (Scale)
[02/19/2021-11:10:15] [V] [TRT] Tactic: 0 time 0.049664
[02/19/2021-11:10:15] [V] [TRT] Fastest Tactic: 0 Time: 0.049664
[02/19/2021-11:10:15] [V] [TRT] *************** Autotuning format combination: Half(1,256,1:8,49152) → Half(1,256,1:8,49152) ***************
[02/19/2021-11:10:15] [V] [TRT] --------------- Timing Runner: (Unnamed Layer* 0) [Constant] + (Unnamed Layer* 1) [Shuffle] + Add_1 (Scale)
[02/19/2021-11:10:15] [V] [TRT] Tactic: 0 time 0.087672
[02/19/2021-11:10:15] [V] [TRT] Fastest Tactic: 0 Time: 0.087672
[02/19/2021-11:10:15] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:15] [V] [TRT] Tactic: 1002 time 0.0699
[02/19/2021-11:10:15] [V] [TRT] Tactic: 0 time 0.12094
[02/19/2021-11:10:15] [V] [TRT] Fastest Tactic: 1002 Time: 0.0699
[02/19/2021-11:10:15] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:16] [V] [TRT] Tactic: 1002 time 0.913276
[02/19/2021-11:10:16] [V] [TRT] Tactic: 0 time 0.060428
[02/19/2021-11:10:16] [V] [TRT] Fastest Tactic: 0 Time: 0.060428
[02/19/2021-11:10:16] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:16] [V] [TRT] Tactic: 1002 time 1.10758
[02/19/2021-11:10:16] [V] [TRT] Tactic: 0 time 0.08
[02/19/2021-11:10:16] [V] [TRT] Fastest Tactic: 0 Time: 0.08
[02/19/2021-11:10:16] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:16] [V] [TRT] Tactic: 1002 time 1.10501
[02/19/2021-11:10:16] [V] [TRT] Tactic: 0 time 0.079872
[02/19/2021-11:10:16] [V] [TRT] Fastest Tactic: 0 Time: 0.079872
[02/19/2021-11:10:16] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:16] [V] [TRT] Tactic: 1002 time 0.90982
[02/19/2021-11:10:16] [V] [TRT] Tactic: 0 time 0.951668
[02/19/2021-11:10:16] [V] [TRT] Fastest Tactic: 1002 Time: 0.90982
[02/19/2021-11:10:16] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:16] [V] [TRT] Tactic: 1002 time 0.072056
[02/19/2021-11:10:16] [V] [TRT] Tactic: 0 time 0.102384
[02/19/2021-11:10:16] [V] [TRT] Fastest Tactic: 1002 Time: 0.072056
[02/19/2021-11:10:16] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:16] [V] [TRT] Tactic: 1002 time 0.911372
[02/19/2021-11:10:16] [V] [TRT] Tactic: 0 time 0.058892
[02/19/2021-11:10:16] [V] [TRT] Fastest Tactic: 0 Time: 0.058892
[02/19/2021-11:10:16] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:16] [V] [TRT] Tactic: 1002 time 1.10722
[02/19/2021-11:10:16] [V] [TRT] Tactic: 0 time 0.079364
[02/19/2021-11:10:16] [V] [TRT] Fastest Tactic: 0 Time: 0.079364
[02/19/2021-11:10:16] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:16] [V] [TRT] Tactic: 1002 time 1.11476
[02/19/2021-11:10:16] [V] [TRT] Tactic: 0 time 0.080028
[02/19/2021-11:10:16] [V] [TRT] Fastest Tactic: 0 Time: 0.080028
[02/19/2021-11:10:16] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:16] [V] [TRT] Tactic: 1002 time 0.91096
[02/19/2021-11:10:16] [V] [TRT] Tactic: 0 time 0.941792
[02/19/2021-11:10:16] [V] [TRT] Fastest Tactic: 1002 Time: 0.91096
[02/19/2021-11:10:16] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:17] [V] [TRT] Tactic: 1002 time 0.195576
[02/19/2021-11:10:17] [V] [TRT] Tactic: 0 time 0.050164
[02/19/2021-11:10:17] [V] [TRT] Fastest Tactic: 0 Time: 0.050164
[02/19/2021-11:10:17] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:17] [V] [TRT] Tactic: 1002 time 0.197008
[02/19/2021-11:10:17] [V] [TRT] Tactic: 0 time 0.041996
[02/19/2021-11:10:17] [V] [TRT] Fastest Tactic: 0 Time: 0.041996
[02/19/2021-11:10:17] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:17] [V] [TRT] Tactic: 1002 time 1.10823
[02/19/2021-11:10:17] [V] [TRT] Tactic: 0 time 0.102376
[02/19/2021-11:10:17] [V] [TRT] Fastest Tactic: 0 Time: 0.102376
[02/19/2021-11:10:17] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:17] [V] [TRT] Tactic: 1002 time 1.11014
[02/19/2021-11:10:17] [V] [TRT] Tactic: 0 time 0.057864
[02/19/2021-11:10:17] [V] [TRT] Fastest Tactic: 0 Time: 0.057864
[02/19/2021-11:10:17] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:17] [V] [TRT] Tactic: 1002 time 0.910964
[02/19/2021-11:10:17] [V] [TRT] Tactic: 0 time 0.944124
[02/19/2021-11:10:17] [V] [TRT] Fastest Tactic: 1002 Time: 0.910964
[02/19/2021-11:10:17] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:17] [V] [TRT] Tactic: 1002 time 0.213516
[02/19/2021-11:10:17] [V] [TRT] Tactic: 0 time 0.060156
[02/19/2021-11:10:17] [V] [TRT] Fastest Tactic: 0 Time: 0.060156
[02/19/2021-11:10:17] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:17] [V] [TRT] Tactic: 1002 time 0.212868
[02/19/2021-11:10:17] [V] [TRT] Tactic: 0 time 0.050828
[02/19/2021-11:10:17] [V] [TRT] Fastest Tactic: 0 Time: 0.050828
[02/19/2021-11:10:17] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:17] [V] [TRT] Tactic: 1002 time 1.10364
[02/19/2021-11:10:17] [V] [TRT] Tactic: 0 time 0.053248
[02/19/2021-11:10:17] [V] [TRT] Fastest Tactic: 0 Time: 0.053248
[02/19/2021-11:10:17] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:17] [V] [TRT] Tactic: 1002 time 0.90702
[02/19/2021-11:10:17] [V] [TRT] Tactic: 0 time 0.101624
[02/19/2021-11:10:17] [V] [TRT] Fastest Tactic: 0 Time: 0.101624
[02/19/2021-11:10:17] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:17] [V] [TRT] Tactic: 1002 time 0.905732
[02/19/2021-11:10:17] [V] [TRT] Tactic: 0 time 1.00557
[02/19/2021-11:10:17] [V] [TRT] Fastest Tactic: 1002 Time: 0.905732
[02/19/2021-11:10:17] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:18] [V] [TRT] Tactic: 1002 time 1.11129
[02/19/2021-11:10:18] [V] [TRT] Tactic: 0 time 0.08026
[02/19/2021-11:10:18] [V] [TRT] Fastest Tactic: 0 Time: 0.08026
[02/19/2021-11:10:18] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:18] [V] [TRT] Tactic: 1002 time 0.916348
[02/19/2021-11:10:18] [V] [TRT] Tactic: 0 time 0.953356
[02/19/2021-11:10:18] [V] [TRT] Fastest Tactic: 1002 Time: 0.916348
[02/19/2021-11:10:18] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:18] [V] [TRT] Tactic: 1002 time 1.11206
[02/19/2021-11:10:18] [V] [TRT] Tactic: 0 time 0.079496
[02/19/2021-11:10:18] [V] [TRT] Fastest Tactic: 0 Time: 0.079496
[02/19/2021-11:10:18] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:18] [V] [TRT] Tactic: 1002 time 0.9248
[02/19/2021-11:10:18] [V] [TRT] Tactic: 0 time 0.940924
[02/19/2021-11:10:18] [V] [TRT] Fastest Tactic: 1002 Time: 0.9248
[02/19/2021-11:10:18] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:18] [V] [TRT] Tactic: 1002 time 1.11117
[02/19/2021-11:10:18] [V] [TRT] Tactic: 0 time 0.103452
[02/19/2021-11:10:18] [V] [TRT] Fastest Tactic: 0 Time: 0.103452
[02/19/2021-11:10:18] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:18] [V] [TRT] Tactic: 1002 time 0.916328
[02/19/2021-11:10:18] [V] [TRT] Tactic: 0 time 0.944916
[02/19/2021-11:10:18] [V] [TRT] Fastest Tactic: 1002 Time: 0.916328
[02/19/2021-11:10:18] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:18] [V] [TRT] Tactic: 1002 time 0.904968
[02/19/2021-11:10:18] [V] [TRT] Tactic: 0 time 0.966268
[02/19/2021-11:10:18] [V] [TRT] Fastest Tactic: 1002 Time: 0.904968
[02/19/2021-11:10:18] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:18] [V] [TRT] Tactic: 1002 time 0.352272
[02/19/2021-11:10:18] [V] [TRT] Tactic: 0 time 0.040588
[02/19/2021-11:10:18] [V] [TRT] Fastest Tactic: 0 Time: 0.040588
[02/19/2021-11:10:18] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:18] [V] [TRT] Tactic: 1002 time 0.351812
[02/19/2021-11:10:18] [V] [TRT] Tactic: 0 time 0.391624
[02/19/2021-11:10:18] [V] [TRT] Fastest Tactic: 1002 Time: 0.351812
[02/19/2021-11:10:18] [V] [TRT] --------------- Timing Runner: (Reformat)
[02/19/2021-11:10:18] [V] [TRT] Tactic: 1002 time 0.351968
[02/19/2021-11:10:18] [V] [TRT] Tactic: 0 time 0.041632
[02/19/2021-11:10:18] [V] [TRT] Fastest Tactic: 0 Time: 0.041632
[02/19/2021-11:10:18] [V] [TRT] *************** Autotuning format combination: Half(1,256,1:4,49152) → Half(1,32,768:16,3072), Half(1,32,768:16,3072), Half(1,64,3072:16,9216) ***************
[02/19/2021-11:10:18] [V] [TRT] --------------- Timing Runner: {Conv_2,Relu_3,Conv_4,Relu_5,Conv_6,Conv_7,Relu_8,Add_9,Conv_10,Relu_11,Conv_12,Conv_13,Relu_14,Add_15,Conv_16,Relu_17,Conv_18,Relu_19,Conv_20,Relu_21,Add_22,Conv_23,Relu_24,Add_25,Add_41,Add_42,Conv_50,Relu_51} (DLA)
[02/19/2021-11:10:19] [W] [TRT] DLA Node compilation Failed.
[02/19/2021-11:10:19] [V] [TRT] Tactic: 549596028979 skipped. ProcessNode failure.
[02/19/2021-11:10:19] [V] [TRT] Fastest Tactic: -3360065831133338131 Time: 3.40282e+38
[02/19/2021-11:10:19] [V] [TRT] *************** Autotuning format combination: Half(1,256,49152:16,49152) → Half(1,32,768:16,3072), Half(1,32,768:16,3072), Half(1,64,3072:16,9216) ***************
[02/19/2021-11:10:19] [V] [TRT] --------------- Timing Runner: {Conv_2,Relu_3,Conv_4,Relu_5,Conv_6,Conv_7,Relu_8,Add_9,Conv_10,Relu_11,Conv_12,Conv_13,Relu_14,Add_15,Conv_16,Relu_17,Conv_18,Relu_19,Conv_20,Relu_21,Add_22,Conv_23,Relu_24,Add_25,Add_41,Add_42,Conv_50,Relu_51} (DLA)
[02/19/2021-11:10:19] [W] [TRT] DLA Node compilation Failed.
[02/19/2021-11:10:19] [V] [TRT] Tactic: 549596028979 skipped. ProcessNode failure.
[02/19/2021-11:10:19] [V] [TRT] Fastest Tactic: -3360065831133338131 Time: 3.40282e+38
[02/19/2021-11:10:19] [E] [TRT] Try increasing the workspace size with IBuilderConfig::setMaxWorkspaceSize() if using IBuilder::buildEngineWithConfig, or IBuilder::setMaxWorkspaceSize() if using IBuilder::buildCudaEngine.
[02/19/2021-11:10:19] [E] [TRT] …/builder/tacticOptimizer.cpp (1715) - TRTInternal Error in computeCosts: 0 (Could not find any implementation for node {Conv_2,Relu_3,Conv_4,Relu_5,Conv_6,Conv_7,Relu_8,Add_9,Conv_10,Relu_11,Conv_12,Conv_13,Relu_14,Add_15,Conv_16,Relu_17,Conv_18,Relu_19,Conv_20,Relu_21,Add_22,Conv_23,Relu_24,Add_25,Add_41,Add_42,Conv_50,Relu_51}.)
[02/19/2021-11:10:19] [V] [TRT] Builder timing cache: created 33 entries, 0 hit(s)
[02/19/2021-11:10:19] [E] [TRT] …/builder/tacticOptimizer.cpp (1715) - TRTInternal Error in computeCosts: 0 (Could not find any implementation for node {Conv_2,Relu_3,Conv_4,Relu_5,Conv_6,Conv_7,Relu_8,Add_9,Conv_10,Relu_11,Conv_12,Conv_13,Relu_14,Add_15,Conv_16,Relu_17,Conv_18,Relu_19,Conv_20,Relu_21,Add_22,Conv_23,Relu_24,Add_25,Add_41,Add_42,Conv_50,Relu_51}.)
[02/19/2021-11:10:19] [E] Engine creation failed
[02/19/2021-11:10:19] [E] Engine set up failed

How do I solve this problem?
Thanks

Environment

TensorRT Version : 7.1.3
GPU Type : Xavier
Nvidia Driver Version : Package:nvidia-jetpack, Version: 4.4.1-b50
CUDA Version : 10.2.89
CUDNN Version : 8.0.0
Operating System + Version : Ubuntu 18.04
Python Version (if applicable) :
TensorFlow Version (if applicable) :
PyTorch Version (if applicable) :
Baremetal or Container (if container which image + tag) :

Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)

Steps To Reproduce

Please include:

  • Exact steps/commands to build your repro
  • Exact steps/commands to run your repro
  • Full traceback of errors encountered
2

Hi, This looks like a Jetson issue. We recommend you to raise it to the respective platform from the below link

Thanks!

3

Thanks for reply,

What’s the difference between GPU Compute and Enqueue time ?
Which term is the running time for model execution?

Thanks

4

Hi @disculus2012,

Hope this will help you.

=== Explanations of the performance metrics ===
Total Host Walltime: the host walltime from when the first query (after warmups) is enqueued to when the last query is completed.
GPU Compute Time: the GPU latency to execute the kernels for a query.
Total GPU Compute Time: the summation of the GPU Compute Time of all the queries. If this is significantly shorter than Total Host Walltime, the GPU may be under-utilized because of host-side overheads or data transfers.
Throughput: the observed throughput computed by dividing the number of queries by the Total Host Walltime. If this is significantly lower than the reciprocal of GPU Compute Time, the GPU may be under-utilized because of host-side overheads or data transfers.
Enqueue Time: the host latency to enqueue a query. If this is longer than GPU Compute Time, the GPU may be under-utilized.
H2D Latency: the latency for host-to-device data transfers for input tensors of a single query.
D2H Latency: the latency for device-to-host data transfers for output tensors of a single query.
Latency: the summation of H2D Latency, GPU Compute Time, and D2H Latency. This is the latency to infer a single query.
End-to-End Host Latency: the duration from when the H2D of a query is called to when the D2H of the same query is completed, which includes the latency to wait for the completion of the previous query. This is the latency of a query if multiple queries are enqueued consecutively.

Thank you.