Cannot create DLA engine using trtexec

Hi,
I’m trying to benchmark Jetson Xavier NX using trtexec but I can’t utilize the DLA cores.

Device: Jetson Xavier NX Dev kit, model p3450.
OS: Linux nvidiajetson 4.9.140-tegra #1 SMP PREEMPT Wed Apr 8 18:10:49 PDT 2020 aarch64 aarch64 aarch64 GNU/Linux

Reproduce: trtexec <…> --useDLACore=# --allowGPUFallback
The error I’m getting is: Cannot create DLA engine, # not available

I’ve tried several onnx and caffe models.
I tried specifying different DLA cores.

Thanks,
Tamir

Hi,

We can use DLA on XavierNX without error.
Could you try it again?

For example:

$ /usr/src/tensorrt/bin/trtexec --onnx=/usr/src/tensorrt/data/mnist/mnist.onnx --useDLACore=0 --allowGPUFallback

&&&& RUNNING TensorRT.trtexec # /usr/src/tensorrt/bin/trtexec --onnx=/usr/src/tensorrt/data/mnist/mnist.onnx --useDLACore=0 --allowGPUFallback
[05/13/2020-11:30:13] [I] === Model Options ===
[05/13/2020-11:30:13] [I] Format: ONNX
[05/13/2020-11:30:13] [I] Model: /usr/src/tensorrt/data/mnist/mnist.onnx
[05/13/2020-11:30:13] [I] Output:
[05/13/2020-11:30:13] [I] === Build Options ===
[05/13/2020-11:30:13] [I] Max batch: 1
[05/13/2020-11:30:13] [I] Workspace: 16 MB
[05/13/2020-11:30:13] [I] minTiming: 1
[05/13/2020-11:30:13] [I] avgTiming: 8
[05/13/2020-11:30:13] [I] Precision: FP32
[05/13/2020-11:30:13] [I] Calibration: 
[05/13/2020-11:30:13] [I] Safe mode: Disabled
[05/13/2020-11:30:13] [I] Save engine: 
[05/13/2020-11:30:13] [I] Load engine: 
[05/13/2020-11:30:13] [I] Builder Cache: Enabled
[05/13/2020-11:30:13] [I] NVTX verbosity: 0
[05/13/2020-11:30:13] [I] Inputs format: fp32:CHW
[05/13/2020-11:30:13] [I] Outputs format: fp32:CHW
[05/13/2020-11:30:13] [I] Input build shapes: model
[05/13/2020-11:30:13] [I] Input calibration shapes: model
[05/13/2020-11:30:13] [I] === System Options ===
[05/13/2020-11:30:13] [I] Device: 0
[05/13/2020-11:30:13] [I] DLACore: 0(With GPU fallback)
[05/13/2020-11:30:13] [I] Plugins:
[05/13/2020-11:30:13] [I] === Inference Options ===
[05/13/2020-11:30:13] [I] Batch: 1
[05/13/2020-11:30:13] [I] Input inference shapes: model
[05/13/2020-11:30:13] [I] Iterations: 10
[05/13/2020-11:30:13] [I] Duration: 3s (+ 200ms warm up)
[05/13/2020-11:30:13] [I] Sleep time: 0ms
[05/13/2020-11:30:13] [I] Streams: 1
[05/13/2020-11:30:13] [I] ExposeDMA: Disabled
[05/13/2020-11:30:13] [I] Spin-wait: Disabled
[05/13/2020-11:30:13] [I] Multithreading: Disabled
[05/13/2020-11:30:13] [I] CUDA Graph: Disabled
[05/13/2020-11:30:13] [I] Skip inference: Disabled
[05/13/2020-11:30:13] [I] Inputs:
[05/13/2020-11:30:13] [I] === Reporting Options ===
[05/13/2020-11:30:13] [I] Verbose: Disabled
[05/13/2020-11:30:13] [I] Averages: 10 inferences
[05/13/2020-11:30:13] [I] Percentile: 99
[05/13/2020-11:30:13] [I] Dump output: Disabled
[05/13/2020-11:30:13] [I] Profile: Disabled
[05/13/2020-11:30:13] [I] Export timing to JSON file: 
[05/13/2020-11:30:13] [I] Export output to JSON file: 
[05/13/2020-11:30:13] [I] Export profile to JSON file: 
[05/13/2020-11:30:13] [I] 
----------------------------------------------------------------
Input filename:   /usr/src/tensorrt/data/mnist/mnist.onnx
ONNX IR version:  0.0.3
Opset version:    8
Producer name:    CNTK
Producer version: 2.5.1
Domain:           ai.cntk
Model version:    1
Doc string:       
----------------------------------------------------------------
[05/13/2020-11:30:15] [W] [TRT] onnx2trt_utils.cpp:217: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[05/13/2020-11:30:15] [W] [TRT] Default DLA is enabled but layer Times212_reshape1 is not supported on DLA, falling back to GPU.
[05/13/2020-11:30:15] [W] [TRT] Default DLA is enabled but layer (Unnamed Layer* 1) [Shuffle] is not supported on DLA, falling back to GPU.
[05/13/2020-11:30:15] [W] [TRT] Default DLA is enabled but layer Plus30 is not supported on DLA, falling back to GPU.
[05/13/2020-11:30:15] [W] [TRT] Default DLA is enabled but layer (Unnamed Layer* 4) [Shuffle] is not supported on DLA, falling back to GPU.
[05/13/2020-11:30:15] [W] [TRT] Default DLA is enabled but layer Plus112 is not supported on DLA, falling back to GPU.
[05/13/2020-11:30:15] [W] [TRT] Default DLA is enabled but layer (Unnamed Layer* 10) [Shuffle] is not supported on DLA, falling back to GPU.
[05/13/2020-11:30:15] [W] [TRT] Default DLA is enabled but layer Times212_reshape0 is not supported on DLA, falling back to GPU.
[05/13/2020-11:30:15] [W] [TRT] Default DLA is enabled but layer Times212 is not supported on DLA, falling back to GPU.
[05/13/2020-11:30:15] [W] [TRT] Default DLA is enabled but layer Plus214 is not supported on DLA, falling back to GPU.
[05/13/2020-11:30:15] [W] [TRT] DLA allows only same dimensions inputs to Elementwise.
[05/13/2020-11:30:15] [W] [TRT] Internal DLA error for layer (Unnamed Layer* 5) [ElementWise]. Switching to GPU fallback.
[05/13/2020-11:30:15] [W] [TRT] DLA allows only same dimensions inputs to Elementwise.
[05/13/2020-11:30:15] [W] [TRT] Internal DLA error for layer (Unnamed Layer* 5) [ElementWise]. Switching to GPU fallback.
[05/13/2020-11:30:15] [W] [TRT] DLA allows only same dimensions inputs to Elementwise.
[05/13/2020-11:30:15] [W] [TRT] Internal DLA error for layer (Unnamed Layer* 11) [ElementWise]. Switching to GPU fallback.
[05/13/2020-11:30:15] [W] [TRT] DLA allows only same dimensions inputs to Elementwise.
[05/13/2020-11:30:15] [W] [TRT] Internal DLA error for layer (Unnamed Layer* 11) [ElementWise]. Switching to GPU fallback.
[05/13/2020-11:30:15] [I] [TRT] 
[05/13/2020-11:30:15] [I] [TRT] --------------- Layers running on DLA: 
[05/13/2020-11:30:15] [I] [TRT] {Convolution28}, {ReLU32,Pooling66,Convolution110}, {ReLU114,Pooling160}, {(Unnamed Layer* 17) [ElementWise]}, 
[05/13/2020-11:30:15] [I] [TRT] --------------- Layers running on GPU: 
[05/13/2020-11:30:15] [I] [TRT] Times212_reshape1 + (Unnamed Layer* 1) [Shuffle], Plus214 + shuffle_(Unnamed Layer* 16) [Constant]_output, Plus30 + (Unnamed Layer* 4) [Shuffle] + (Unnamed Layer* 5) [ElementWise], Plus112 + (Unnamed Layer* 10) [Shuffle] + (Unnamed Layer* 11) [ElementWise], Times212_reshape0, Times212, shuffle_Times212_Output_0, 
[05/13/2020-11:30:20] [I] [TRT] Detected 1 inputs and 1 output network tensors.
[05/13/2020-11:30:20] [I] Starting inference threads
[05/13/2020-11:30:23] [I] Warmup completed 65 queries over 200 ms
[05/13/2020-11:30:23] [I] Timing trace has 1540 queries over 3.00178 s
[05/13/2020-11:30:23] [I] Trace averages of 10 runs:
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.25926 ms - Host latency: 2.31029 ms (end to end 2.33315 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.12102 ms - Host latency: 2.17472 ms (end to end 2.19744 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.27145 ms - Host latency: 2.3285 ms (end to end 2.3485 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.22612 ms - Host latency: 2.27352 ms (end to end 2.29343 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.98644 ms - Host latency: 2.04094 ms (end to end 2.0653 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.13427 ms - Host latency: 2.18671 ms (end to end 2.20759 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.18175 ms - Host latency: 2.24178 ms (end to end 2.26644 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.12298 ms - Host latency: 2.17021 ms (end to end 2.19175 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.02318 ms - Host latency: 2.07219 ms (end to end 2.09181 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.17599 ms - Host latency: 2.22441 ms (end to end 2.24572 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.09725 ms - Host latency: 2.15331 ms (end to end 2.17357 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.13118 ms - Host latency: 2.18824 ms (end to end 2.20901 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.16983 ms - Host latency: 2.22609 ms (end to end 2.25282 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.18455 ms - Host latency: 2.23422 ms (end to end 2.25529 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.15133 ms - Host latency: 2.20685 ms (end to end 2.22948 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.04279 ms - Host latency: 2.09619 ms (end to end 2.11642 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.12896 ms - Host latency: 2.18678 ms (end to end 2.20734 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.96642 ms - Host latency: 2.02186 ms (end to end 2.04253 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.08946 ms - Host latency: 2.13667 ms (end to end 2.15648 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.10998 ms - Host latency: 2.16209 ms (end to end 2.18196 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.05159 ms - Host latency: 2.10229 ms (end to end 2.12456 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.00067 ms - Host latency: 2.05045 ms (end to end 2.07114 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.09932 ms - Host latency: 2.15175 ms (end to end 2.1738 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.04647 ms - Host latency: 2.09622 ms (end to end 2.11979 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.05337 ms - Host latency: 2.10727 ms (end to end 2.13351 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.97419 ms - Host latency: 2.02648 ms (end to end 2.04552 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.17649 ms - Host latency: 2.22481 ms (end to end 2.25114 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.09991 ms - Host latency: 2.15068 ms (end to end 2.17311 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.97551 ms - Host latency: 2.0377 ms (end to end 2.05728 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.07507 ms - Host latency: 2.1264 ms (end to end 2.14628 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.99385 ms - Host latency: 2.04572 ms (end to end 2.06857 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.08351 ms - Host latency: 2.13513 ms (end to end 2.15377 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.06802 ms - Host latency: 2.12167 ms (end to end 2.14286 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.91091 ms - Host latency: 1.9625 ms (end to end 1.98395 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.93581 ms - Host latency: 1.98777 ms (end to end 2.00571 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.02736 ms - Host latency: 2.08398 ms (end to end 2.10292 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.87901 ms - Host latency: 1.93368 ms (end to end 1.95191 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.93173 ms - Host latency: 1.99455 ms (end to end 2.01329 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.97783 ms - Host latency: 2.02336 ms (end to end 2.04501 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.86157 ms - Host latency: 1.9101 ms (end to end 1.93345 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.85065 ms - Host latency: 1.89771 ms (end to end 1.9191 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.77128 ms - Host latency: 1.81991 ms (end to end 1.83872 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.90112 ms - Host latency: 1.95021 ms (end to end 1.96675 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.94144 ms - Host latency: 1.9965 ms (end to end 2.0172 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.88627 ms - Host latency: 1.93561 ms (end to end 1.95482 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.97374 ms - Host latency: 2.0252 ms (end to end 2.04662 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 3.23518 ms - Host latency: 3.2936 ms (end to end 3.31641 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.59121 ms - Host latency: 2.65276 ms (end to end 2.67594 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.91168 ms - Host latency: 1.96165 ms (end to end 1.97948 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.83235 ms - Host latency: 1.88783 ms (end to end 1.90701 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 2.94857 ms - Host latency: 3.0116 ms (end to end 3.03269 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.86616 ms - Host latency: 1.91748 ms (end to end 1.93849 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.7866 ms - Host latency: 1.85126 ms (end to end 1.86913 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.89004 ms - Host latency: 1.94792 ms (end to end 1.96642 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.89628 ms - Host latency: 1.9516 ms (end to end 1.97168 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.80956 ms - Host latency: 1.85792 ms (end to end 1.87533 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.78693 ms - Host latency: 1.83684 ms (end to end 1.85674 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.75554 ms - Host latency: 1.80596 ms (end to end 1.82585 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.73459 ms - Host latency: 1.78706 ms (end to end 1.80688 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.72297 ms - Host latency: 1.77076 ms (end to end 1.79753 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.9374 ms - Host latency: 2.00439 ms (end to end 2.03171 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.78407 ms - Host latency: 1.83702 ms (end to end 1.86226 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.83423 ms - Host latency: 1.8918 ms (end to end 1.90948 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.86104 ms - Host latency: 1.91277 ms (end to end 1.93103 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.86229 ms - Host latency: 1.93099 ms (end to end 1.9519 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.81978 ms - Host latency: 1.87217 ms (end to end 1.89181 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.75409 ms - Host latency: 1.81379 ms (end to end 1.85073 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.85697 ms - Host latency: 1.91086 ms (end to end 1.92952 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.69637 ms - Host latency: 1.75231 ms (end to end 1.77292 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.80356 ms - Host latency: 1.85833 ms (end to end 1.88099 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.93512 ms - Host latency: 1.99202 ms (end to end 2.01205 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.97 ms - Host latency: 2.02329 ms (end to end 2.0418 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.89891 ms - Host latency: 1.94901 ms (end to end 1.97203 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.86681 ms - Host latency: 1.91183 ms (end to end 1.93254 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.78345 ms - Host latency: 1.83392 ms (end to end 1.85211 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.68514 ms - Host latency: 1.74285 ms (end to end 1.76709 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.71904 ms - Host latency: 1.76615 ms (end to end 1.78923 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.69663 ms - Host latency: 1.74105 ms (end to end 1.75983 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.72567 ms - Host latency: 1.77148 ms (end to end 1.78838 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.66008 ms - Host latency: 1.70439 ms (end to end 1.7212 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.7478 ms - Host latency: 1.79055 ms (end to end 1.80779 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.68026 ms - Host latency: 1.72751 ms (end to end 1.74718 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.70464 ms - Host latency: 1.74856 ms (end to end 1.76552 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.70009 ms - Host latency: 1.74696 ms (end to end 1.76353 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.66777 ms - Host latency: 1.72197 ms (end to end 1.73911 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.69312 ms - Host latency: 1.74492 ms (end to end 1.76185 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.74454 ms - Host latency: 1.78917 ms (end to end 1.80618 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.79611 ms - Host latency: 1.84186 ms (end to end 1.86699 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.68409 ms - Host latency: 1.73959 ms (end to end 1.75713 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.80068 ms - Host latency: 1.85381 ms (end to end 1.87161 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.66667 ms - Host latency: 1.71548 ms (end to end 1.73228 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.65247 ms - Host latency: 1.70664 ms (end to end 1.72476 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.64299 ms - Host latency: 1.69004 ms (end to end 1.70896 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.67141 ms - Host latency: 1.71853 ms (end to end 1.73782 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.66816 ms - Host latency: 1.72146 ms (end to end 1.73853 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.69995 ms - Host latency: 1.75701 ms (end to end 1.78284 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.71516 ms - Host latency: 1.77507 ms (end to end 1.79563 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.64861 ms - Host latency: 1.70378 ms (end to end 1.72112 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.69177 ms - Host latency: 1.73689 ms (end to end 1.75845 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.75518 ms - Host latency: 1.80625 ms (end to end 1.82778 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.63035 ms - Host latency: 1.67686 ms (end to end 1.69343 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.72063 ms - Host latency: 1.77915 ms (end to end 1.79585 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.63586 ms - Host latency: 1.68574 ms (end to end 1.70254 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.73777 ms - Host latency: 1.79976 ms (end to end 1.81714 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.6512 ms - Host latency: 1.69854 ms (end to end 1.71636 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.66985 ms - Host latency: 1.71836 ms (end to end 1.73789 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.64434 ms - Host latency: 1.68801 ms (end to end 1.70513 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.65903 ms - Host latency: 1.71589 ms (end to end 1.73262 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.68301 ms - Host latency: 1.74077 ms (end to end 1.75779 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.70486 ms - Host latency: 1.75208 ms (end to end 1.76912 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.62737 ms - Host latency: 1.68323 ms (end to end 1.70227 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.72585 ms - Host latency: 1.77771 ms (end to end 1.7948 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.63166 ms - Host latency: 1.68889 ms (end to end 1.70576 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.7124 ms - Host latency: 1.76604 ms (end to end 1.78884 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.66831 ms - Host latency: 1.71648 ms (end to end 1.73638 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.62554 ms - Host latency: 1.67419 ms (end to end 1.69119 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.5916 ms - Host latency: 1.64253 ms (end to end 1.66155 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.62688 ms - Host latency: 1.67478 ms (end to end 1.69187 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.60388 ms - Host latency: 1.65725 ms (end to end 1.67363 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.71631 ms - Host latency: 1.76633 ms (end to end 1.78345 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.73398 ms - Host latency: 1.79177 ms (end to end 1.81187 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.63687 ms - Host latency: 1.6887 ms (end to end 1.70583 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.66169 ms - Host latency: 1.71448 ms (end to end 1.73064 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.68333 ms - Host latency: 1.72891 ms (end to end 1.74602 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.59426 ms - Host latency: 1.64429 ms (end to end 1.66328 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.70789 ms - Host latency: 1.75894 ms (end to end 1.77593 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.61499 ms - Host latency: 1.66572 ms (end to end 1.6873 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.77642 ms - Host latency: 1.82424 ms (end to end 1.84211 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.68584 ms - Host latency: 1.74094 ms (end to end 1.75806 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.61367 ms - Host latency: 1.66025 ms (end to end 1.67773 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.66509 ms - Host latency: 1.71116 ms (end to end 1.73323 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.61643 ms - Host latency: 1.66604 ms (end to end 1.68445 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.59121 ms - Host latency: 1.64153 ms (end to end 1.65781 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.64465 ms - Host latency: 1.69438 ms (end to end 1.71067 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.7803 ms - Host latency: 1.83721 ms (end to end 1.85649 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.73652 ms - Host latency: 1.78582 ms (end to end 1.80264 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.59697 ms - Host latency: 1.65977 ms (end to end 1.68123 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.58228 ms - Host latency: 1.6325 ms (end to end 1.64888 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.63062 ms - Host latency: 1.6772 ms (end to end 1.69587 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.652 ms - Host latency: 1.70435 ms (end to end 1.72341 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.62732 ms - Host latency: 1.67446 ms (end to end 1.69441 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.67273 ms - Host latency: 1.71604 ms (end to end 1.73491 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.58542 ms - Host latency: 1.63162 ms (end to end 1.64731 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.81489 ms - Host latency: 1.87053 ms (end to end 1.89167 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.57537 ms - Host latency: 1.62161 ms (end to end 1.64089 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.62021 ms - Host latency: 1.66995 ms (end to end 1.68621 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.5969 ms - Host latency: 1.64229 ms (end to end 1.65835 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.54824 ms - Host latency: 1.59546 ms (end to end 1.61638 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.60945 ms - Host latency: 1.65659 ms (end to end 1.67349 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.70107 ms - Host latency: 1.75186 ms (end to end 1.76882 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.57024 ms - Host latency: 1.61858 ms (end to end 1.63474 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.56904 ms - Host latency: 1.62246 ms (end to end 1.63845 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.55649 ms - Host latency: 1.6064 ms (end to end 1.62236 ms)
[05/13/2020-11:30:23] [I] Average on 10 runs - GPU latency: 1.56201 ms - Host latency: 1.60989 ms (end to end 1.62825 ms)
[05/13/2020-11:30:23] [I] Host latency
[05/13/2020-11:30:23] [I] min: 1.49585 ms (end to end 1.51245 ms)
[05/13/2020-11:30:23] [I] max: 6.35095 ms (end to end 6.37268 ms)
[05/13/2020-11:30:23] [I] mean: 1.88404 ms (end to end 1.90375 ms)
[05/13/2020-11:30:23] [I] median: 1.77917 ms (end to end 1.79852 ms)
[05/13/2020-11:30:23] [I] percentile: 3.25458 ms at 99% (end to end 3.27576 ms at 99%)
[05/13/2020-11:30:23] [I] throughput: 513.03 qps
[05/13/2020-11:30:23] [I] walltime: 3.00178 s
[05/13/2020-11:30:23] [I] GPU Compute
[05/13/2020-11:30:23] [I] min: 1.45264 ms
[05/13/2020-11:30:23] [I] max: 6.29065 ms
[05/13/2020-11:30:23] [I] mean: 1.83215 ms
[05/13/2020-11:30:23] [I] median: 1.72852 ms
[05/13/2020-11:30:23] [I] percentile: 3.19846 ms at 99%
[05/13/2020-11:30:23] [I] total compute time: 2.82152 s
&&&& PASSED TensorRT.trtexec # /usr/src/tensorrt/bin/trtexec --onnx=/usr/src/tensorrt/data/mnist/mnist.onnx --useDLACore=0 --allowGPUFallback

Thanks.