&&&& RUNNING TensorRT.trtexec [TensorRT v8001] # /usr/src/tensorrt/bin/trtexec --onnx=/home/acer/nfs-share/epoch_250.onnx --int8 [09/25/2021-10:21:24] [I] === Model Options === [09/25/2021-10:21:24] [I] Format: ONNX [09/25/2021-10:21:24] [I] Model: /home/acer/nfs-share/epoch_250.onnx [09/25/2021-10:21:24] [I] Output: [09/25/2021-10:21:24] [I] === Build Options === [09/25/2021-10:21:24] [I] Max batch: explicit [09/25/2021-10:21:24] [I] Workspace: 16 MiB [09/25/2021-10:21:24] [I] minTiming: 1 [09/25/2021-10:21:24] [I] avgTiming: 8 [09/25/2021-10:21:24] [I] Precision: FP32+INT8 [09/25/2021-10:21:24] [I] Calibration: Dynamic [09/25/2021-10:21:24] [I] Refit: Disabled [09/25/2021-10:21:24] [I] Sparsity: Disabled [09/25/2021-10:21:24] [I] Safe mode: Disabled [09/25/2021-10:21:24] [I] Restricted mode: Disabled [09/25/2021-10:21:24] [I] Save engine: [09/25/2021-10:21:24] [I] Load engine: [09/25/2021-10:21:24] [I] NVTX verbosity: 0 [09/25/2021-10:21:24] [I] Tactic sources: Using default tactic sources [09/25/2021-10:21:24] [I] timingCacheMode: local [09/25/2021-10:21:24] [I] timingCacheFile: [09/25/2021-10:21:24] [I] Input(s)s format: fp32:CHW [09/25/2021-10:21:24] [I] Output(s)s format: fp32:CHW [09/25/2021-10:21:24] [I] Input build shapes: model [09/25/2021-10:21:24] [I] Input calibration shapes: model [09/25/2021-10:21:24] [I] === System Options === [09/25/2021-10:21:24] [I] Device: 0 [09/25/2021-10:21:24] [I] DLACore: [09/25/2021-10:21:24] [I] Plugins: [09/25/2021-10:21:24] [I] === Inference Options === [09/25/2021-10:21:24] [I] Batch: Explicit [09/25/2021-10:21:24] [I] Input inference shapes: model [09/25/2021-10:21:24] [I] Iterations: 10 [09/25/2021-10:21:24] [I] Duration: 3s (+ 200ms warm up) [09/25/2021-10:21:24] [I] Sleep time: 0ms [09/25/2021-10:21:24] [I] Streams: 1 [09/25/2021-10:21:24] [I] ExposeDMA: Disabled [09/25/2021-10:21:24] [I] Data transfers: Enabled [09/25/2021-10:21:24] [I] Spin-wait: Disabled [09/25/2021-10:21:24] [I] Multithreading: Disabled [09/25/2021-10:21:24] [I] CUDA Graph: Disabled [09/25/2021-10:21:24] [I] Separate profiling: Disabled [09/25/2021-10:21:24] [I] Time Deserialize: Disabled [09/25/2021-10:21:24] [I] Time Refit: Disabled [09/25/2021-10:21:24] [I] Skip inference: Disabled [09/25/2021-10:21:24] [I] Inputs: [09/25/2021-10:21:24] [I] === Reporting Options === [09/25/2021-10:21:24] [I] Verbose: Disabled [09/25/2021-10:21:24] [I] Averages: 10 inferences [09/25/2021-10:21:24] [I] Percentile: 99 [09/25/2021-10:21:24] [I] Dump refittable layers:Disabled [09/25/2021-10:21:24] [I] Dump output: Disabled [09/25/2021-10:21:24] [I] Profile: Disabled [09/25/2021-10:21:24] [I] Export timing to JSON file: [09/25/2021-10:21:24] [I] Export output to JSON file: [09/25/2021-10:21:24] [I] Export profile to JSON file: [09/25/2021-10:21:24] [I] [09/25/2021-10:21:24] [I] === Device Information === [09/25/2021-10:21:24] [I] Selected Device: Xavier [09/25/2021-10:21:24] [I] Compute Capability: 7.2 [09/25/2021-10:21:24] [I] SMs: 6 [09/25/2021-10:21:24] [I] Compute Clock Rate: 1.109 GHz [09/25/2021-10:21:24] [I] Device Global Memory: 7773 MiB [09/25/2021-10:21:24] [I] Shared Memory per SM: 96 KiB [09/25/2021-10:21:24] [I] Memory Bus Width: 256 bits (ECC disabled) [09/25/2021-10:21:24] [I] Memory Clock Rate: 1.109 GHz [09/25/2021-10:21:24] [I] [09/25/2021-10:21:24] [I] TensorRT version: 8001 [09/25/2021-10:21:25] [I] [TRT] [MemUsageChange] Init CUDA: CPU +353, GPU +0, now: CPU 371, GPU 4143 (MiB) [09/25/2021-10:21:25] [I] Start parsing network model [09/25/2021-10:21:25] [I] [TRT] ---------------------------------------------------------------- [09/25/2021-10:21:25] [I] [TRT] Input filename: /home/acer/nfs-share/epoch_250.onnx [09/25/2021-10:21:25] [I] [TRT] ONNX IR version: 0.0.6 [09/25/2021-10:21:25] [I] [TRT] Opset version: 13 [09/25/2021-10:21:25] [I] [TRT] Producer name: pytorch [09/25/2021-10:21:25] [I] [TRT] Producer version: 1.8 [09/25/2021-10:21:25] [I] [TRT] Domain: [09/25/2021-10:21:25] [I] [TRT] Model version: 0 [09/25/2021-10:21:25] [I] [TRT] Doc string: [09/25/2021-10:21:25] [I] [TRT] ---------------------------------------------------------------- [09/25/2021-10:21:25] [09/25/2021-10:21:25] [I] Finish parsing network model [09/25/2021-10:21:25] [I] [TRT] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 374, GPU 4149 (MiB) [09/25/2021-10:21:25] [I] FP32 and INT8 precisions have been specified - more performance might be enabled by additionally specifying --fp16 or --best [09/25/2021-10:21:25] [I] [TRT] [MemUsageSnapshot] Builder begin: CPU 374 MiB, GPU 4149 MiB [09/25/2021-10:21:25] [09/25/2021-10:21:25] [I] [TRT] ---------- Layers Running on DLA ---------- [09/25/2021-10:21:25] [I] [TRT] ---------- Layers Running on GPU ---------- [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_0 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_1 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_2 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_3 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_4 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_5 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_6 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_7 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_8 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_9 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_10 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_11 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_12 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_13 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_14 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_15 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_16 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_17 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_18 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_19 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_20 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_21 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_22 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_54 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_23 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_24 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_25 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_26 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_27 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_28 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_29 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_30 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_31 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_32 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_33 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_34 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_35 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_36 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_37 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_38 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_39 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_40 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_41 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_42 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_43 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_44 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_45 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_46 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_56 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_47 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_48 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_49 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_50 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_51 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_52 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_53 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_58 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_59 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_122 || Conv_123 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_124 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_125 || Conv_126 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Resize_78 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_127 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] PWN(LeakyRelu_57, Add_79) [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_128 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_80 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] 748 copy [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] 754 copy [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_81 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] PWN(Relu_130) [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_113 || Conv_114 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_149 || Conv_177 || Conv_205 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_115 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_116 || Conv_117 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Resize_100 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_118 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] PWN(LeakyRelu_55, Add_101) [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_119 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_102 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] 733 copy [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] 739 copy [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_103 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] PWN(Relu_121) [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Transpose_150 + Reshape_157 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Transpose_178 + Reshape_185 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Transpose_206 + Reshape_213 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_104 || Conv_105 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_140 || Conv_168 || Conv_196 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_106 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_107 || Conv_108 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] LeakyRelu_109 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_110 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] 718 copy [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] 724 copy [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] PWN(Relu_112) [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Transpose_141 + Reshape_148 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Transpose_169 + Reshape_176 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Transpose_197 + Reshape_204 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Conv_131 || Conv_159 || Conv_187 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Transpose_132 + Reshape_139 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Transpose_160 + Reshape_167 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Transpose_188 + Reshape_195 [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] 497 copy [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] 512 copy [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] 527 copy [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] 543 copy [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] 558 copy [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] 573 copy [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] 589 copy [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] 604 copy [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] 619 copy [09/25/2021-10:21:25] [I] [TRT] [GpuLayer] Softmax_215 [09/25/2021-10:21:26] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +226, GPU +292, now: CPU 601, GPU 4441 (MiB) [09/25/2021-10:21:27] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +307, GPU +396, now: CPU 908, GPU 4837 (MiB) [09/25/2021-10:21:27] [09/25/2021-10:23:09] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [09/25/2021-10:28:51] [I] [TRT] Detected 1 inputs and 9 output network tensors. [09/25/2021-10:28:51] [I] [TRT] Total Host Persistent Memory: 95168 [09/25/2021-10:28:51] [I] [TRT] Total Device Persistent Memory: 1280512 [09/25/2021-10:28:51] [I] [TRT] Total Scratch Memory: 0 [09/25/2021-10:28:51] [I] [TRT] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 2 MiB, GPU 59 MiB [09/25/2021-10:28:51] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 1386, GPU 5484 (MiB) [09/25/2021-10:28:52] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 1386, GPU 5492 (MiB) [09/25/2021-10:28:52] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1386, GPU 5479 (MiB) [09/25/2021-10:28:52] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1385, GPU 5461 (MiB) [09/25/2021-10:28:52] [I] [TRT] [MemUsageSnapshot] Builder end: CPU 1384 MiB, GPU 5461 MiB [09/25/2021-10:28:52] [I] [TRT] Loaded engine size: 4 MB [09/25/2021-10:28:52] [I] [TRT] [MemUsageSnapshot] deserializeCudaEngine begin: CPU 1378 MiB, GPU 5455 MiB [09/25/2021-10:28:52] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +7, now: CPU 1384, GPU 5462 (MiB) [09/25/2021-10:28:52] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +10, now: CPU 1384, GPU 5472 (MiB) [09/25/2021-10:28:52] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1384, GPU 5457 (MiB) [09/25/2021-10:28:52] [I] [TRT] [MemUsageSnapshot] deserializeCudaEngine end: CPU 1384 MiB, GPU 5457 MiB [09/25/2021-10:28:52] [I] Engine built in 447.925 sec. [09/25/2021-10:28:52] [I] [TRT] [MemUsageSnapshot] ExecutionContext creation begin: CPU 1381 MiB, GPU 5457 MiB [09/25/2021-10:28:52] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +5, now: CPU 1381, GPU 5462 (MiB) [09/25/2021-10:28:52] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +10, now: CPU 1381, GPU 5472 (MiB) [09/25/2021-10:28:52] [I] [TRT] [MemUsageSnapshot] ExecutionContext creation end: CPU 1381 MiB, GPU 5477 MiB [09/25/2021-10:28:52] [I] Created input binding for input.1 with dimensions 1x3x640x352 [09/25/2021-10:28:52] [I] Created output binding for 528 with dimensions 1x9240x4 [09/25/2021-10:28:52] [I] Created output binding for 620 with dimensions 1x9240x10 [09/25/2021-10:28:52] [I] Created output binding for 621 with dimensions 1x9240x2 [09/25/2021-10:28:52] [I] Starting inference [09/25/2021-10:28:55] [I] Warmup completed 82 queries over 200 ms [09/25/2021-10:28:55] [I] Timing trace has 1244 queries over 3.00547 s [09/25/2021-10:28:55] [I] [09/25/2021-10:28:55] [I] === Trace details === [09/25/2021-10:28:55] [I] Trace averages of 10 runs: [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25485 ms - Host latency: 2.40568 ms (end to end 2.41749 ms, enqueue 1.78736 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25157 ms - Host latency: 2.40209 ms (end to end 2.41111 ms, enqueue 1.79705 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.24953 ms - Host latency: 2.39977 ms (end to end 2.40923 ms, enqueue 1.80574 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25359 ms - Host latency: 2.40408 ms (end to end 2.41311 ms, enqueue 1.79756 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25127 ms - Host latency: 2.40151 ms (end to end 2.41357 ms, enqueue 1.79114 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25379 ms - Host latency: 2.40413 ms (end to end 2.41475 ms, enqueue 1.76942 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25373 ms - Host latency: 2.40425 ms (end to end 2.41357 ms, enqueue 1.77314 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.2503 ms - Host latency: 2.40051 ms (end to end 2.41166 ms, enqueue 1.78271 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.2488 ms - Host latency: 2.3993 ms (end to end 2.41022 ms, enqueue 1.76222 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25283 ms - Host latency: 2.40305 ms (end to end 2.41507 ms, enqueue 1.77188 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25408 ms - Host latency: 2.40436 ms (end to end 2.41538 ms, enqueue 1.762 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.2516 ms - Host latency: 2.40275 ms (end to end 2.41327 ms, enqueue 1.81776 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25428 ms - Host latency: 2.40495 ms (end to end 2.41558 ms, enqueue 1.76402 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25074 ms - Host latency: 2.40143 ms (end to end 2.4122 ms, enqueue 1.77304 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25104 ms - Host latency: 2.40151 ms (end to end 2.41231 ms, enqueue 1.7897 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25259 ms - Host latency: 2.40298 ms (end to end 2.41392 ms, enqueue 1.77136 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25681 ms - Host latency: 2.40729 ms (end to end 2.41829 ms, enqueue 1.77422 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25238 ms - Host latency: 2.4033 ms (end to end 2.41351 ms, enqueue 1.80088 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25528 ms - Host latency: 2.40607 ms (end to end 2.41586 ms, enqueue 1.76802 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25376 ms - Host latency: 2.40483 ms (end to end 2.4174 ms, enqueue 1.79651 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25457 ms - Host latency: 2.40478 ms (end to end 2.4137 ms, enqueue 1.77661 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25047 ms - Host latency: 2.40042 ms (end to end 2.41066 ms, enqueue 1.77613 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25464 ms - Host latency: 2.40476 ms (end to end 2.41561 ms, enqueue 1.77491 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.252 ms - Host latency: 2.40224 ms (end to end 2.41221 ms, enqueue 1.76031 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.24875 ms - Host latency: 2.39921 ms (end to end 2.41138 ms, enqueue 1.78238 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.24876 ms - Host latency: 2.39916 ms (end to end 2.41014 ms, enqueue 1.78665 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25677 ms - Host latency: 2.40664 ms (end to end 2.4168 ms, enqueue 1.77681 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25315 ms - Host latency: 2.40398 ms (end to end 2.4151 ms, enqueue 1.78049 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.2509 ms - Host latency: 2.40103 ms (end to end 2.40945 ms, enqueue 1.76879 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25061 ms - Host latency: 2.40088 ms (end to end 2.4118 ms, enqueue 1.79374 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.24796 ms - Host latency: 2.39814 ms (end to end 2.40955 ms, enqueue 1.87123 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25093 ms - Host latency: 2.40146 ms (end to end 2.41307 ms, enqueue 1.78862 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25608 ms - Host latency: 2.40803 ms (end to end 2.41856 ms, enqueue 1.75831 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25858 ms - Host latency: 2.40896 ms (end to end 2.42054 ms, enqueue 1.74742 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25359 ms - Host latency: 2.40433 ms (end to end 2.41746 ms, enqueue 1.75564 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25364 ms - Host latency: 2.40425 ms (end to end 2.41423 ms, enqueue 1.75355 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25508 ms - Host latency: 2.405 ms (end to end 2.41563 ms, enqueue 1.74979 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25513 ms - Host latency: 2.40591 ms (end to end 2.41776 ms, enqueue 1.75457 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25585 ms - Host latency: 2.40613 ms (end to end 2.41758 ms, enqueue 1.74772 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25398 ms - Host latency: 2.40455 ms (end to end 2.41694 ms, enqueue 1.75907 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25272 ms - Host latency: 2.40356 ms (end to end 2.41433 ms, enqueue 1.76064 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.2541 ms - Host latency: 2.40465 ms (end to end 2.4163 ms, enqueue 1.75587 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25482 ms - Host latency: 2.40527 ms (end to end 2.41752 ms, enqueue 1.74513 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25181 ms - Host latency: 2.40244 ms (end to end 2.41555 ms, enqueue 1.74572 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25244 ms - Host latency: 2.40297 ms (end to end 2.41337 ms, enqueue 1.77286 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25369 ms - Host latency: 2.40383 ms (end to end 2.41375 ms, enqueue 1.74713 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25376 ms - Host latency: 2.40469 ms (end to end 2.41486 ms, enqueue 1.76505 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.2564 ms - Host latency: 2.4066 ms (end to end 2.41901 ms, enqueue 1.74038 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25304 ms - Host latency: 2.40342 ms (end to end 2.41288 ms, enqueue 1.73989 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25386 ms - Host latency: 2.40427 ms (end to end 2.41567 ms, enqueue 1.76849 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25336 ms - Host latency: 2.40359 ms (end to end 2.41426 ms, enqueue 1.75123 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25227 ms - Host latency: 2.40319 ms (end to end 2.41328 ms, enqueue 1.77408 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25776 ms - Host latency: 2.40934 ms (end to end 2.42169 ms, enqueue 1.74474 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.2533 ms - Host latency: 2.40352 ms (end to end 2.41349 ms, enqueue 1.75596 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.2572 ms - Host latency: 2.40797 ms (end to end 2.41964 ms, enqueue 1.74943 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25297 ms - Host latency: 2.40316 ms (end to end 2.41538 ms, enqueue 1.76254 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25647 ms - Host latency: 2.40656 ms (end to end 2.41646 ms, enqueue 1.76119 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25383 ms - Host latency: 2.40448 ms (end to end 2.41493 ms, enqueue 1.76741 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25269 ms - Host latency: 2.40381 ms (end to end 2.41549 ms, enqueue 1.79326 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25247 ms - Host latency: 2.40315 ms (end to end 2.41512 ms, enqueue 1.7772 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25588 ms - Host latency: 2.40642 ms (end to end 2.41583 ms, enqueue 1.76042 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25559 ms - Host latency: 2.4054 ms (end to end 2.41748 ms, enqueue 1.74187 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25282 ms - Host latency: 2.40388 ms (end to end 2.4168 ms, enqueue 1.77659 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25079 ms - Host latency: 2.40076 ms (end to end 2.40964 ms, enqueue 1.77135 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25638 ms - Host latency: 2.40688 ms (end to end 2.4192 ms, enqueue 1.75052 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.2507 ms - Host latency: 2.40209 ms (end to end 2.41418 ms, enqueue 1.74205 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25112 ms - Host latency: 2.40138 ms (end to end 2.41147 ms, enqueue 1.76154 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.2505 ms - Host latency: 2.40123 ms (end to end 2.41157 ms, enqueue 1.77948 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25266 ms - Host latency: 2.40328 ms (end to end 2.41241 ms, enqueue 1.76218 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25458 ms - Host latency: 2.40488 ms (end to end 2.41481 ms, enqueue 1.76321 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25281 ms - Host latency: 2.40333 ms (end to end 2.41284 ms, enqueue 1.76482 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25219 ms - Host latency: 2.40226 ms (end to end 2.41553 ms, enqueue 1.75465 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25353 ms - Host latency: 2.40397 ms (end to end 2.41409 ms, enqueue 1.7495 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25477 ms - Host latency: 2.40593 ms (end to end 2.41678 ms, enqueue 1.75511 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25016 ms - Host latency: 2.40073 ms (end to end 2.41147 ms, enqueue 1.80646 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25096 ms - Host latency: 2.40178 ms (end to end 2.41169 ms, enqueue 1.76694 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25177 ms - Host latency: 2.40238 ms (end to end 2.4111 ms, enqueue 1.75881 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25254 ms - Host latency: 2.40269 ms (end to end 2.41267 ms, enqueue 1.7373 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25317 ms - Host latency: 2.40366 ms (end to end 2.41406 ms, enqueue 1.73696 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25798 ms - Host latency: 2.40859 ms (end to end 2.41958 ms, enqueue 1.73608 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25439 ms - Host latency: 2.40486 ms (end to end 2.41594 ms, enqueue 1.74563 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25576 ms - Host latency: 2.40613 ms (end to end 2.41841 ms, enqueue 1.74836 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25425 ms - Host latency: 2.40442 ms (end to end 2.41643 ms, enqueue 1.73916 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25684 ms - Host latency: 2.40657 ms (end to end 2.41794 ms, enqueue 1.7426 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25437 ms - Host latency: 2.40488 ms (end to end 2.4168 ms, enqueue 1.75242 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25513 ms - Host latency: 2.40569 ms (end to end 2.41821 ms, enqueue 1.75112 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25232 ms - Host latency: 2.40273 ms (end to end 2.41416 ms, enqueue 1.73694 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25386 ms - Host latency: 2.40408 ms (end to end 2.41458 ms, enqueue 1.73843 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25781 ms - Host latency: 2.40811 ms (end to end 2.4177 ms, enqueue 1.7366 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25181 ms - Host latency: 2.40237 ms (end to end 2.41279 ms, enqueue 1.74817 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25569 ms - Host latency: 2.40625 ms (end to end 2.41577 ms, enqueue 1.75332 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.2541 ms - Host latency: 2.40457 ms (end to end 2.41433 ms, enqueue 1.74927 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25381 ms - Host latency: 2.40427 ms (end to end 2.41543 ms, enqueue 1.74878 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25259 ms - Host latency: 2.40334 ms (end to end 2.41555 ms, enqueue 1.7395 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25225 ms - Host latency: 2.40391 ms (end to end 2.41448 ms, enqueue 1.76213 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25735 ms - Host latency: 2.4082 ms (end to end 2.4207 ms, enqueue 1.72915 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25342 ms - Host latency: 2.40403 ms (end to end 2.41504 ms, enqueue 1.74084 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.2521 ms - Host latency: 2.40249 ms (end to end 2.41189 ms, enqueue 1.72957 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25405 ms - Host latency: 2.40413 ms (end to end 2.41523 ms, enqueue 1.75027 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.253 ms - Host latency: 2.40325 ms (end to end 2.41396 ms, enqueue 1.76191 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25312 ms - Host latency: 2.40369 ms (end to end 2.41594 ms, enqueue 1.75288 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25488 ms - Host latency: 2.40493 ms (end to end 2.41616 ms, enqueue 1.73599 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25244 ms - Host latency: 2.40261 ms (end to end 2.41406 ms, enqueue 1.74785 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25247 ms - Host latency: 2.40271 ms (end to end 2.4123 ms, enqueue 1.74941 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25349 ms - Host latency: 2.40381 ms (end to end 2.4145 ms, enqueue 1.73735 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25317 ms - Host latency: 2.40337 ms (end to end 2.41455 ms, enqueue 1.73025 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25696 ms - Host latency: 2.40908 ms (end to end 2.42207 ms, enqueue 1.7481 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25564 ms - Host latency: 2.40591 ms (end to end 2.41711 ms, enqueue 1.74148 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25603 ms - Host latency: 2.40649 ms (end to end 2.41675 ms, enqueue 1.73677 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25195 ms - Host latency: 2.40244 ms (end to end 2.41289 ms, enqueue 1.74548 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25566 ms - Host latency: 2.40623 ms (end to end 2.41675 ms, enqueue 1.73672 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25596 ms - Host latency: 2.40645 ms (end to end 2.41763 ms, enqueue 1.73181 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25286 ms - Host latency: 2.4033 ms (end to end 2.41104 ms, enqueue 1.77761 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25798 ms - Host latency: 2.40911 ms (end to end 2.4209 ms, enqueue 1.74211 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25515 ms - Host latency: 2.40562 ms (end to end 2.41538 ms, enqueue 1.7384 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25576 ms - Host latency: 2.40645 ms (end to end 2.41731 ms, enqueue 1.7376 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.2562 ms - Host latency: 2.40659 ms (end to end 2.41934 ms, enqueue 1.72773 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.2523 ms - Host latency: 2.40244 ms (end to end 2.41362 ms, enqueue 1.74702 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25496 ms - Host latency: 2.40642 ms (end to end 2.41836 ms, enqueue 1.74224 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25393 ms - Host latency: 2.40469 ms (end to end 2.41597 ms, enqueue 1.73105 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25591 ms - Host latency: 2.40603 ms (end to end 2.41526 ms, enqueue 1.73481 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25327 ms - Host latency: 2.40413 ms (end to end 2.41545 ms, enqueue 1.73191 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25342 ms - Host latency: 2.4041 ms (end to end 2.41428 ms, enqueue 1.75886 ms) [09/25/2021-10:28:55] [I] Average on 10 runs - GPU latency: 2.25532 ms - Host latency: 2.40591 ms (end to end 2.41707 ms, enqueue 1.73611 ms) [09/25/2021-10:28:55] [I] [09/25/2021-10:28:55] [I] === Performance summary === [09/25/2021-10:28:55] [I] Throughput: 413.913 qps [09/25/2021-10:28:55] [I] Latency: min = 2.37933 ms, max = 2.49512 ms, mean = 2.40421 ms, median = 2.40414 ms, percentile(99%) = 2.4187 ms [09/25/2021-10:28:55] [I] End-to-End Host Latency: min = 2.38696 ms, max = 2.50586 ms, mean = 2.41509 ms, median = 2.41504 ms, percentile(99%) = 2.43213 ms [09/25/2021-10:28:55] [I] Enqueue Time: min = 1.66211 ms, max = 2.1673 ms, mean = 1.75936 ms, median = 1.75409 ms, percentile(99%) = 1.91425 ms [09/25/2021-10:28:55] [I] H2D Latency: min = 0.114502 ms, max = 0.12793 ms, mean = 0.116151 ms, median = 0.115967 ms, percentile(99%) = 0.119873 ms [09/25/2021-10:28:55] [I] GPU Compute Time: min = 2.22925 ms, max = 2.3418 ms, mean = 2.25368 ms, median = 2.25385 ms, percentile(99%) = 2.26611 ms [09/25/2021-10:28:55] [I] D2H Latency: min = 0.0322266 ms, max = 0.0371094 ms, mean = 0.034373 ms, median = 0.0341797 ms, percentile(99%) = 0.0361328 ms [09/25/2021-10:28:55] [I] Total Host Walltime: 3.00547 s [09/25/2021-10:28:55] [I] Total GPU Compute Time: 2.80358 s [09/25/2021-10:28:55] [I] Explanations of the performance metrics are printed in the verbose logs. [09/25/2021-10:28:55] [I] &&&& PASSED TensorRT.trtexec [TensorRT v8001] # /usr/src/tensorrt/bin/trtexec --onnx=/home/acer/nfs-share/epoch_250.onnx --int8 [09/25/2021-10:28:55] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1381, GPU 5460 (MiB)