I am running infrence using deepstream_tao_apps
I use this command to find the infrence speed of my custom model built using transfer learning toolkit
$ /usr/src/tensorrt/bin/trtexec --batch=1 --useSpinWait --loadEngine=/home/vaaan/Downloads/deepstream_tao_apps/models/yolov4/yolov4_resnet18_epoch_080.etlt_b1_gpu0_fp16.engine
as a result i get this log
&&&& RUNNING TensorRT.trtexec [TensorRT v8001] # /usr/src/tensorrt/bin/trtexec --batch=1 --useSpinWait --loadEngine=/home/vaaan/Downloads/deepstream_tao_apps/models/yolov4/yolov4_resnet18_epoch_080.etlt_b1_gpu0_fp16.engine
[11/24/2021-14:27:11] [I] === Model Options ===
[11/24/2021-14:27:11] [I] Format: *
[11/24/2021-14:27:11] [I] Model:
[11/24/2021-14:27:11] [I] Output:
[11/24/2021-14:27:11] [I] === Build Options ===
[11/24/2021-14:27:11] [I] Max batch: 1
[11/24/2021-14:27:11] [I] Workspace: 16 MiB
[11/24/2021-14:27:11] [I] minTiming: 1
[11/24/2021-14:27:11] [I] avgTiming: 8
[11/24/2021-14:27:11] [I] Precision: FP32
[11/24/2021-14:27:11] [I] Calibration:
[11/24/2021-14:27:11] [I] Refit: Disabled
[11/24/2021-14:27:11] [I] Sparsity: Disabled
[11/24/2021-14:27:11] [I] Safe mode: Disabled
[11/24/2021-14:27:11] [I] Restricted mode: Disabled
[11/24/2021-14:27:11] [I] Save engine:
[11/24/2021-14:27:11] [I] Load engine: /home/vaaan/Downloads/deepstream_tao_apps/models/yolov4/yolov4_resnet18_epoch_080.etlt_b1_gpu0_fp16.engine
[11/24/2021-14:27:11] [I] NVTX verbosity: 0
[11/24/2021-14:27:11] [I] Tactic sources: Using default tactic sources
[11/24/2021-14:27:11] [I] timingCacheMode: local
[11/24/2021-14:27:11] [I] timingCacheFile:
[11/24/2021-14:27:11] [I] Input(s)s format: fp32:CHW
[11/24/2021-14:27:11] [I] Output(s)s format: fp32:CHW
[11/24/2021-14:27:11] [I] Input build shapes: model
[11/24/2021-14:27:11] [I] Input calibration shapes: model
[11/24/2021-14:27:11] [I] === System Options ===
[11/24/2021-14:27:11] [I] Device: 0
[11/24/2021-14:27:11] [I] DLACore:
[11/24/2021-14:27:11] [I] Plugins:
[11/24/2021-14:27:11] [I] === Inference Options ===
[11/24/2021-14:27:11] [I] Batch: 1
[11/24/2021-14:27:11] [I] Input inference shapes: model
[11/24/2021-14:27:11] [I] Iterations: 10
[11/24/2021-14:27:11] [I] Duration: 3s (+ 200ms warm up)
[11/24/2021-14:27:11] [I] Sleep time: 0ms
[11/24/2021-14:27:11] [I] Streams: 1
[11/24/2021-14:27:11] [I] ExposeDMA: Disabled
[11/24/2021-14:27:11] [I] Data transfers: Enabled
[11/24/2021-14:27:11] [I] Spin-wait: Enabled
[11/24/2021-14:27:11] [I] Multithreading: Disabled
[11/24/2021-14:27:11] [I] CUDA Graph: Disabled
[11/24/2021-14:27:11] [I] Separate profiling: Disabled
[11/24/2021-14:27:11] [I] Time Deserialize: Disabled
[11/24/2021-14:27:11] [I] Time Refit: Disabled
[11/24/2021-14:27:11] [I] Skip inference: Disabled
[11/24/2021-14:27:11] [I] Inputs:
[11/24/2021-14:27:11] [I] === Reporting Options ===
[11/24/2021-14:27:11] [I] Verbose: Disabled
[11/24/2021-14:27:11] [I] Averages: 10 inferences
[11/24/2021-14:27:11] [I] Percentile: 99
[11/24/2021-14:27:11] [I] Dump refittable layers:Disabled
[11/24/2021-14:27:11] [I] Dump output: Disabled
[11/24/2021-14:27:11] [I] Profile: Disabled
[11/24/2021-14:27:11] [I] Export timing to JSON file:
[11/24/2021-14:27:11] [I] Export output to JSON file:
[11/24/2021-14:27:11] [I] Export profile to JSON file:
[11/24/2021-14:27:11] [I]
[11/24/2021-14:27:12] [I] === Device Information ===
[11/24/2021-14:27:12] [I] Selected Device: Quadro RTX 5000
[11/24/2021-14:27:12] [I] Compute Capability: 7.5
[11/24/2021-14:27:12] [I] SMs: 48
[11/24/2021-14:27:12] [I] Compute Clock Rate: 1.815 GHz
[11/24/2021-14:27:12] [I] Device Global Memory: 16125 MiB
[11/24/2021-14:27:12] [I] Shared Memory per SM: 64 KiB
[11/24/2021-14:27:12] [I] Memory Bus Width: 256 bits (ECC disabled)
[11/24/2021-14:27:12] [I] Memory Clock Rate: 7.001 GHz
[11/24/2021-14:27:12] [I]
[11/24/2021-14:27:12] [I] TensorRT version: 8001
[11/24/2021-14:27:12] [I] [TRT] [MemUsageChange] Init CUDA: CPU +330, GPU +0, now: CPU 342, GPU 271 (MiB)
[11/24/2021-14:27:12] [I] [TRT] Loaded engine size: 4 MB
[11/24/2021-14:27:12] [I] [TRT] [MemUsageSnapshot] deserializeCudaEngine begin: CPU 342 MiB, GPU 271 MiB
[11/24/2021-14:27:12] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +493, GPU +210, now: CPU 838, GPU 483 (MiB)
[11/24/2021-14:27:13] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +425, GPU +168, now: CPU 1263, GPU 651 (MiB)
[11/24/2021-14:27:13] [W] [TRT] TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.1.0
[11/24/2021-14:27:13] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1263, GPU 635 (MiB)
[11/24/2021-14:27:13] [I] [TRT] [MemUsageSnapshot] deserializeCudaEngine end: CPU 1263 MiB, GPU 635 MiB
[11/24/2021-14:27:13] [I] Engine loaded in 1.05656 sec.
[11/24/2021-14:27:13] [I] [TRT] [MemUsageSnapshot] ExecutionContext creation begin: CPU 1258 MiB, GPU 635 MiB
[11/24/2021-14:27:13] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 1258, GPU 643 (MiB)
[11/24/2021-14:27:13] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +1, GPU +8, now: CPU 1259, GPU 651 (MiB)
[11/24/2021-14:27:13] [W] [TRT] TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.1.0
[11/24/2021-14:27:13] [I] [TRT] [MemUsageSnapshot] ExecutionContext creation end: CPU 1259 MiB, GPU 665 MiB
[11/24/2021-14:27:13] [I] Created input binding for Input with dimensions 1x3x608x608
[11/24/2021-14:27:13] [I] Created output binding for BatchedNMS with dimensions 1x1
[11/24/2021-14:27:13] [I] Created output binding for BatchedNMS_1 with dimensions 1x200x4
[11/24/2021-14:27:13] [I] Created output binding for BatchedNMS_2 with dimensions 1x200
[11/24/2021-14:27:13] [I] Created output binding for BatchedNMS_3 with dimensions 1x200
[11/24/2021-14:27:13] [I] Starting inference
[11/24/2021-14:27:16] [I] Warmup completed 192 queries over 200 ms
[11/24/2021-14:27:16] [I] Timing trace has 3763 queries over 3.00217 s
[11/24/2021-14:27:16] [I]
[11/24/2021-14:27:16] [I] === Trace details ===
[11/24/2021-14:27:16] [I] Trace averages of 10 runs:
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.944611 ms - Host latency: 1.32908 ms (end to end 1.88051 ms, enqueue 0.418878 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.840288 ms - Host latency: 1.22707 ms (end to end 1.68802 ms, enqueue 0.423877 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.792784 ms - Host latency: 1.17908 ms (end to end 1.57767 ms, enqueue 0.419191 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.791817 ms - Host latency: 1.17754 ms (end to end 1.57319 ms, enqueue 0.415685 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.792877 ms - Host latency: 1.1808 ms (end to end 1.57878 ms, enqueue 0.429143 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.791098 ms - Host latency: 1.17755 ms (end to end 1.57238 ms, enqueue 0.429158 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.793819 ms - Host latency: 1.18167 ms (end to end 1.57762 ms, enqueue 0.428886 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.792923 ms - Host latency: 1.17973 ms (end to end 1.57637 ms, enqueue 0.43429 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.792636 ms - Host latency: 1.17969 ms (end to end 1.57725 ms, enqueue 0.426642 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794031 ms - Host latency: 1.18276 ms (end to end 1.57835 ms, enqueue 0.431015 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794989 ms - Host latency: 1.18472 ms (end to end 1.58167 ms, enqueue 0.4328 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.793454 ms - Host latency: 1.1819 ms (end to end 1.57896 ms, enqueue 0.425293 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.792163 ms - Host latency: 1.17823 ms (end to end 1.57575 ms, enqueue 0.420944 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794006 ms - Host latency: 1.18234 ms (end to end 1.57954 ms, enqueue 0.420126 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.792709 ms - Host latency: 1.1801 ms (end to end 1.57624 ms, enqueue 0.435449 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79241 ms - Host latency: 1.17958 ms (end to end 1.57704 ms, enqueue 0.422586 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.792181 ms - Host latency: 1.17972 ms (end to end 1.57625 ms, enqueue 0.424414 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.791489 ms - Host latency: 1.17833 ms (end to end 1.57542 ms, enqueue 0.427515 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.792688 ms - Host latency: 1.18012 ms (end to end 1.57567 ms, enqueue 0.426535 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79267 ms - Host latency: 1.17901 ms (end to end 1.57625 ms, enqueue 0.429987 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79187 ms - Host latency: 1.17963 ms (end to end 1.57628 ms, enqueue 0.428659 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.793259 ms - Host latency: 1.18067 ms (end to end 1.57732 ms, enqueue 0.424661 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.791223 ms - Host latency: 1.17648 ms (end to end 1.57461 ms, enqueue 0.424826 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.792325 ms - Host latency: 1.18008 ms (end to end 1.57662 ms, enqueue 0.420914 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.791888 ms - Host latency: 1.17816 ms (end to end 1.57501 ms, enqueue 0.421271 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.793158 ms - Host latency: 1.17958 ms (end to end 1.57685 ms, enqueue 0.424136 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.791974 ms - Host latency: 1.1792 ms (end to end 1.57584 ms, enqueue 0.420908 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.793164 ms - Host latency: 1.18133 ms (end to end 1.57694 ms, enqueue 0.427667 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79147 ms - Host latency: 1.17917 ms (end to end 1.57471 ms, enqueue 0.41925 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.793027 ms - Host latency: 1.18077 ms (end to end 1.57733 ms, enqueue 0.421774 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794284 ms - Host latency: 1.18076 ms (end to end 1.57795 ms, enqueue 0.430701 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.793002 ms - Host latency: 1.18053 ms (end to end 1.57727 ms, enqueue 0.427145 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796289 ms - Host latency: 1.18383 ms (end to end 1.58237 ms, enqueue 0.429419 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.799747 ms - Host latency: 1.19036 ms (end to end 1.59085 ms, enqueue 0.431442 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79877 ms - Host latency: 1.18761 ms (end to end 1.59053 ms, enqueue 0.424738 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797665 ms - Host latency: 1.18365 ms (end to end 1.58705 ms, enqueue 0.42597 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.799445 ms - Host latency: 1.18687 ms (end to end 1.59157 ms, enqueue 0.422858 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797934 ms - Host latency: 1.18501 ms (end to end 1.58746 ms, enqueue 0.419907 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796445 ms - Host latency: 1.18266 ms (end to end 1.5839 ms, enqueue 0.421835 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797052 ms - Host latency: 1.18448 ms (end to end 1.58633 ms, enqueue 0.420651 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797772 ms - Host latency: 1.18612 ms (end to end 1.58408 ms, enqueue 0.429633 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.798059 ms - Host latency: 1.18413 ms (end to end 1.58588 ms, enqueue 0.418884 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.80022 ms - Host latency: 1.18753 ms (end to end 1.59034 ms, enqueue 0.429291 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.799841 ms - Host latency: 1.18699 ms (end to end 1.59012 ms, enqueue 0.428101 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.798975 ms - Host latency: 1.18563 ms (end to end 1.58929 ms, enqueue 0.427808 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.799829 ms - Host latency: 1.18679 ms (end to end 1.59156 ms, enqueue 0.42998 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797491 ms - Host latency: 1.18369 ms (end to end 1.58665 ms, enqueue 0.427539 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797803 ms - Host latency: 1.18485 ms (end to end 1.58646 ms, enqueue 0.423132 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797137 ms - Host latency: 1.18445 ms (end to end 1.58593 ms, enqueue 0.428833 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796417 ms - Host latency: 1.18473 ms (end to end 1.58469 ms, enqueue 0.418579 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796185 ms - Host latency: 1.18488 ms (end to end 1.58442 ms, enqueue 0.422748 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79729 ms - Host latency: 1.18555 ms (end to end 1.58671 ms, enqueue 0.42309 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.801678 ms - Host latency: 1.19247 ms (end to end 1.59384 ms, enqueue 0.435175 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797058 ms - Host latency: 1.18553 ms (end to end 1.58774 ms, enqueue 0.424219 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796387 ms - Host latency: 1.18497 ms (end to end 1.58441 ms, enqueue 0.43244 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797424 ms - Host latency: 1.18563 ms (end to end 1.58651 ms, enqueue 0.433252 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796143 ms - Host latency: 1.18487 ms (end to end 1.58429 ms, enqueue 0.435937 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.799231 ms - Host latency: 1.18875 ms (end to end 1.5902 ms, enqueue 0.43219 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796381 ms - Host latency: 1.18241 ms (end to end 1.58482 ms, enqueue 0.426917 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794281 ms - Host latency: 1.18033 ms (end to end 1.58018 ms, enqueue 0.432953 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795764 ms - Host latency: 1.18199 ms (end to end 1.5834 ms, enqueue 0.438159 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795062 ms - Host latency: 1.1815 ms (end to end 1.58151 ms, enqueue 0.424982 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797449 ms - Host latency: 1.1847 ms (end to end 1.58657 ms, enqueue 0.421338 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796442 ms - Host latency: 1.18592 ms (end to end 1.58423 ms, enqueue 0.424561 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794012 ms - Host latency: 1.1798 ms (end to end 1.58044 ms, enqueue 0.419629 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79483 ms - Host latency: 1.18048 ms (end to end 1.58051 ms, enqueue 0.421552 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796515 ms - Host latency: 1.18371 ms (end to end 1.58408 ms, enqueue 0.420148 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.798218 ms - Host latency: 1.18635 ms (end to end 1.58908 ms, enqueue 0.434637 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796942 ms - Host latency: 1.18433 ms (end to end 1.58528 ms, enqueue 0.432385 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.798096 ms - Host latency: 1.18705 ms (end to end 1.58685 ms, enqueue 0.434784 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796149 ms - Host latency: 1.18247 ms (end to end 1.58438 ms, enqueue 0.434448 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794623 ms - Host latency: 1.18312 ms (end to end 1.58065 ms, enqueue 0.426843 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796155 ms - Host latency: 1.18486 ms (end to end 1.58416 ms, enqueue 0.427905 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795349 ms - Host latency: 1.18275 ms (end to end 1.58264 ms, enqueue 0.422321 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795392 ms - Host latency: 1.18203 ms (end to end 1.58159 ms, enqueue 0.417334 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796082 ms - Host latency: 1.18402 ms (end to end 1.58466 ms, enqueue 0.417053 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794861 ms - Host latency: 1.18124 ms (end to end 1.58029 ms, enqueue 0.417419 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797437 ms - Host latency: 1.1861 ms (end to end 1.58668 ms, enqueue 0.428497 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79483 ms - Host latency: 1.18044 ms (end to end 1.58152 ms, enqueue 0.420599 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795184 ms - Host latency: 1.18113 ms (end to end 1.58135 ms, enqueue 0.41958 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794769 ms - Host latency: 1.18113 ms (end to end 1.58109 ms, enqueue 0.42854 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794507 ms - Host latency: 1.1804 ms (end to end 1.57917 ms, enqueue 0.428912 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797675 ms - Host latency: 1.18667 ms (end to end 1.58747 ms, enqueue 0.430511 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797992 ms - Host latency: 1.1855 ms (end to end 1.58589 ms, enqueue 0.436127 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79455 ms - Host latency: 1.18156 ms (end to end 1.58139 ms, enqueue 0.425153 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796985 ms - Host latency: 1.18507 ms (end to end 1.58488 ms, enqueue 0.430908 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795941 ms - Host latency: 1.18391 ms (end to end 1.58321 ms, enqueue 0.422418 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794464 ms - Host latency: 1.18005 ms (end to end 1.58087 ms, enqueue 0.421619 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797662 ms - Host latency: 1.18589 ms (end to end 1.58683 ms, enqueue 0.422162 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796497 ms - Host latency: 1.18473 ms (end to end 1.58492 ms, enqueue 0.424615 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795581 ms - Host latency: 1.18312 ms (end to end 1.58319 ms, enqueue 0.419934 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795129 ms - Host latency: 1.18182 ms (end to end 1.58176 ms, enqueue 0.419135 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794708 ms - Host latency: 1.18244 ms (end to end 1.58063 ms, enqueue 0.423981 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795337 ms - Host latency: 1.18194 ms (end to end 1.58101 ms, enqueue 0.435712 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794269 ms - Host latency: 1.18258 ms (end to end 1.58152 ms, enqueue 0.431714 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795776 ms - Host latency: 1.18466 ms (end to end 1.58274 ms, enqueue 0.433038 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.798761 ms - Host latency: 1.18652 ms (end to end 1.58719 ms, enqueue 0.430902 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796863 ms - Host latency: 1.18466 ms (end to end 1.58537 ms, enqueue 0.420691 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796106 ms - Host latency: 1.18537 ms (end to end 1.58561 ms, enqueue 0.431305 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795929 ms - Host latency: 1.18316 ms (end to end 1.58376 ms, enqueue 0.416827 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796582 ms - Host latency: 1.18486 ms (end to end 1.58527 ms, enqueue 0.424536 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794641 ms - Host latency: 1.18008 ms (end to end 1.58076 ms, enqueue 0.420428 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795813 ms - Host latency: 1.18353 ms (end to end 1.58407 ms, enqueue 0.420734 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795227 ms - Host latency: 1.18173 ms (end to end 1.58243 ms, enqueue 0.418677 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794495 ms - Host latency: 1.18007 ms (end to end 1.57976 ms, enqueue 0.422644 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795532 ms - Host latency: 1.18269 ms (end to end 1.58322 ms, enqueue 0.432751 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795508 ms - Host latency: 1.18306 ms (end to end 1.5829 ms, enqueue 0.42948 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795801 ms - Host latency: 1.18121 ms (end to end 1.58173 ms, enqueue 0.430481 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79668 ms - Host latency: 1.18429 ms (end to end 1.58544 ms, enqueue 0.429138 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.798364 ms - Host latency: 1.18936 ms (end to end 1.58799 ms, enqueue 0.426685 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.799402 ms - Host latency: 1.19045 ms (end to end 1.58961 ms, enqueue 0.430688 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796594 ms - Host latency: 1.18438 ms (end to end 1.5833 ms, enqueue 0.424231 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79679 ms - Host latency: 1.18423 ms (end to end 1.58417 ms, enqueue 0.419971 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.7953 ms - Host latency: 1.18322 ms (end to end 1.5825 ms, enqueue 0.425378 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795935 ms - Host latency: 1.18416 ms (end to end 1.58546 ms, enqueue 0.41936 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795203 ms - Host latency: 1.18412 ms (end to end 1.58347 ms, enqueue 0.42196 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795813 ms - Host latency: 1.18351 ms (end to end 1.58176 ms, enqueue 0.420593 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796179 ms - Host latency: 1.18376 ms (end to end 1.58278 ms, enqueue 0.428711 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795251 ms - Host latency: 1.18184 ms (end to end 1.58225 ms, enqueue 0.42843 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795667 ms - Host latency: 1.18248 ms (end to end 1.58251 ms, enqueue 0.428003 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794775 ms - Host latency: 1.18063 ms (end to end 1.58007 ms, enqueue 0.427344 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796436 ms - Host latency: 1.18517 ms (end to end 1.58323 ms, enqueue 0.427759 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.798633 ms - Host latency: 1.18899 ms (end to end 1.58849 ms, enqueue 0.422864 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797791 ms - Host latency: 1.18938 ms (end to end 1.58805 ms, enqueue 0.437585 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795703 ms - Host latency: 1.18301 ms (end to end 1.58239 ms, enqueue 0.421863 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796423 ms - Host latency: 1.18295 ms (end to end 1.58357 ms, enqueue 0.424097 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794275 ms - Host latency: 1.18116 ms (end to end 1.58019 ms, enqueue 0.421423 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795862 ms - Host latency: 1.18276 ms (end to end 1.58197 ms, enqueue 0.418762 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795142 ms - Host latency: 1.18145 ms (end to end 1.58247 ms, enqueue 0.417932 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794739 ms - Host latency: 1.1811 ms (end to end 1.58071 ms, enqueue 0.419006 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796069 ms - Host latency: 1.18414 ms (end to end 1.58483 ms, enqueue 0.425879 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794421 ms - Host latency: 1.18268 ms (end to end 1.58018 ms, enqueue 0.422839 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795972 ms - Host latency: 1.1842 ms (end to end 1.58315 ms, enqueue 0.428357 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.793335 ms - Host latency: 1.17953 ms (end to end 1.57739 ms, enqueue 0.427112 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796985 ms - Host latency: 1.18494 ms (end to end 1.58492 ms, enqueue 0.42373 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795667 ms - Host latency: 1.18658 ms (end to end 1.5859 ms, enqueue 0.429346 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.799365 ms - Host latency: 1.19025 ms (end to end 1.58976 ms, enqueue 0.428369 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797205 ms - Host latency: 1.18452 ms (end to end 1.58381 ms, enqueue 0.423743 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79646 ms - Host latency: 1.18417 ms (end to end 1.58291 ms, enqueue 0.431018 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79679 ms - Host latency: 1.18654 ms (end to end 1.58569 ms, enqueue 0.421899 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794446 ms - Host latency: 1.1822 ms (end to end 1.58066 ms, enqueue 0.428296 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794519 ms - Host latency: 1.1818 ms (end to end 1.57983 ms, enqueue 0.426819 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797424 ms - Host latency: 1.18732 ms (end to end 1.58616 ms, enqueue 0.432874 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796252 ms - Host latency: 1.18568 ms (end to end 1.58418 ms, enqueue 0.430493 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795825 ms - Host latency: 1.18527 ms (end to end 1.58286 ms, enqueue 0.430286 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797278 ms - Host latency: 1.18848 ms (end to end 1.58682 ms, enqueue 0.432861 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796143 ms - Host latency: 1.18534 ms (end to end 1.5839 ms, enqueue 0.430493 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.7974 ms - Host latency: 1.18436 ms (end to end 1.58629 ms, enqueue 0.423254 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79613 ms - Host latency: 1.18452 ms (end to end 1.58564 ms, enqueue 0.429419 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794617 ms - Host latency: 1.18113 ms (end to end 1.57986 ms, enqueue 0.420569 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794995 ms - Host latency: 1.18118 ms (end to end 1.58134 ms, enqueue 0.419519 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795032 ms - Host latency: 1.18309 ms (end to end 1.58239 ms, enqueue 0.426355 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795203 ms - Host latency: 1.18184 ms (end to end 1.58053 ms, enqueue 0.420093 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794897 ms - Host latency: 1.18159 ms (end to end 1.5847 ms, enqueue 0.421509 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795691 ms - Host latency: 1.18228 ms (end to end 1.5825 ms, enqueue 0.417065 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794373 ms - Host latency: 1.18241 ms (end to end 1.58054 ms, enqueue 0.428967 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794836 ms - Host latency: 1.18163 ms (end to end 1.58041 ms, enqueue 0.429468 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795313 ms - Host latency: 1.18347 ms (end to end 1.58344 ms, enqueue 0.427979 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796228 ms - Host latency: 1.18357 ms (end to end 1.58356 ms, enqueue 0.431995 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796985 ms - Host latency: 1.18402 ms (end to end 1.58474 ms, enqueue 0.425427 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797815 ms - Host latency: 1.18645 ms (end to end 1.58652 ms, enqueue 0.4276 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796631 ms - Host latency: 1.18463 ms (end to end 1.58568 ms, enqueue 0.425024 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795728 ms - Host latency: 1.18264 ms (end to end 1.58269 ms, enqueue 0.419727 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794202 ms - Host latency: 1.18069 ms (end to end 1.57994 ms, enqueue 0.419836 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797864 ms - Host latency: 1.18545 ms (end to end 1.58776 ms, enqueue 0.41897 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796252 ms - Host latency: 1.18311 ms (end to end 1.58071 ms, enqueue 0.417273 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795337 ms - Host latency: 1.18258 ms (end to end 1.58261 ms, enqueue 0.420801 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796692 ms - Host latency: 1.18564 ms (end to end 1.58477 ms, enqueue 0.425183 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79469 ms - Host latency: 1.18088 ms (end to end 1.58042 ms, enqueue 0.427942 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795911 ms - Host latency: 1.18354 ms (end to end 1.58365 ms, enqueue 0.428687 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795679 ms - Host latency: 1.18174 ms (end to end 1.58231 ms, enqueue 0.437683 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796814 ms - Host latency: 1.18501 ms (end to end 1.58398 ms, enqueue 0.43335 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797778 ms - Host latency: 1.18807 ms (end to end 1.58896 ms, enqueue 0.432031 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797046 ms - Host latency: 1.18381 ms (end to end 1.58601 ms, enqueue 0.425659 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795251 ms - Host latency: 1.18103 ms (end to end 1.58145 ms, enqueue 0.421753 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796265 ms - Host latency: 1.18375 ms (end to end 1.58407 ms, enqueue 0.425586 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797046 ms - Host latency: 1.1837 ms (end to end 1.58474 ms, enqueue 0.42793 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794934 ms - Host latency: 1.18301 ms (end to end 1.58315 ms, enqueue 0.423291 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796057 ms - Host latency: 1.18479 ms (end to end 1.58455 ms, enqueue 0.421741 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795483 ms - Host latency: 1.18251 ms (end to end 1.5825 ms, enqueue 0.424084 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795837 ms - Host latency: 1.1828 ms (end to end 1.58132 ms, enqueue 0.425403 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795044 ms - Host latency: 1.18232 ms (end to end 1.58263 ms, enqueue 0.430786 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795215 ms - Host latency: 1.1827 ms (end to end 1.58124 ms, enqueue 0.423926 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796252 ms - Host latency: 1.18278 ms (end to end 1.58273 ms, enqueue 0.426709 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796899 ms - Host latency: 1.18518 ms (end to end 1.58635 ms, enqueue 0.436926 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797559 ms - Host latency: 1.18383 ms (end to end 1.58262 ms, enqueue 0.430457 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795557 ms - Host latency: 1.18314 ms (end to end 1.58436 ms, enqueue 0.426843 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794128 ms - Host latency: 1.18019 ms (end to end 1.57893 ms, enqueue 0.417163 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795642 ms - Host latency: 1.18257 ms (end to end 1.58264 ms, enqueue 0.418872 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794214 ms - Host latency: 1.18158 ms (end to end 1.58168 ms, enqueue 0.419202 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796411 ms - Host latency: 1.1842 ms (end to end 1.58451 ms, enqueue 0.419641 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794543 ms - Host latency: 1.18053 ms (end to end 1.58201 ms, enqueue 0.41665 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794495 ms - Host latency: 1.17977 ms (end to end 1.58064 ms, enqueue 0.417773 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796436 ms - Host latency: 1.18496 ms (end to end 1.58341 ms, enqueue 0.432288 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796594 ms - Host latency: 1.18434 ms (end to end 1.58363 ms, enqueue 0.427832 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.798096 ms - Host latency: 1.18683 ms (end to end 1.58728 ms, enqueue 0.43512 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795959 ms - Host latency: 1.18604 ms (end to end 1.58555 ms, enqueue 0.438001 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794836 ms - Host latency: 1.18073 ms (end to end 1.58099 ms, enqueue 0.427808 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795251 ms - Host latency: 1.18376 ms (end to end 1.58271 ms, enqueue 0.422168 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795386 ms - Host latency: 1.18213 ms (end to end 1.58032 ms, enqueue 0.42229 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794214 ms - Host latency: 1.17986 ms (end to end 1.58018 ms, enqueue 0.421338 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794043 ms - Host latency: 1.17991 ms (end to end 1.58008 ms, enqueue 0.423083 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795508 ms - Host latency: 1.18229 ms (end to end 1.583 ms, enqueue 0.417529 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79762 ms - Host latency: 1.18716 ms (end to end 1.58678 ms, enqueue 0.423975 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796045 ms - Host latency: 1.18425 ms (end to end 1.58402 ms, enqueue 0.419788 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79541 ms - Host latency: 1.18286 ms (end to end 1.58174 ms, enqueue 0.427454 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796436 ms - Host latency: 1.18562 ms (end to end 1.58567 ms, enqueue 0.426294 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795117 ms - Host latency: 1.1823 ms (end to end 1.58192 ms, enqueue 0.429199 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796631 ms - Host latency: 1.18375 ms (end to end 1.5853 ms, enqueue 0.432275 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796887 ms - Host latency: 1.18502 ms (end to end 1.58468 ms, enqueue 0.43313 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796497 ms - Host latency: 1.18297 ms (end to end 1.58534 ms, enqueue 0.415637 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.7979 ms - Host latency: 1.18761 ms (end to end 1.58678 ms, enqueue 0.429797 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795532 ms - Host latency: 1.18192 ms (end to end 1.58186 ms, enqueue 0.417078 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795093 ms - Host latency: 1.18136 ms (end to end 1.5818 ms, enqueue 0.422205 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796619 ms - Host latency: 1.18428 ms (end to end 1.58505 ms, enqueue 0.42074 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796057 ms - Host latency: 1.1849 ms (end to end 1.58378 ms, enqueue 0.427405 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.793787 ms - Host latency: 1.17955 ms (end to end 1.57963 ms, enqueue 0.418396 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794861 ms - Host latency: 1.18123 ms (end to end 1.58208 ms, enqueue 0.417261 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.798206 ms - Host latency: 1.18877 ms (end to end 1.58719 ms, enqueue 0.428931 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796472 ms - Host latency: 1.1837 ms (end to end 1.5843 ms, enqueue 0.426465 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795251 ms - Host latency: 1.18156 ms (end to end 1.58086 ms, enqueue 0.430005 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794238 ms - Host latency: 1.18208 ms (end to end 1.57955 ms, enqueue 0.432568 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795459 ms - Host latency: 1.18323 ms (end to end 1.58254 ms, enqueue 0.425146 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796118 ms - Host latency: 1.18429 ms (end to end 1.58302 ms, enqueue 0.428455 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79563 ms - Host latency: 1.18125 ms (end to end 1.58191 ms, enqueue 0.419519 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796728 ms - Host latency: 1.18445 ms (end to end 1.58501 ms, enqueue 0.426062 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79491 ms - Host latency: 1.18175 ms (end to end 1.58265 ms, enqueue 0.419666 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79585 ms - Host latency: 1.18237 ms (end to end 1.58186 ms, enqueue 0.421252 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795288 ms - Host latency: 1.18267 ms (end to end 1.58357 ms, enqueue 0.423352 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796057 ms - Host latency: 1.18353 ms (end to end 1.58466 ms, enqueue 0.417883 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795764 ms - Host latency: 1.18306 ms (end to end 1.58209 ms, enqueue 0.425098 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797095 ms - Host latency: 1.1851 ms (end to end 1.5858 ms, enqueue 0.424524 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795972 ms - Host latency: 1.18518 ms (end to end 1.58259 ms, enqueue 0.435303 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795972 ms - Host latency: 1.18513 ms (end to end 1.58311 ms, enqueue 0.436206 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795117 ms - Host latency: 1.18237 ms (end to end 1.58247 ms, enqueue 0.434644 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795044 ms - Host latency: 1.18159 ms (end to end 1.58057 ms, enqueue 0.419312 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795947 ms - Host latency: 1.18401 ms (end to end 1.5844 ms, enqueue 0.424756 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794775 ms - Host latency: 1.18101 ms (end to end 1.58186 ms, enqueue 0.413184 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797607 ms - Host latency: 1.18438 ms (end to end 1.58508 ms, enqueue 0.418384 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795435 ms - Host latency: 1.18191 ms (end to end 1.58218 ms, enqueue 0.423193 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795801 ms - Host latency: 1.18142 ms (end to end 1.58276 ms, enqueue 0.419116 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79519 ms - Host latency: 1.18298 ms (end to end 1.58281 ms, enqueue 0.420166 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794946 ms - Host latency: 1.18062 ms (end to end 1.58191 ms, enqueue 0.412891 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797095 ms - Host latency: 1.18335 ms (end to end 1.58516 ms, enqueue 0.422119 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796338 ms - Host latency: 1.1833 ms (end to end 1.58508 ms, enqueue 0.422192 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797559 ms - Host latency: 1.18416 ms (end to end 1.58599 ms, enqueue 0.428442 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797705 ms - Host latency: 1.18635 ms (end to end 1.58586 ms, enqueue 0.434082 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.798706 ms - Host latency: 1.18743 ms (end to end 1.58726 ms, enqueue 0.429932 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794116 ms - Host latency: 1.18093 ms (end to end 1.57949 ms, enqueue 0.430542 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795532 ms - Host latency: 1.18286 ms (end to end 1.58335 ms, enqueue 0.416992 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794556 ms - Host latency: 1.18103 ms (end to end 1.58142 ms, enqueue 0.419702 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796777 ms - Host latency: 1.18538 ms (end to end 1.58479 ms, enqueue 0.423706 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79563 ms - Host latency: 1.18335 ms (end to end 1.58425 ms, enqueue 0.417847 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795068 ms - Host latency: 1.18093 ms (end to end 1.5813 ms, enqueue 0.422925 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79458 ms - Host latency: 1.18059 ms (end to end 1.58152 ms, enqueue 0.418115 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795044 ms - Host latency: 1.18323 ms (end to end 1.58159 ms, enqueue 0.423975 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795752 ms - Host latency: 1.18379 ms (end to end 1.58521 ms, enqueue 0.425562 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794629 ms - Host latency: 1.18062 ms (end to end 1.57883 ms, enqueue 0.423071 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.798413 ms - Host latency: 1.18823 ms (end to end 1.58835 ms, enqueue 0.436377 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.798071 ms - Host latency: 1.18828 ms (end to end 1.58831 ms, enqueue 0.435083 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795776 ms - Host latency: 1.18191 ms (end to end 1.58257 ms, enqueue 0.422974 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796436 ms - Host latency: 1.18376 ms (end to end 1.58481 ms, enqueue 0.431152 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.793726 ms - Host latency: 1.17944 ms (end to end 1.57817 ms, enqueue 0.415527 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797778 ms - Host latency: 1.18655 ms (end to end 1.58672 ms, enqueue 0.419507 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794141 ms - Host latency: 1.17954 ms (end to end 1.579 ms, enqueue 0.417603 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796631 ms - Host latency: 1.18518 ms (end to end 1.58491 ms, enqueue 0.423047 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794482 ms - Host latency: 1.18066 ms (end to end 1.58186 ms, enqueue 0.418799 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795728 ms - Host latency: 1.18196 ms (end to end 1.58159 ms, enqueue 0.423706 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.7948 ms - Host latency: 1.18132 ms (end to end 1.58225 ms, enqueue 0.425049 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.7948 ms - Host latency: 1.18149 ms (end to end 1.58152 ms, enqueue 0.422949 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79519 ms - Host latency: 1.18108 ms (end to end 1.58257 ms, enqueue 0.423315 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795386 ms - Host latency: 1.18186 ms (end to end 1.58147 ms, enqueue 0.431372 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796924 ms - Host latency: 1.18547 ms (end to end 1.58616 ms, enqueue 0.432349 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796777 ms - Host latency: 1.18286 ms (end to end 1.58384 ms, enqueue 0.427075 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795654 ms - Host latency: 1.18191 ms (end to end 1.5823 ms, enqueue 0.425146 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796728 ms - Host latency: 1.18499 ms (end to end 1.5854 ms, enqueue 0.421704 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796118 ms - Host latency: 1.18489 ms (end to end 1.58408 ms, enqueue 0.425366 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79668 ms - Host latency: 1.18474 ms (end to end 1.58508 ms, enqueue 0.423901 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795557 ms - Host latency: 1.18315 ms (end to end 1.58464 ms, enqueue 0.42478 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795874 ms - Host latency: 1.18298 ms (end to end 1.58364 ms, enqueue 0.418433 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.798315 ms - Host latency: 1.18567 ms (end to end 1.58784 ms, enqueue 0.426929 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794043 ms - Host latency: 1.18132 ms (end to end 1.57986 ms, enqueue 0.424268 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796118 ms - Host latency: 1.18347 ms (end to end 1.58401 ms, enqueue 0.426001 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794141 ms - Host latency: 1.18127 ms (end to end 1.58081 ms, enqueue 0.432251 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797876 ms - Host latency: 1.18757 ms (end to end 1.58745 ms, enqueue 0.432568 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796777 ms - Host latency: 1.18411 ms (end to end 1.58335 ms, enqueue 0.427124 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796118 ms - Host latency: 1.18721 ms (end to end 1.58479 ms, enqueue 0.435742 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796533 ms - Host latency: 1.18328 ms (end to end 1.58379 ms, enqueue 0.416821 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79646 ms - Host latency: 1.1843 ms (end to end 1.58542 ms, enqueue 0.425488 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797022 ms - Host latency: 1.18416 ms (end to end 1.58701 ms, enqueue 0.422949 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79563 ms - Host latency: 1.18171 ms (end to end 1.58333 ms, enqueue 0.418384 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795703 ms - Host latency: 1.18276 ms (end to end 1.58313 ms, enqueue 0.419946 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795166 ms - Host latency: 1.18296 ms (end to end 1.5833 ms, enqueue 0.423071 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79563 ms - Host latency: 1.18123 ms (end to end 1.58174 ms, enqueue 0.419775 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796436 ms - Host latency: 1.18438 ms (end to end 1.58369 ms, enqueue 0.424561 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796045 ms - Host latency: 1.18435 ms (end to end 1.58462 ms, enqueue 0.429419 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.798511 ms - Host latency: 1.18667 ms (end to end 1.58569 ms, enqueue 0.428564 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795044 ms - Host latency: 1.18123 ms (end to end 1.58225 ms, enqueue 0.42583 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795459 ms - Host latency: 1.18269 ms (end to end 1.58345 ms, enqueue 0.426929 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794727 ms - Host latency: 1.18149 ms (end to end 1.58206 ms, enqueue 0.423242 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795679 ms - Host latency: 1.18306 ms (end to end 1.58423 ms, enqueue 0.418823 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795557 ms - Host latency: 1.18259 ms (end to end 1.58201 ms, enqueue 0.42334 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794922 ms - Host latency: 1.18123 ms (end to end 1.58054 ms, enqueue 0.419824 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797559 ms - Host latency: 1.18535 ms (end to end 1.58379 ms, enqueue 0.426294 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794751 ms - Host latency: 1.18027 ms (end to end 1.57949 ms, enqueue 0.416675 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.793823 ms - Host latency: 1.17959 ms (end to end 1.58013 ms, enqueue 0.425586 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.793701 ms - Host latency: 1.17998 ms (end to end 1.57883 ms, enqueue 0.427173 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795264 ms - Host latency: 1.18264 ms (end to end 1.58276 ms, enqueue 0.419238 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796069 ms - Host latency: 1.18523 ms (end to end 1.58333 ms, enqueue 0.429224 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797974 ms - Host latency: 1.18665 ms (end to end 1.58765 ms, enqueue 0.436182 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796509 ms - Host latency: 1.18374 ms (end to end 1.58501 ms, enqueue 0.429175 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79502 ms - Host latency: 1.1842 ms (end to end 1.58252 ms, enqueue 0.430518 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79519 ms - Host latency: 1.18203 ms (end to end 1.58186 ms, enqueue 0.426562 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795532 ms - Host latency: 1.18203 ms (end to end 1.58267 ms, enqueue 0.423267 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796753 ms - Host latency: 1.18486 ms (end to end 1.58501 ms, enqueue 0.422339 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79585 ms - Host latency: 1.18159 ms (end to end 1.58279 ms, enqueue 0.4198 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795361 ms - Host latency: 1.1825 ms (end to end 1.58296 ms, enqueue 0.42478 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797168 ms - Host latency: 1.18367 ms (end to end 1.58589 ms, enqueue 0.423169 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.793506 ms - Host latency: 1.17991 ms (end to end 1.57859 ms, enqueue 0.422656 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797339 ms - Host latency: 1.18369 ms (end to end 1.58567 ms, enqueue 0.417505 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794312 ms - Host latency: 1.18074 ms (end to end 1.58208 ms, enqueue 0.425317 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795703 ms - Host latency: 1.18191 ms (end to end 1.58184 ms, enqueue 0.428076 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796802 ms - Host latency: 1.18501 ms (end to end 1.58494 ms, enqueue 0.430444 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79541 ms - Host latency: 1.18159 ms (end to end 1.58291 ms, enqueue 0.425024 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794873 ms - Host latency: 1.18054 ms (end to end 1.58066 ms, enqueue 0.424731 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797339 ms - Host latency: 1.18333 ms (end to end 1.58582 ms, enqueue 0.420337 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794678 ms - Host latency: 1.18232 ms (end to end 1.58184 ms, enqueue 0.423877 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795679 ms - Host latency: 1.18296 ms (end to end 1.58303 ms, enqueue 0.42146 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797192 ms - Host latency: 1.18691 ms (end to end 1.58572 ms, enqueue 0.425244 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797095 ms - Host latency: 1.18438 ms (end to end 1.58616 ms, enqueue 0.427881 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79563 ms - Host latency: 1.18289 ms (end to end 1.58225 ms, enqueue 0.424756 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.798389 ms - Host latency: 1.1896 ms (end to end 1.58816 ms, enqueue 0.43374 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796338 ms - Host latency: 1.18452 ms (end to end 1.5842 ms, enqueue 0.423291 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.799438 ms - Host latency: 1.18901 ms (end to end 1.59077 ms, enqueue 0.427466 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795752 ms - Host latency: 1.18428 ms (end to end 1.58457 ms, enqueue 0.43352 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795996 ms - Host latency: 1.18667 ms (end to end 1.58354 ms, enqueue 0.438184 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.798755 ms - Host latency: 1.18726 ms (end to end 1.58806 ms, enqueue 0.427051 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.798315 ms - Host latency: 1.18828 ms (end to end 1.5885 ms, enqueue 0.428711 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795679 ms - Host latency: 1.18262 ms (end to end 1.58313 ms, enqueue 0.421558 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796143 ms - Host latency: 1.1843 ms (end to end 1.58508 ms, enqueue 0.425 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795947 ms - Host latency: 1.18359 ms (end to end 1.58362 ms, enqueue 0.41748 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.798193 ms - Host latency: 1.18811 ms (end to end 1.58586 ms, enqueue 0.432935 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.798169 ms - Host latency: 1.18699 ms (end to end 1.58733 ms, enqueue 0.423901 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797534 ms - Host latency: 1.18645 ms (end to end 1.58711 ms, enqueue 0.427417 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.798291 ms - Host latency: 1.18848 ms (end to end 1.58838 ms, enqueue 0.424609 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795728 ms - Host latency: 1.18262 ms (end to end 1.58394 ms, enqueue 0.429199 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795996 ms - Host latency: 1.18469 ms (end to end 1.58435 ms, enqueue 0.429541 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796606 ms - Host latency: 1.18362 ms (end to end 1.58313 ms, enqueue 0.431836 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796875 ms - Host latency: 1.1845 ms (end to end 1.58494 ms, enqueue 0.430127 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.797095 ms - Host latency: 1.18657 ms (end to end 1.58606 ms, enqueue 0.427588 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795313 ms - Host latency: 1.18145 ms (end to end 1.58201 ms, enqueue 0.422144 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796167 ms - Host latency: 1.18357 ms (end to end 1.58398 ms, enqueue 0.422461 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795166 ms - Host latency: 1.18147 ms (end to end 1.58279 ms, enqueue 0.422827 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795093 ms - Host latency: 1.18242 ms (end to end 1.58201 ms, enqueue 0.41792 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795288 ms - Host latency: 1.18167 ms (end to end 1.58228 ms, enqueue 0.424902 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79541 ms - Host latency: 1.18242 ms (end to end 1.5821 ms, enqueue 0.421533 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795874 ms - Host latency: 1.18264 ms (end to end 1.58394 ms, enqueue 0.423657 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794263 ms - Host latency: 1.18018 ms (end to end 1.58018 ms, enqueue 0.426514 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79541 ms - Host latency: 1.18269 ms (end to end 1.58174 ms, enqueue 0.424268 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.794727 ms - Host latency: 1.1825 ms (end to end 1.58135 ms, enqueue 0.430786 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79646 ms - Host latency: 1.18213 ms (end to end 1.58376 ms, enqueue 0.428442 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796021 ms - Host latency: 1.18276 ms (end to end 1.58477 ms, enqueue 0.427686 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795581 ms - Host latency: 1.18247 ms (end to end 1.58208 ms, enqueue 0.429688 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796875 ms - Host latency: 1.18521 ms (end to end 1.58616 ms, enqueue 0.420068 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795288 ms - Host latency: 1.18198 ms (end to end 1.58125 ms, enqueue 0.423145 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795776 ms - Host latency: 1.18281 ms (end to end 1.58374 ms, enqueue 0.419141 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796216 ms - Host latency: 1.18259 ms (end to end 1.58381 ms, enqueue 0.422876 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795703 ms - Host latency: 1.18357 ms (end to end 1.58396 ms, enqueue 0.419165 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.79668 ms - Host latency: 1.18389 ms (end to end 1.58521 ms, enqueue 0.425195 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796265 ms - Host latency: 1.18374 ms (end to end 1.58564 ms, enqueue 0.42478 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.798462 ms - Host latency: 1.18796 ms (end to end 1.58872 ms, enqueue 0.428003 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.796265 ms - Host latency: 1.1832 ms (end to end 1.5845 ms, enqueue 0.425708 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795386 ms - Host latency: 1.18145 ms (end to end 1.58142 ms, enqueue 0.427002 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795825 ms - Host latency: 1.18296 ms (end to end 1.5824 ms, enqueue 0.430688 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795532 ms - Host latency: 1.18293 ms (end to end 1.5823 ms, enqueue 0.429175 ms)
[11/24/2021-14:27:16] [I] Average on 10 runs - GPU latency: 0.795947 ms - Host latency: 1.18328 ms (end to end 1.58389 ms, enqueue 0.418677 ms)
[11/24/2021-14:27:16] [I]
[11/24/2021-14:27:16] [I] === Performance summary ===
[11/24/2021-14:27:16] [I] Throughput: 1253.43 qps
[11/24/2021-14:27:16] [I] Latency: min = 1.16089 ms, max = 1.33911 ms, mean = 1.18384 ms, median = 1.18188 ms, percentile(99%) = 1.20386 ms
[11/24/2021-14:27:16] [I] End-to-End Host Latency: min = 1.55884 ms, max = 1.88782 ms, mean = 1.58428 ms, median = 1.58252 ms, percentile(99%) = 1.6048 ms
[11/24/2021-14:27:16] [I] Enqueue Time: min = 0.405151 ms, max = 0.486633 ms, mean = 0.425364 ms, median = 0.424683 ms, percentile(99%) = 0.464111 ms
[11/24/2021-14:27:16] [I] H2D Latency: min = 0.376022 ms, max = 0.406982 ms, mean = 0.380031 ms, median = 0.378418 ms, percentile(99%) = 0.395752 ms
[11/24/2021-14:27:16] [I] GPU Compute Time: min = 0.776367 ms, max = 0.956161 ms, mean = 0.796362 ms, median = 0.794922 ms, percentile(99%) = 0.809937 ms
[11/24/2021-14:27:16] [I] D2H Latency: min = 0.00537109 ms, max = 0.0205078 ms, mean = 0.00744834 ms, median = 0.00732422 ms, percentile(99%) = 0.010498 ms
[11/24/2021-14:27:16] [I] Total Host Walltime: 3.00217 s
[11/24/2021-14:27:16] [I] Total GPU Compute Time: 2.99671 s
[11/24/2021-14:27:16] [I] Explanations of the performance metrics are printed in the verbose logs.
[11/24/2021-14:27:16] [I]
&&&& PASSED TensorRT.trtexec [TensorRT v8001] # /usr/src/tensorrt/bin/trtexec --batch=1 --useSpinWait --loadEngine=/home/vaaan/Downloads/deepstream_tao_apps/models/yolov4/yolov4_resnet18_epoch_080.etlt_b1_gpu0_fp16.engine
[11/24/2021-14:27:16] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1280, GPU 655 (MiB)
So in this log i came to the conclusion that the inference speed
is around 1250 fps: 3763 /3.00217
[11/24/2021-14:27:13] [I] Starting inference
[11/24/2021-14:27:16] [I] Warmup completed 192 queries over 200 ms
[11/24/2021-14:27:16] [I] Timing trace has 3763 queries over 3.00217 s
So is this the inference time?
as i physically clock the model running time and dividing it with the number of input frames :
44500 frames /126s = 356 fps with GPU utilisation at 85 percent.
what is the inference time here?