&&&& RUNNING TensorRT.trtexec [TensorRT v8203] # trtexec --onnx=resnet50_quant_sparse.onnx --int8 --sparsity=enable --shapes=input:128x3x224x224
[03/25/2022-13:17:47] [I] === Model Options ===
[03/25/2022-13:17:47] [I] Format: ONNX
[03/25/2022-13:17:47] [I] Model: resnet50_quant_sparse.onnx
[03/25/2022-13:17:47] [I] Output:
[03/25/2022-13:17:47] [I] === Build Options ===
[03/25/2022-13:17:47] [I] Max batch: explicit batch
[03/25/2022-13:17:47] [I] Workspace: 16 MiB
[03/25/2022-13:17:47] [I] minTiming: 1
[03/25/2022-13:17:47] [I] avgTiming: 8
[03/25/2022-13:17:47] [I] Precision: FP32+INT8
[03/25/2022-13:17:47] [I] Calibration: Dynamic
[03/25/2022-13:17:47] [I] Refit: Disabled
[03/25/2022-13:17:47] [I] Sparsity: Enabled
[03/25/2022-13:17:47] [I] Safe mode: Disabled
[03/25/2022-13:17:47] [I] DirectIO mode: Disabled
[03/25/2022-13:17:47] [I] Restricted mode: Disabled
[03/25/2022-13:17:47] [I] Save engine: 
[03/25/2022-13:17:47] [I] Load engine: 
[03/25/2022-13:17:47] [I] Profiling verbosity: 0
[03/25/2022-13:17:47] [I] Tactic sources: Using default tactic sources
[03/25/2022-13:17:47] [I] timingCacheMode: local
[03/25/2022-13:17:47] [I] timingCacheFile: 
[03/25/2022-13:17:47] [I] Input(s)s format: fp32:CHW
[03/25/2022-13:17:47] [I] Output(s)s format: fp32:CHW
[03/25/2022-13:17:47] [I] Input build shape: input=128x3x224x224+128x3x224x224+128x3x224x224
[03/25/2022-13:17:47] [I] Input calibration shapes: model
[03/25/2022-13:17:47] [I] === System Options ===
[03/25/2022-13:17:47] [I] Device: 0
[03/25/2022-13:17:47] [I] DLACore: 
[03/25/2022-13:17:47] [I] Plugins:
[03/25/2022-13:17:47] [I] === Inference Options ===
[03/25/2022-13:17:47] [I] Batch: Explicit
[03/25/2022-13:17:47] [I] Input inference shape: input=128x3x224x224
[03/25/2022-13:17:47] [I] Iterations: 10
[03/25/2022-13:17:47] [I] Duration: 3s (+ 200ms warm up)
[03/25/2022-13:17:47] [I] Sleep time: 0ms
[03/25/2022-13:17:47] [I] Idle time: 0ms
[03/25/2022-13:17:47] [I] Streams: 1
[03/25/2022-13:17:47] [I] ExposeDMA: Disabled
[03/25/2022-13:17:47] [I] Data transfers: Enabled
[03/25/2022-13:17:47] [I] Spin-wait: Disabled
[03/25/2022-13:17:47] [I] Multithreading: Disabled
[03/25/2022-13:17:47] [I] CUDA Graph: Disabled
[03/25/2022-13:17:47] [I] Separate profiling: Disabled
[03/25/2022-13:17:47] [I] Time Deserialize: Disabled
[03/25/2022-13:17:47] [I] Time Refit: Disabled
[03/25/2022-13:17:47] [I] Skip inference: Disabled
[03/25/2022-13:17:47] [I] Inputs:
[03/25/2022-13:17:47] [I] === Reporting Options ===
[03/25/2022-13:17:47] [I] Verbose: Disabled
[03/25/2022-13:17:47] [I] Averages: 10 inferences
[03/25/2022-13:17:47] [I] Percentile: 99
[03/25/2022-13:17:47] [I] Dump refittable layers:Disabled
[03/25/2022-13:17:47] [I] Dump output: Disabled
[03/25/2022-13:17:47] [I] Profile: Disabled
[03/25/2022-13:17:47] [I] Export timing to JSON file: 
[03/25/2022-13:17:47] [I] Export output to JSON file: 
[03/25/2022-13:17:47] [I] Export profile to JSON file: 
[03/25/2022-13:17:47] [I] 
[03/25/2022-13:17:47] [I] === Device Information ===
[03/25/2022-13:17:47] [I] Selected Device: A100-SXM4-40GB
[03/25/2022-13:17:47] [I] Compute Capability: 8.0
[03/25/2022-13:17:47] [I] SMs: 108
[03/25/2022-13:17:47] [I] Compute Clock Rate: 1.41 GHz
[03/25/2022-13:17:47] [I] Device Global Memory: 40536 MiB
[03/25/2022-13:17:47] [I] Shared Memory per SM: 164 KiB
[03/25/2022-13:17:47] [I] Memory Bus Width: 5120 bits (ECC enabled)
[03/25/2022-13:17:47] [I] Memory Clock Rate: 1.215 GHz
[03/25/2022-13:17:47] [I] 
[03/25/2022-13:17:47] [I] TensorRT version: 8.2.3
[03/25/2022-13:17:47] [I] [TRT] [MemUsageChange] Init CUDA: CPU +426, GPU +0, now: CPU 438, GPU 686 (MiB)
[03/25/2022-13:17:48] [I] [TRT] [MemUsageSnapshot] Begin constructing builder kernel library: CPU 438 MiB, GPU 686 MiB
[03/25/2022-13:17:48] [I] [TRT] [MemUsageSnapshot] End constructing builder kernel library: CPU 654 MiB, GPU 758 MiB
[03/25/2022-13:17:48] [I] Start parsing network model
[03/25/2022-13:17:48] [I] [TRT] ----------------------------------------------------------------
[03/25/2022-13:17:48] [I] [TRT] Input filename:   resnet50_quant_sparse.onnx
[03/25/2022-13:17:48] [I] [TRT] ONNX IR version:  0.0.8
[03/25/2022-13:17:48] [I] [TRT] Opset version:    11
[03/25/2022-13:17:48] [I] [TRT] Producer name:    
[03/25/2022-13:17:48] [I] [TRT] Producer version: 
[03/25/2022-13:17:48] [I] [TRT] Domain:           
[03/25/2022-13:17:48] [I] [TRT] Model version:    0
[03/25/2022-13:17:48] [I] [TRT] Doc string:       
[03/25/2022-13:17:48] [I] [TRT] ----------------------------------------------------------------
[03/25/2022-13:17:48] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:364: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[03/25/2022-13:17:49] [I] Finish parsing network model
[03/25/2022-13:17:49] [I] FP32 and INT8 precisions have been specified - more performance might be enabled by additionally specifying --fp16 or --best
[03/25/2022-13:17:49] [W] [TRT] Calibrator won't be used in explicit precision mode. Use quantization aware training to generate network with Quantize/Dequantize nodes.
[03/25/2022-13:17:53] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +808, GPU +350, now: CPU 1668, GPU 1188 (MiB)
[03/25/2022-13:17:53] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +126, GPU +60, now: CPU 1794, GPU 1248 (MiB)
[03/25/2022-13:17:53] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored.
[03/25/2022-13:17:54] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:17:54] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:17:54] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:17:54] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:17:54] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:17:54] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:17:54] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:17:54] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:17:54] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:17:57] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:17:57] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:17:57] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:17:57] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:17:57] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:17:57] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:17:57] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:17:57] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:17:59] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:17:59] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:17:59] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:17:59] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:17:59] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:17:59] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:17:59] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:17:59] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:01] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:01] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:01] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:01] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:01] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:01] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:01] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:01] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:02] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:02] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:02] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:02] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:02] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:02] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:02] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:04] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:04] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:04] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:04] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:04] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:04] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:04] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:05] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:05] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:05] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:05] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:06] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:06] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:06] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:06] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:06] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:06] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:06] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:06] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:11] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:11] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:11] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:11] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:11] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:11] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:11] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:11] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:12] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:12] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:12] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:13] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:13] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:13] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:13] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:13] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:13] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:13] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:14] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:14] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:14] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:14] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:14] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:14] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:14] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:14] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:14] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:14] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:15] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:15] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:15] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:15] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:15] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:15] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:15] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:20] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:20] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:20] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:20] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:21] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:21] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:21] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:21] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:23] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:23] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:23] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:23] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:23] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:23] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:23] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:23] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:24] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:24] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:24] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:24] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:24] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:24] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:24] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:26] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:26] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:26] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:26] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:26] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:26] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:26] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:28] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:28] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:28] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:28] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:28] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:28] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:28] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:29] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:29] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:29] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:29] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:29] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:29] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:29] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:31] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:31] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:31] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:31] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:31] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:31] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:31] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:31] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:31] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:31] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:38] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:38] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:38] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:38] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:38] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:38] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:38] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:39] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:45] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:45] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:45] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:49] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:49] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:49] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:50] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
[03/25/2022-13:18:52] [I] [TRT] Detected 1 inputs and 2 output network tensors.
[03/25/2022-13:18:52] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:52] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:52] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:52] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:52] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:52] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:52] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:52] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:52] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:52] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:52] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:52] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:52] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:52] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:53] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:53] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:53] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:53] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:53] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:53] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:53] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:53] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:53] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:53] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:53] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range.
[03/25/2022-13:18:53] [I] [TRT] Total Host Persistent Memory: 113072
[03/25/2022-13:18:53] [I] [TRT] Total Device Persistent Memory: 37537792
[03/25/2022-13:18:53] [I] [TRT] Total Scratch Memory: 0
[03/25/2022-13:18:53] [I] [TRT] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 120 MiB, GPU 369 MiB
[03/25/2022-13:18:53] [I] [TRT] [BlockAssignment] Algorithm ShiftNTopDown took 1.67339ms to assign 4 blocks to 61 nodes requiring 269746176 bytes.
[03/25/2022-13:18:53] [I] [TRT] Total Activation Memory: 269746176
[03/25/2022-13:18:53] [I] [TRT] (Sparsity) Layers eligible for sparse math: sections.0.0.conv1.module.weight + QuantizeLinear_24_quantize_scale_node + Conv_28 + Relu_30, sections.0.0.identity.conv.module.weight + QuantizeLinear_68_quantize_scale_node + Conv_72, sections.0.0.conv2.module.weight + QuantizeLinear_39_quantize_scale_node + Conv_43 + Relu_45, sections.0.0.conv3.module.weight + QuantizeLinear_54_quantize_scale_node + Conv_58 + Add_80 + Relu_81, sections.0.1.conv1.module.weight + QuantizeLinear_90_quantize_scale_node + Conv_94 + Relu_96, sections.0.1.conv2.module.weight + QuantizeLinear_105_quantize_scale_node + Conv_109 + Relu_111, sections.0.1.conv3.module.weight + QuantizeLinear_120_quantize_scale_node + Conv_124 + Add_132 + Relu_133, sections.0.2.conv1.module.weight + QuantizeLinear_142_quantize_scale_node + Conv_146 + Relu_148, sections.0.2.conv2.module.weight + QuantizeLinear_157_quantize_scale_node + Conv_161 + Relu_163, sections.0.2.conv3.module.weight + QuantizeLinear_172_quantize_scale_node + Conv_176 + Add_184 + Relu_185, sections.1.0.conv1.module.weight + QuantizeLinear_194_quantize_scale_node + Conv_198 + Relu_200, sections.1.0.identity.conv.module.weight + QuantizeLinear_238_quantize_scale_node + Conv_242, sections.1.0.conv2.module.weight + QuantizeLinear_209_quantize_scale_node + Conv_213 + Relu_215, sections.1.0.conv3.module.weight + QuantizeLinear_224_quantize_scale_node + Conv_228 + Add_250 + Relu_251, sections.1.1.conv1.module.weight + QuantizeLinear_260_quantize_scale_node + Conv_264 + Relu_266, sections.1.1.conv2.module.weight + QuantizeLinear_275_quantize_scale_node + Conv_279 + Relu_281, sections.1.1.conv3.module.weight + QuantizeLinear_290_quantize_scale_node + Conv_294 + Add_302 + Relu_303, sections.1.2.conv1.module.weight + QuantizeLinear_312_quantize_scale_node + Conv_316 + Relu_318, sections.1.2.conv2.module.weight + QuantizeLinear_327_quantize_scale_node + Conv_331 + Relu_333, sections.1.2.conv3.module.weight + QuantizeLinear_342_quantize_scale_node + Conv_346 + Add_354 + Relu_355, sections.1.3.conv1.module.weight + QuantizeLinear_364_quantize_scale_node + Conv_368 + Relu_370, sections.1.3.conv2.module.weight + QuantizeLinear_379_quantize_scale_node + Conv_383 + Relu_385, sections.1.3.conv3.module.weight + QuantizeLinear_394_quantize_scale_node + Conv_398 + Add_406 + Relu_407, sections.2.0.conv1.module.weight + QuantizeLinear_416_quantize_scale_node + Conv_420 + Relu_422, sections.2.0.identity.conv.module.weight + QuantizeLinear_460_quantize_scale_node + Conv_464, sections.2.0.conv2.module.weight + QuantizeLinear_431_quantize_scale_node + Conv_435 + Relu_437, sections.2.0.conv3.module.weight + QuantizeLinear_446_quantize_scale_node + Conv_450 + Add_472 + Relu_473, sections.2.1.conv1.module.weight + QuantizeLinear_482_quantize_scale_node + Conv_486 + Relu_488, sections.2.1.conv2.module.weight + QuantizeLinear_497_quantize_scale_node + Conv_501 + Relu_503, sections.2.1.conv3.module.weight + QuantizeLinear_512_quantize_scale_node + Conv_516 + Add_524 + Relu_525, sections.2.2.conv1.module.weight + QuantizeLinear_534_quantize_scale_node + Conv_538 + Relu_540, sections.2.2.conv2.module.weight + QuantizeLinear_549_quantize_scale_node + Conv_553 + Relu_555, sections.2.2.conv3.module.weight + QuantizeLinear_564_quantize_scale_node + Conv_568 + Add_576 + Relu_577, sections.2.3.conv1.module.weight + QuantizeLinear_586_quantize_scale_node + Conv_590 + Relu_592, sections.2.3.conv2.module.weight + QuantizeLinear_601_quantize_scale_node + Conv_605 + Relu_607, sections.2.3.conv3.module.weight + QuantizeLinear_616_quantize_scale_node + Conv_620 + Add_628 + Relu_629, sections.2.4.conv1.module.weight + QuantizeLinear_638_quantize_scale_node + Conv_642 + Relu_644, sections.2.4.conv2.module.weight + QuantizeLinear_653_quantize_scale_node + Conv_657 + Relu_659, sections.2.4.conv3.module.weight + QuantizeLinear_668_quantize_scale_node + Conv_672 + Add_680 + Relu_681, sections.2.5.conv1.module.weight + QuantizeLinear_690_quantize_scale_node + Conv_694 + Relu_696, sections.2.5.conv2.module.weight + QuantizeLinear_705_quantize_scale_node + Conv_709 + Relu_711, sections.2.5.conv3.module.weight + QuantizeLinear_720_quantize_scale_node + Conv_724 + Add_732 + Relu_733, sections.3.0.conv1.module.weight + QuantizeLinear_742_quantize_scale_node + Conv_746 + Relu_748, sections.3.0.identity.conv.module.weight + QuantizeLinear_786_quantize_scale_node + Conv_790, sections.3.0.conv2.module.weight + QuantizeLinear_757_quantize_scale_node + Conv_761 + Relu_763, sections.3.0.conv3.module.weight + QuantizeLinear_772_quantize_scale_node + Conv_776 + Add_798 + Relu_799, sections.3.1.conv1.module.weight + QuantizeLinear_808_quantize_scale_node + Conv_812 + Relu_814, sections.3.1.conv2.module.weight + QuantizeLinear_823_quantize_scale_node + Conv_827 + Relu_829, sections.3.1.conv3.module.weight + QuantizeLinear_838_quantize_scale_node + Conv_842 + Add_850 + Relu_851, sections.3.2.conv1.module.weight + QuantizeLinear_860_quantize_scale_node + Conv_864 + Relu_866, sections.3.2.conv2.module.weight + QuantizeLinear_875_quantize_scale_node + Conv_879 + Relu_881, sections.3.2.conv3.module.weight + QuantizeLinear_890_quantize_scale_node + Conv_894 + Add_902 + Relu_903, Gemm_911
[03/25/2022-13:18:53] [I] [TRT] (Sparsity) TRT inference plan picked sparse implementation for layers: sections.0.1.conv1.module.weight + QuantizeLinear_90_quantize_scale_node + Conv_94 + Relu_96, sections.0.2.conv1.module.weight + QuantizeLinear_142_quantize_scale_node + Conv_146 + Relu_148, sections.1.1.conv2.module.weight + QuantizeLinear_275_quantize_scale_node + Conv_279 + Relu_281, sections.1.2.conv2.module.weight + QuantizeLinear_327_quantize_scale_node + Conv_331 + Relu_333, sections.1.3.conv2.module.weight + QuantizeLinear_379_quantize_scale_node + Conv_383 + Relu_385, sections.2.0.identity.conv.module.weight + QuantizeLinear_460_quantize_scale_node + Conv_464, sections.2.0.conv2.module.weight + QuantizeLinear_431_quantize_scale_node + Conv_435 + Relu_437, sections.2.1.conv1.module.weight + QuantizeLinear_482_quantize_scale_node + Conv_486 + Relu_488, sections.2.1.conv2.module.weight + QuantizeLinear_497_quantize_scale_node + Conv_501 + Relu_503, sections.2.2.conv1.module.weight + QuantizeLinear_534_quantize_scale_node + Conv_538 + Relu_540, sections.2.2.conv2.module.weight + QuantizeLinear_549_quantize_scale_node + Conv_553 + Relu_555, sections.2.3.conv1.module.weight + QuantizeLinear_586_quantize_scale_node + Conv_590 + Relu_592, sections.2.3.conv2.module.weight + QuantizeLinear_601_quantize_scale_node + Conv_605 + Relu_607, sections.2.4.conv1.module.weight + QuantizeLinear_638_quantize_scale_node + Conv_642 + Relu_644, sections.2.4.conv2.module.weight + QuantizeLinear_653_quantize_scale_node + Conv_657 + Relu_659, sections.2.5.conv1.module.weight + QuantizeLinear_690_quantize_scale_node + Conv_694 + Relu_696, sections.2.5.conv2.module.weight + QuantizeLinear_705_quantize_scale_node + Conv_709 + Relu_711, sections.3.0.identity.conv.module.weight + QuantizeLinear_786_quantize_scale_node + Conv_790, sections.3.0.conv2.module.weight + QuantizeLinear_757_quantize_scale_node + Conv_761 + Relu_763, sections.3.1.conv1.module.weight + QuantizeLinear_808_quantize_scale_node + Conv_812 + Relu_814, sections.3.1.conv2.module.weight + QuantizeLinear_823_quantize_scale_node + Conv_827 + Relu_829, sections.3.2.conv1.module.weight + QuantizeLinear_860_quantize_scale_node + Conv_864 + Relu_866, sections.3.2.conv2.module.weight + QuantizeLinear_875_quantize_scale_node + Conv_879 + Relu_881
[03/25/2022-13:18:53] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 2675, GPU 1686 (MiB)
[03/25/2022-13:18:53] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +10, now: CPU 2675, GPU 1696 (MiB)
[03/25/2022-13:18:53] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in building engine: CPU +22, GPU +36, now: CPU 22, GPU 36 (MiB)
[03/25/2022-13:18:53] [I] [TRT] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 2597, GPU 1626 (MiB)
[03/25/2022-13:18:53] [I] [TRT] Loaded engine size: 35 MiB
[03/25/2022-13:18:53] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +10, now: CPU 2598, GPU 1672 (MiB)
[03/25/2022-13:18:53] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 2598, GPU 1680 (MiB)
[03/25/2022-13:18:53] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +35, now: CPU 0, GPU 35 (MiB)
[03/25/2022-13:18:53] [I] Engine built in 66.3209 sec.
[03/25/2022-13:18:53] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +10, now: CPU 2246, GPU 1562 (MiB)
[03/25/2022-13:18:53] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 2246, GPU 1570 (MiB)
[03/25/2022-13:18:53] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +293, now: CPU 0, GPU 328 (MiB)
[03/25/2022-13:18:53] [I] Using random values for input input
[03/25/2022-13:18:53] [I] Created input binding for input with dimensions 128x3x224x224
[03/25/2022-13:18:53] [I] Using random values for output output_0
[03/25/2022-13:18:53] [I] Created output binding for output_0 with dimensions 128x1000
[03/25/2022-13:18:53] [I] Using random values for output output_1
[03/25/2022-13:18:53] [I] Created output binding for output_1 with dimensions 128x1000
[03/25/2022-13:18:53] [I] Starting inference
[03/25/2022-13:18:57] [I] Warmup completed 39 queries over 200 ms
[03/25/2022-13:18:57] [I] Timing trace has 602 queries over 3.01702 s
[03/25/2022-13:18:57] [I] 
[03/25/2022-13:18:57] [I] === Trace details ===
[03/25/2022-13:18:57] [I] Trace averages of 10 runs:
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 5.00245 ms - Host latency: 8.49963 ms (end to end 9.90602 ms, enqueue 0.516444 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 5.00081 ms - Host latency: 8.54102 ms (end to end 9.90293 ms, enqueue 0.513441 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 5.00213 ms - Host latency: 8.54685 ms (end to end 9.90481 ms, enqueue 0.513266 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99907 ms - Host latency: 8.54342 ms (end to end 9.89951 ms, enqueue 0.511456 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 5.00142 ms - Host latency: 8.54437 ms (end to end 9.90378 ms, enqueue 0.512524 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 5.00337 ms - Host latency: 8.54199 ms (end to end 9.90971 ms, enqueue 0.512381 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.98442 ms - Host latency: 8.53061 ms (end to end 9.74345 ms, enqueue 0.512061 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99836 ms - Host latency: 8.53592 ms (end to end 9.91873 ms, enqueue 0.510022 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99742 ms - Host latency: 8.53326 ms (end to end 9.89499 ms, enqueue 0.511682 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99752 ms - Host latency: 8.53472 ms (end to end 9.89295 ms, enqueue 0.511395 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99683 ms - Host latency: 8.53513 ms (end to end 9.89702 ms, enqueue 0.511548 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.9963 ms - Host latency: 8.53285 ms (end to end 9.89439 ms, enqueue 0.511371 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.9964 ms - Host latency: 8.53118 ms (end to end 9.89607 ms, enqueue 0.512982 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99855 ms - Host latency: 8.5328 ms (end to end 9.899 ms, enqueue 0.512152 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99775 ms - Host latency: 8.53524 ms (end to end 9.89659 ms, enqueue 0.513312 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99517 ms - Host latency: 8.53299 ms (end to end 9.89399 ms, enqueue 0.510773 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99722 ms - Host latency: 8.53352 ms (end to end 9.89609 ms, enqueue 0.512793 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99651 ms - Host latency: 8.52856 ms (end to end 9.89287 ms, enqueue 0.510889 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99702 ms - Host latency: 8.53372 ms (end to end 9.89522 ms, enqueue 0.513635 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99928 ms - Host latency: 8.53197 ms (end to end 9.90153 ms, enqueue 0.515942 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99845 ms - Host latency: 8.5297 ms (end to end 9.85966 ms, enqueue 0.512842 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99752 ms - Host latency: 8.52555 ms (end to end 9.89608 ms, enqueue 0.511426 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99773 ms - Host latency: 8.52616 ms (end to end 9.89208 ms, enqueue 0.512073 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99959 ms - Host latency: 8.52811 ms (end to end 9.89634 ms, enqueue 0.510706 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99622 ms - Host latency: 8.52147 ms (end to end 9.88768 ms, enqueue 0.508899 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99725 ms - Host latency: 8.52011 ms (end to end 9.89459 ms, enqueue 0.50929 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99733 ms - Host latency: 8.52228 ms (end to end 9.89368 ms, enqueue 0.509314 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99845 ms - Host latency: 8.52461 ms (end to end 9.89825 ms, enqueue 0.510608 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99763 ms - Host latency: 8.52091 ms (end to end 9.88804 ms, enqueue 0.507715 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99528 ms - Host latency: 8.51367 ms (end to end 9.89037 ms, enqueue 0.50885 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99587 ms - Host latency: 8.51342 ms (end to end 9.88844 ms, enqueue 0.508716 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99927 ms - Host latency: 8.51729 ms (end to end 9.89954 ms, enqueue 0.508972 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.9968 ms - Host latency: 8.52147 ms (end to end 9.88962 ms, enqueue 0.509058 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99557 ms - Host latency: 8.50912 ms (end to end 9.89426 ms, enqueue 0.509668 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99398 ms - Host latency: 8.5105 ms (end to end 9.88884 ms, enqueue 0.507422 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99681 ms - Host latency: 8.51428 ms (end to end 9.89202 ms, enqueue 0.50929 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99569 ms - Host latency: 8.51683 ms (end to end 9.85474 ms, enqueue 0.56936 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.9968 ms - Host latency: 8.52278 ms (end to end 9.85791 ms, enqueue 0.575098 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99631 ms - Host latency: 8.51643 ms (end to end 9.8721 ms, enqueue 0.539307 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99478 ms - Host latency: 8.50647 ms (end to end 9.89165 ms, enqueue 0.512769 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.9979 ms - Host latency: 8.50464 ms (end to end 9.89258 ms, enqueue 0.51189 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99746 ms - Host latency: 8.50713 ms (end to end 9.89771 ms, enqueue 0.513525 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.9967 ms - Host latency: 8.50356 ms (end to end 9.89297 ms, enqueue 0.513281 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99753 ms - Host latency: 8.50879 ms (end to end 9.89663 ms, enqueue 0.509717 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.9968 ms - Host latency: 8.50532 ms (end to end 9.89285 ms, enqueue 0.509888 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99536 ms - Host latency: 8.50188 ms (end to end 9.89336 ms, enqueue 0.509839 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99673 ms - Host latency: 8.50256 ms (end to end 9.89229 ms, enqueue 0.510864 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99592 ms - Host latency: 8.50195 ms (end to end 9.88835 ms, enqueue 0.512085 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99646 ms - Host latency: 8.50144 ms (end to end 9.89221 ms, enqueue 0.510449 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99722 ms - Host latency: 8.50469 ms (end to end 9.89612 ms, enqueue 0.509668 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99631 ms - Host latency: 8.50186 ms (end to end 9.88838 ms, enqueue 0.511645 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99678 ms - Host latency: 8.49634 ms (end to end 9.89006 ms, enqueue 0.511768 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.9967 ms - Host latency: 8.49822 ms (end to end 9.89214 ms, enqueue 0.510352 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99641 ms - Host latency: 8.50173 ms (end to end 9.8925 ms, enqueue 0.51106 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99766 ms - Host latency: 8.50146 ms (end to end 9.89299 ms, enqueue 0.510693 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99792 ms - Host latency: 8.49744 ms (end to end 9.89724 ms, enqueue 0.511768 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99744 ms - Host latency: 8.50083 ms (end to end 9.89209 ms, enqueue 0.510254 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99761 ms - Host latency: 8.49778 ms (end to end 9.89968 ms, enqueue 0.510059 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99624 ms - Host latency: 8.49724 ms (end to end 9.89133 ms, enqueue 0.509912 ms)
[03/25/2022-13:18:57] [I] Average on 10 runs - GPU latency: 4.99824 ms - Host latency: 8.49778 ms (end to end 9.89692 ms, enqueue 0.511279 ms)
[03/25/2022-13:18:57] [I] 
[03/25/2022-13:18:57] [I] === Performance summary ===
[03/25/2022-13:18:57] [I] Throughput: 199.535 qps
[03/25/2022-13:18:57] [I] Latency: min = 8.43875 ms, max = 8.6521 ms, mean = 8.5184 ms, median = 8.51758 ms, percentile(99%) = 8.55493 ms
[03/25/2022-13:18:57] [I] End-to-End Host Latency: min = 8.6106 ms, max = 10.108 ms, mean = 9.89082 ms, median = 9.89441 ms, percentile(99%) = 9.93066 ms
[03/25/2022-13:18:57] [I] Enqueue Time: min = 0.504639 ms, max = 0.581543 ms, mean = 0.51368 ms, median = 0.510498 ms, percentile(99%) = 0.57666 ms
[03/25/2022-13:18:57] [I] H2D Latency: min = 3.35872 ms, max = 3.50482 ms, mean = 3.44017 ms, median = 3.44043 ms, percentile(99%) = 3.47601 ms
[03/25/2022-13:18:57] [I] GPU Compute Time: min = 4.96027 ms, max = 5.15796 ms, mean = 4.99751 ms, median = 4.99719 ms, percentile(99%) = 5.00635 ms
[03/25/2022-13:18:57] [I] D2H Latency: min = 0.0773926 ms, max = 0.104431 ms, mean = 0.0807253 ms, median = 0.0805664 ms, percentile(99%) = 0.0830078 ms
[03/25/2022-13:18:57] [I] Total Host Walltime: 3.01702 s
[03/25/2022-13:18:57] [I] Total GPU Compute Time: 3.0085 s
[03/25/2022-13:18:57] [I] Explanations of the performance metrics are printed in the verbose logs.
[03/25/2022-13:18:57] [I] 
&&&& PASSED TensorRT.trtexec [TensorRT v8203] # trtexec --onnx=resnet50_quant_sparse.onnx --int8 --sparsity=enable --shapes=input:128x3x224x224