&&&& RUNNING TensorRT.trtexec [TensorRT v8203] # trtexec --onnx=resnet50_quant_sparse.onnx --int8 --sparsity=force --shapes=input:128x3x224x224 [03/25/2022-13:16:22] [I] === Model Options === [03/25/2022-13:16:22] [I] Format: ONNX [03/25/2022-13:16:22] [I] Model: resnet50_quant_sparse.onnx [03/25/2022-13:16:22] [I] Output: [03/25/2022-13:16:22] [I] === Build Options === [03/25/2022-13:16:22] [I] Max batch: explicit batch [03/25/2022-13:16:22] [I] Workspace: 16 MiB [03/25/2022-13:16:22] [I] minTiming: 1 [03/25/2022-13:16:22] [I] avgTiming: 8 [03/25/2022-13:16:22] [I] Precision: FP32+INT8 [03/25/2022-13:16:22] [I] Calibration: Dynamic [03/25/2022-13:16:22] [I] Refit: Disabled [03/25/2022-13:16:22] [I] Sparsity: Forced [03/25/2022-13:16:22] [I] Safe mode: Disabled [03/25/2022-13:16:22] [I] DirectIO mode: Disabled [03/25/2022-13:16:22] [I] Restricted mode: Disabled [03/25/2022-13:16:22] [I] Save engine: [03/25/2022-13:16:22] [I] Load engine: [03/25/2022-13:16:22] [I] Profiling verbosity: 0 [03/25/2022-13:16:22] [I] Tactic sources: Using default tactic sources [03/25/2022-13:16:22] [I] timingCacheMode: local [03/25/2022-13:16:22] [I] timingCacheFile: [03/25/2022-13:16:22] [I] Input(s)s format: fp32:CHW [03/25/2022-13:16:22] [I] Output(s)s format: fp32:CHW [03/25/2022-13:16:22] [I] Input build shape: input=128x3x224x224+128x3x224x224+128x3x224x224 [03/25/2022-13:16:22] [I] Input calibration shapes: model [03/25/2022-13:16:22] [I] === System Options === [03/25/2022-13:16:22] [I] Device: 0 [03/25/2022-13:16:22] [I] DLACore: [03/25/2022-13:16:22] [I] Plugins: [03/25/2022-13:16:22] [I] === Inference Options === [03/25/2022-13:16:22] [I] Batch: Explicit [03/25/2022-13:16:22] [I] Input inference shape: input=128x3x224x224 [03/25/2022-13:16:22] [I] Iterations: 10 [03/25/2022-13:16:22] [I] Duration: 3s (+ 200ms warm up) [03/25/2022-13:16:22] [I] Sleep time: 0ms [03/25/2022-13:16:22] [I] Idle time: 0ms [03/25/2022-13:16:22] [I] Streams: 1 [03/25/2022-13:16:22] [I] ExposeDMA: Disabled [03/25/2022-13:16:22] [I] Data transfers: Enabled [03/25/2022-13:16:22] [I] Spin-wait: Disabled [03/25/2022-13:16:22] [I] Multithreading: Disabled [03/25/2022-13:16:22] [I] CUDA Graph: Disabled [03/25/2022-13:16:22] [I] Separate profiling: Disabled [03/25/2022-13:16:22] [I] Time Deserialize: Disabled [03/25/2022-13:16:22] [I] Time Refit: Disabled [03/25/2022-13:16:22] [I] Skip inference: Disabled [03/25/2022-13:16:22] [I] Inputs: [03/25/2022-13:16:22] [I] === Reporting Options === [03/25/2022-13:16:22] [I] Verbose: Disabled [03/25/2022-13:16:22] [I] Averages: 10 inferences [03/25/2022-13:16:22] [I] Percentile: 99 [03/25/2022-13:16:22] [I] Dump refittable layers:Disabled [03/25/2022-13:16:22] [I] Dump output: Disabled [03/25/2022-13:16:22] [I] Profile: Disabled [03/25/2022-13:16:22] [I] Export timing to JSON file: [03/25/2022-13:16:22] [I] Export output to JSON file: [03/25/2022-13:16:22] [I] Export profile to JSON file: [03/25/2022-13:16:22] [I] [03/25/2022-13:16:22] [I] === Device Information === [03/25/2022-13:16:22] [I] Selected Device: A100-SXM4-40GB [03/25/2022-13:16:22] [I] Compute Capability: 8.0 [03/25/2022-13:16:22] [I] SMs: 108 [03/25/2022-13:16:22] [I] Compute Clock Rate: 1.41 GHz [03/25/2022-13:16:22] [I] Device Global Memory: 40536 MiB [03/25/2022-13:16:22] [I] Shared Memory per SM: 164 KiB [03/25/2022-13:16:22] [I] Memory Bus Width: 5120 bits (ECC enabled) [03/25/2022-13:16:22] [I] Memory Clock Rate: 1.215 GHz [03/25/2022-13:16:22] [I] [03/25/2022-13:16:22] [I] TensorRT version: 8.2.3 [03/25/2022-13:16:23] [I] [TRT] [MemUsageChange] Init CUDA: CPU +426, GPU +0, now: CPU 438, GPU 686 (MiB) [03/25/2022-13:16:23] [I] [TRT] [MemUsageSnapshot] Begin constructing builder kernel library: CPU 438 MiB, GPU 686 MiB [03/25/2022-13:16:23] [I] [TRT] [MemUsageSnapshot] End constructing builder kernel library: CPU 654 MiB, GPU 758 MiB [03/25/2022-13:16:23] [I] Start parsing network model [03/25/2022-13:16:23] [I] [TRT] ---------------------------------------------------------------- [03/25/2022-13:16:23] [I] [TRT] Input filename: resnet50_quant_sparse.onnx [03/25/2022-13:16:23] [I] [TRT] ONNX IR version: 0.0.8 [03/25/2022-13:16:23] [I] [TRT] Opset version: 11 [03/25/2022-13:16:23] [I] [TRT] Producer name: [03/25/2022-13:16:23] [I] [TRT] Producer version: [03/25/2022-13:16:23] [I] [TRT] Domain: [03/25/2022-13:16:23] [I] [TRT] Model version: 0 [03/25/2022-13:16:23] [I] [TRT] Doc string: [03/25/2022-13:16:23] [I] [TRT] ---------------------------------------------------------------- [03/25/2022-13:16:23] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:364: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [03/25/2022-13:16:24] [I] Finish parsing network model [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [E] Error[3]: [layers.h::setKernelWeights::266] Error Code 3: API Usage Error (Parameter check failed at: /_src/build/cuda-11.4/8.2/x86_64/release/optimizer/api/layers.h::setKernelWeights::266, condition: kernelWeights.values != nullptr ) [03/25/2022-13:16:24] [I] FP32 and INT8 precisions have been specified - more performance might be enabled by additionally specifying --fp16 or --best [03/25/2022-13:16:24] [W] [TRT] Calibrator won't be used in explicit precision mode. Use quantization aware training to generate network with Quantize/Dequantize nodes. [03/25/2022-13:16:28] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +809, GPU +350, now: CPU 1676, GPU 1188 (MiB) [03/25/2022-13:16:28] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +126, GPU +60, now: CPU 1802, GPU 1248 (MiB) [03/25/2022-13:16:28] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored. [03/25/2022-13:16:29] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:29] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:29] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:29] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:29] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:29] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:29] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:29] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:29] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:32] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:32] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:32] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:32] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:32] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:32] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:32] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:32] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:34] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:34] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:34] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:34] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:35] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:35] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:35] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:35] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:36] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:36] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:36] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:36] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:36] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:36] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:36] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:36] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:37] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:37] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:37] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:37] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:37] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:37] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:37] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:39] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:39] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:39] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:39] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:39] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:39] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:40] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:40] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:40] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:40] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:40] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:41] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:41] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:41] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:41] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:41] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:41] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:41] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:41] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:46] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:46] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:46] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:46] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:46] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:46] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:46] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:46] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:47] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:47] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:47] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:48] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:48] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:48] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:48] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:48] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:48] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:48] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:49] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:49] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:49] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:49] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:49] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:49] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:49] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:49] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:49] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:49] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:50] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:50] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:50] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:50] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:50] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:50] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:50] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:56] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:56] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:56] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:56] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:56] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:56] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:56] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:56] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:58] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:58] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:58] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:59] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:59] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:59] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:59] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:16:59] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:00] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:00] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:00] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:00] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:00] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:00] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:00] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:01] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:01] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:01] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:01] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:01] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:01] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:01] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:03] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:03] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:03] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:03] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:03] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:03] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:03] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:04] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:04] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:04] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:04] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:04] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:04] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:04] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:07] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:07] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:07] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:07] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:07] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:07] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:07] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:07] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:07] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:07] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:13] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:13] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:13] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:13] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:14] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:14] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:14] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:14] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:21] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:21] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:21] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:24] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:24] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:24] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:25] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output. [03/25/2022-13:17:27] [I] [TRT] Detected 1 inputs and 2 output network tensors. [03/25/2022-13:17:27] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:27] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:27] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:27] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:27] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:27] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:27] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:27] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:27] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:27] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:27] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:27] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:27] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:27] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:28] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:28] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:28] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:28] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:28] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:28] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:28] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:28] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:28] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:28] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:28] [W] [TRT] Some weights are outside of int8_t range and will be clipped to int8_t range. [03/25/2022-13:17:28] [I] [TRT] Total Host Persistent Memory: 113072 [03/25/2022-13:17:28] [I] [TRT] Total Device Persistent Memory: 37537792 [03/25/2022-13:17:28] [I] [TRT] Total Scratch Memory: 0 [03/25/2022-13:17:28] [I] [TRT] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 120 MiB, GPU 369 MiB [03/25/2022-13:17:28] [I] [TRT] [BlockAssignment] Algorithm ShiftNTopDown took 1.6844ms to assign 4 blocks to 61 nodes requiring 269746176 bytes. [03/25/2022-13:17:28] [I] [TRT] Total Activation Memory: 269746176 [03/25/2022-13:17:28] [I] [TRT] (Sparsity) Layers eligible for sparse math: sections.0.0.conv1.module.weight + QuantizeLinear_24_quantize_scale_node + Conv_28 + Relu_30, sections.0.0.identity.conv.module.weight + QuantizeLinear_68_quantize_scale_node + Conv_72, sections.0.0.conv2.module.weight + QuantizeLinear_39_quantize_scale_node + Conv_43 + Relu_45, sections.0.0.conv3.module.weight + QuantizeLinear_54_quantize_scale_node + Conv_58 + Add_80 + Relu_81, sections.0.1.conv1.module.weight + QuantizeLinear_90_quantize_scale_node + Conv_94 + Relu_96, sections.0.1.conv2.module.weight + QuantizeLinear_105_quantize_scale_node + Conv_109 + Relu_111, sections.0.1.conv3.module.weight + QuantizeLinear_120_quantize_scale_node + Conv_124 + Add_132 + Relu_133, sections.0.2.conv1.module.weight + QuantizeLinear_142_quantize_scale_node + Conv_146 + Relu_148, sections.0.2.conv2.module.weight + QuantizeLinear_157_quantize_scale_node + Conv_161 + Relu_163, sections.0.2.conv3.module.weight + QuantizeLinear_172_quantize_scale_node + Conv_176 + Add_184 + Relu_185, sections.1.0.conv1.module.weight + QuantizeLinear_194_quantize_scale_node + Conv_198 + Relu_200, sections.1.0.identity.conv.module.weight + QuantizeLinear_238_quantize_scale_node + Conv_242, sections.1.0.conv2.module.weight + QuantizeLinear_209_quantize_scale_node + Conv_213 + Relu_215, sections.1.0.conv3.module.weight + QuantizeLinear_224_quantize_scale_node + Conv_228 + Add_250 + Relu_251, sections.1.1.conv1.module.weight + QuantizeLinear_260_quantize_scale_node + Conv_264 + Relu_266, sections.1.1.conv2.module.weight + QuantizeLinear_275_quantize_scale_node + Conv_279 + Relu_281, sections.1.1.conv3.module.weight + QuantizeLinear_290_quantize_scale_node + Conv_294 + Add_302 + Relu_303, sections.1.2.conv1.module.weight + QuantizeLinear_312_quantize_scale_node + Conv_316 + Relu_318, sections.1.2.conv2.module.weight + QuantizeLinear_327_quantize_scale_node + Conv_331 + Relu_333, sections.1.2.conv3.module.weight + QuantizeLinear_342_quantize_scale_node + Conv_346 + Add_354 + Relu_355, sections.1.3.conv1.module.weight + QuantizeLinear_364_quantize_scale_node + Conv_368 + Relu_370, sections.1.3.conv2.module.weight + QuantizeLinear_379_quantize_scale_node + Conv_383 + Relu_385, sections.1.3.conv3.module.weight + QuantizeLinear_394_quantize_scale_node + Conv_398 + Add_406 + Relu_407, sections.2.0.conv1.module.weight + QuantizeLinear_416_quantize_scale_node + Conv_420 + Relu_422, sections.2.0.identity.conv.module.weight + QuantizeLinear_460_quantize_scale_node + Conv_464, sections.2.0.conv2.module.weight + QuantizeLinear_431_quantize_scale_node + Conv_435 + Relu_437, sections.2.0.conv3.module.weight + QuantizeLinear_446_quantize_scale_node + Conv_450 + Add_472 + Relu_473, sections.2.1.conv1.module.weight + QuantizeLinear_482_quantize_scale_node + Conv_486 + Relu_488, sections.2.1.conv2.module.weight + QuantizeLinear_497_quantize_scale_node + Conv_501 + Relu_503, sections.2.1.conv3.module.weight + QuantizeLinear_512_quantize_scale_node + Conv_516 + Add_524 + Relu_525, sections.2.2.conv1.module.weight + QuantizeLinear_534_quantize_scale_node + Conv_538 + Relu_540, sections.2.2.conv2.module.weight + QuantizeLinear_549_quantize_scale_node + Conv_553 + Relu_555, sections.2.2.conv3.module.weight + QuantizeLinear_564_quantize_scale_node + Conv_568 + Add_576 + Relu_577, sections.2.3.conv1.module.weight + QuantizeLinear_586_quantize_scale_node + Conv_590 + Relu_592, sections.2.3.conv2.module.weight + QuantizeLinear_601_quantize_scale_node + Conv_605 + Relu_607, sections.2.3.conv3.module.weight + QuantizeLinear_616_quantize_scale_node + Conv_620 + Add_628 + Relu_629, sections.2.4.conv1.module.weight + QuantizeLinear_638_quantize_scale_node + Conv_642 + Relu_644, sections.2.4.conv2.module.weight + QuantizeLinear_653_quantize_scale_node + Conv_657 + Relu_659, sections.2.4.conv3.module.weight + QuantizeLinear_668_quantize_scale_node + Conv_672 + Add_680 + Relu_681, sections.2.5.conv1.module.weight + QuantizeLinear_690_quantize_scale_node + Conv_694 + Relu_696, sections.2.5.conv2.module.weight + QuantizeLinear_705_quantize_scale_node + Conv_709 + Relu_711, sections.2.5.conv3.module.weight + QuantizeLinear_720_quantize_scale_node + Conv_724 + Add_732 + Relu_733, sections.3.0.conv1.module.weight + QuantizeLinear_742_quantize_scale_node + Conv_746 + Relu_748, sections.3.0.identity.conv.module.weight + QuantizeLinear_786_quantize_scale_node + Conv_790, sections.3.0.conv2.module.weight + QuantizeLinear_757_quantize_scale_node + Conv_761 + Relu_763, sections.3.0.conv3.module.weight + QuantizeLinear_772_quantize_scale_node + Conv_776 + Add_798 + Relu_799, sections.3.1.conv1.module.weight + QuantizeLinear_808_quantize_scale_node + Conv_812 + Relu_814, sections.3.1.conv2.module.weight + QuantizeLinear_823_quantize_scale_node + Conv_827 + Relu_829, sections.3.1.conv3.module.weight + QuantizeLinear_838_quantize_scale_node + Conv_842 + Add_850 + Relu_851, sections.3.2.conv1.module.weight + QuantizeLinear_860_quantize_scale_node + Conv_864 + Relu_866, sections.3.2.conv2.module.weight + QuantizeLinear_875_quantize_scale_node + Conv_879 + Relu_881, sections.3.2.conv3.module.weight + QuantizeLinear_890_quantize_scale_node + Conv_894 + Add_902 + Relu_903, Gemm_911 [03/25/2022-13:17:28] [I] [TRT] (Sparsity) TRT inference plan picked sparse implementation for layers: sections.0.1.conv1.module.weight + QuantizeLinear_90_quantize_scale_node + Conv_94 + Relu_96, sections.0.2.conv1.module.weight + QuantizeLinear_142_quantize_scale_node + Conv_146 + Relu_148, sections.1.1.conv2.module.weight + QuantizeLinear_275_quantize_scale_node + Conv_279 + Relu_281, sections.1.2.conv2.module.weight + QuantizeLinear_327_quantize_scale_node + Conv_331 + Relu_333, sections.1.3.conv2.module.weight + QuantizeLinear_379_quantize_scale_node + Conv_383 + Relu_385, sections.2.0.identity.conv.module.weight + QuantizeLinear_460_quantize_scale_node + Conv_464, sections.2.0.conv2.module.weight + QuantizeLinear_431_quantize_scale_node + Conv_435 + Relu_437, sections.2.1.conv1.module.weight + QuantizeLinear_482_quantize_scale_node + Conv_486 + Relu_488, sections.2.1.conv2.module.weight + QuantizeLinear_497_quantize_scale_node + Conv_501 + Relu_503, sections.2.2.conv1.module.weight + QuantizeLinear_534_quantize_scale_node + Conv_538 + Relu_540, sections.2.2.conv2.module.weight + QuantizeLinear_549_quantize_scale_node + Conv_553 + Relu_555, sections.2.3.conv1.module.weight + QuantizeLinear_586_quantize_scale_node + Conv_590 + Relu_592, sections.2.3.conv2.module.weight + QuantizeLinear_601_quantize_scale_node + Conv_605 + Relu_607, sections.2.4.conv1.module.weight + QuantizeLinear_638_quantize_scale_node + Conv_642 + Relu_644, sections.2.4.conv2.module.weight + QuantizeLinear_653_quantize_scale_node + Conv_657 + Relu_659, sections.2.5.conv1.module.weight + QuantizeLinear_690_quantize_scale_node + Conv_694 + Relu_696, sections.2.5.conv2.module.weight + QuantizeLinear_705_quantize_scale_node + Conv_709 + Relu_711, sections.3.0.identity.conv.module.weight + QuantizeLinear_786_quantize_scale_node + Conv_790, sections.3.0.conv2.module.weight + QuantizeLinear_757_quantize_scale_node + Conv_761 + Relu_763, sections.3.1.conv1.module.weight + QuantizeLinear_808_quantize_scale_node + Conv_812 + Relu_814, sections.3.1.conv2.module.weight + QuantizeLinear_823_quantize_scale_node + Conv_827 + Relu_829, sections.3.2.conv1.module.weight + QuantizeLinear_860_quantize_scale_node + Conv_864 + Relu_866, sections.3.2.conv2.module.weight + QuantizeLinear_875_quantize_scale_node + Conv_879 + Relu_881 [03/25/2022-13:17:28] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 2683, GPU 1686 (MiB) [03/25/2022-13:17:28] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +10, now: CPU 2683, GPU 1696 (MiB) [03/25/2022-13:17:28] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in building engine: CPU +22, GPU +36, now: CPU 22, GPU 36 (MiB) [03/25/2022-13:17:28] [I] [TRT] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 2605, GPU 1626 (MiB) [03/25/2022-13:17:28] [I] [TRT] Loaded engine size: 35 MiB [03/25/2022-13:17:28] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +10, now: CPU 2606, GPU 1672 (MiB) [03/25/2022-13:17:28] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 2606, GPU 1680 (MiB) [03/25/2022-13:17:28] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +35, now: CPU 0, GPU 35 (MiB) [03/25/2022-13:17:28] [I] Engine built in 66.0971 sec. [03/25/2022-13:17:28] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +10, now: CPU 2246, GPU 1560 (MiB) [03/25/2022-13:17:28] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 2246, GPU 1568 (MiB) [03/25/2022-13:17:28] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +293, now: CPU 0, GPU 328 (MiB) [03/25/2022-13:17:28] [I] Using random values for input input [03/25/2022-13:17:28] [I] Created input binding for input with dimensions 128x3x224x224 [03/25/2022-13:17:28] [I] Using random values for output output_0 [03/25/2022-13:17:28] [I] Created output binding for output_0 with dimensions 128x1000 [03/25/2022-13:17:28] [I] Using random values for output output_1 [03/25/2022-13:17:28] [I] Created output binding for output_1 with dimensions 128x1000 [03/25/2022-13:17:28] [I] Starting inference [03/25/2022-13:17:32] [I] Warmup completed 39 queries over 200 ms [03/25/2022-13:17:32] [I] Timing trace has 601 queries over 3.01845 s [03/25/2022-13:17:32] [I] [03/25/2022-13:17:32] [I] === Trace details === [03/25/2022-13:17:32] [I] Trace averages of 10 runs: [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00572 ms - Host latency: 8.49689 ms (end to end 9.91381 ms, enqueue 0.510344 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00685 ms - Host latency: 8.54925 ms (end to end 9.80264 ms, enqueue 0.509467 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00511 ms - Host latency: 8.541 ms (end to end 9.91182 ms, enqueue 0.512747 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00408 ms - Host latency: 8.53745 ms (end to end 9.91083 ms, enqueue 0.510675 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00634 ms - Host latency: 8.53811 ms (end to end 9.91613 ms, enqueue 0.512341 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00439 ms - Host latency: 8.53318 ms (end to end 9.90637 ms, enqueue 0.509512 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00695 ms - Host latency: 8.53447 ms (end to end 9.91903 ms, enqueue 0.508423 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00603 ms - Host latency: 8.53345 ms (end to end 9.91183 ms, enqueue 0.507269 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00398 ms - Host latency: 8.52896 ms (end to end 9.91049 ms, enqueue 0.50788 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 4.9881 ms - Host latency: 8.51074 ms (end to end 9.75754 ms, enqueue 0.507758 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 4.99916 ms - Host latency: 8.52132 ms (end to end 9.90275 ms, enqueue 0.506555 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 4.99886 ms - Host latency: 8.52199 ms (end to end 9.89955 ms, enqueue 0.507922 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00049 ms - Host latency: 8.5248 ms (end to end 9.90139 ms, enqueue 0.508545 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00173 ms - Host latency: 8.52447 ms (end to end 9.90546 ms, enqueue 0.50899 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.0005 ms - Host latency: 8.52136 ms (end to end 9.90242 ms, enqueue 0.511823 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 4.99907 ms - Host latency: 8.51988 ms (end to end 9.89937 ms, enqueue 0.512878 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00316 ms - Host latency: 8.52353 ms (end to end 9.90682 ms, enqueue 0.511688 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 4.99985 ms - Host latency: 8.52083 ms (end to end 9.90269 ms, enqueue 0.511682 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 4.99968 ms - Host latency: 8.51829 ms (end to end 9.89852 ms, enqueue 0.511072 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00176 ms - Host latency: 8.51979 ms (end to end 9.90315 ms, enqueue 0.511853 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00114 ms - Host latency: 8.51909 ms (end to end 9.90487 ms, enqueue 0.510559 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00201 ms - Host latency: 8.51421 ms (end to end 9.90338 ms, enqueue 0.512866 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00028 ms - Host latency: 8.51746 ms (end to end 9.90181 ms, enqueue 0.511829 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 4.99863 ms - Host latency: 8.51567 ms (end to end 9.90002 ms, enqueue 0.513257 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 4.99854 ms - Host latency: 8.51638 ms (end to end 9.90037 ms, enqueue 0.511487 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00153 ms - Host latency: 8.51866 ms (end to end 9.90862 ms, enqueue 0.512244 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 4.99968 ms - Host latency: 8.5152 ms (end to end 9.90286 ms, enqueue 0.508972 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00081 ms - Host latency: 8.5158 ms (end to end 9.90227 ms, enqueue 0.509033 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 4.99806 ms - Host latency: 8.51311 ms (end to end 9.89745 ms, enqueue 0.508118 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00009 ms - Host latency: 8.51494 ms (end to end 9.90338 ms, enqueue 0.509399 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00032 ms - Host latency: 8.51415 ms (end to end 9.90424 ms, enqueue 0.509326 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00245 ms - Host latency: 8.51337 ms (end to end 9.90531 ms, enqueue 0.509119 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00021 ms - Host latency: 8.51157 ms (end to end 9.90156 ms, enqueue 0.509717 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 4.99978 ms - Host latency: 8.51228 ms (end to end 9.89963 ms, enqueue 0.509241 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00029 ms - Host latency: 8.51122 ms (end to end 9.90363 ms, enqueue 0.509143 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00081 ms - Host latency: 8.50962 ms (end to end 9.90424 ms, enqueue 0.509192 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 4.99896 ms - Host latency: 8.55797 ms (end to end 9.83521 ms, enqueue 0.509888 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00276 ms - Host latency: 8.52434 ms (end to end 9.81882 ms, enqueue 0.508398 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00081 ms - Host latency: 8.51289 ms (end to end 9.90745 ms, enqueue 0.510498 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 4.99805 ms - Host latency: 8.50845 ms (end to end 9.89827 ms, enqueue 0.512402 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00125 ms - Host latency: 8.51074 ms (end to end 9.90212 ms, enqueue 0.511133 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 4.99988 ms - Host latency: 8.51187 ms (end to end 9.89988 ms, enqueue 0.511597 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 4.99866 ms - Host latency: 8.50874 ms (end to end 9.89543 ms, enqueue 0.509253 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00186 ms - Host latency: 8.50979 ms (end to end 9.90217 ms, enqueue 0.509277 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00071 ms - Host latency: 8.50801 ms (end to end 9.89922 ms, enqueue 0.508203 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 4.99995 ms - Host latency: 8.50667 ms (end to end 9.90103 ms, enqueue 0.508814 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00073 ms - Host latency: 8.50901 ms (end to end 9.90254 ms, enqueue 0.50874 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00051 ms - Host latency: 8.50725 ms (end to end 9.90042 ms, enqueue 0.511377 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00095 ms - Host latency: 8.50767 ms (end to end 9.90244 ms, enqueue 0.51062 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00127 ms - Host latency: 8.50776 ms (end to end 9.90098 ms, enqueue 0.509546 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00029 ms - Host latency: 8.50425 ms (end to end 9.89878 ms, enqueue 0.510205 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 4.9979 ms - Host latency: 8.50269 ms (end to end 9.89663 ms, enqueue 0.510156 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 4.99973 ms - Host latency: 8.50334 ms (end to end 9.8979 ms, enqueue 0.509839 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00183 ms - Host latency: 8.5061 ms (end to end 9.90447 ms, enqueue 0.510376 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00081 ms - Host latency: 8.5041 ms (end to end 9.90093 ms, enqueue 0.51311 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.0009 ms - Host latency: 8.50032 ms (end to end 9.8981 ms, enqueue 0.514624 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00027 ms - Host latency: 8.50115 ms (end to end 9.89819 ms, enqueue 0.513965 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00088 ms - Host latency: 8.50095 ms (end to end 9.90049 ms, enqueue 0.513379 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 4.99944 ms - Host latency: 8.49915 ms (end to end 9.89763 ms, enqueue 0.513184 ms) [03/25/2022-13:17:32] [I] Average on 10 runs - GPU latency: 5.00098 ms - Host latency: 8.49995 ms (end to end 9.90266 ms, enqueue 0.512256 ms) [03/25/2022-13:17:32] [I] [03/25/2022-13:17:32] [I] === Performance summary === [03/25/2022-13:17:32] [I] Throughput: 199.109 qps [03/25/2022-13:17:32] [I] Latency: min = 8.43204 ms, max = 8.97388 ms, mean = 8.51629 ms, median = 8.51392 ms, percentile(99%) = 8.55383 ms [03/25/2022-13:17:32] [I] End-to-End Host Latency: min = 8.58008 ms, max = 10.1192 ms, mean = 9.8967 ms, median = 9.90271 ms, percentile(99%) = 9.94336 ms [03/25/2022-13:17:32] [I] Enqueue Time: min = 0.497803 ms, max = 0.536865 ms, mean = 0.510393 ms, median = 0.509277 ms, percentile(99%) = 0.525879 ms [03/25/2022-13:17:32] [I] H2D Latency: min = 3.35222 ms, max = 3.87866 ms, mean = 3.43631 ms, median = 3.43481 ms, percentile(99%) = 3.46454 ms [03/25/2022-13:17:32] [I] GPU Compute Time: min = 4.95923 ms, max = 5.13647 ms, mean = 5.00116 ms, median = 5.00122 ms, percentile(99%) = 5.01147 ms [03/25/2022-13:17:32] [I] D2H Latency: min = 0.0756836 ms, max = 0.106201 ms, mean = 0.0788235 ms, median = 0.0786743 ms, percentile(99%) = 0.0811768 ms [03/25/2022-13:17:32] [I] Total Host Walltime: 3.01845 s [03/25/2022-13:17:32] [I] Total GPU Compute Time: 3.0057 s [03/25/2022-13:17:32] [I] Explanations of the performance metrics are printed in the verbose logs. [03/25/2022-13:17:32] [I] &&&& PASSED TensorRT.trtexec [TensorRT v8203] # trtexec --onnx=resnet50_quant_sparse.onnx --int8 --sparsity=force --shapes=input:128x3x224x224