I’m running into some issues setting the input shape with trtexec, as shown in example 4: https://github.com/NVIDIA/TensorRT/blob/master/samples/opensource/trtexec/README.md
For example, with the ResNet 50 model from ONNX’s model zoo (https://github.com/onnx/models/tree/master/vision/classification/resnet), I can run trtexec normally. However, if I try to specify the input shape, even to match the existing input shape, the program ‘runs’, but no queries actually get processed
./trtexec --onnx=resnet50v1.onnx --shapes=data:1x3x224x224
Hi,
Can you try using TRT 7, it seems to be working fine on latest TRT version:
trtexec --onnx=/test/resnet50v1.onnx --shapes=data:1x3x224x224 --explicitBatch
Also, if the model itself is static dimension, other dimensions will not work.
Thanks
Hello, I am trying to profile ResNet50 on 2080Ti with trtexec, I am really confused by throughput calculation. By setting up explicit batch and shape, it results in 0 qps. If not set, it has unbelievably high qps. Am I doing some big mistake here?
&&&& RUNNING TensorRT.trtexec # trtexec --loadEngine=resnet50.trt --batch=32
[08/05/2021-19:03:11] [I] === Model Options ===
[08/05/2021-19:03:11] [I] Format: *
[08/05/2021-19:03:11] [I] Model:
[08/05/2021-19:03:11] [I] Output:
[08/05/2021-19:03:11] [I] === Build Options ===
[08/05/2021-19:03:11] [I] Max batch: 32
[08/05/2021-19:03:11] [I] Workspace: 16 MB
[08/05/2021-19:03:11] [I] minTiming: 1
[08/05/2021-19:03:11] [I] avgTiming: 8
[08/05/2021-19:03:11] [I] Precision: FP32
[08/05/2021-19:03:11] [I] Calibration:
[08/05/2021-19:03:11] [I] Safe mode: Disabled
[08/05/2021-19:03:11] [I] Save engine:
[08/05/2021-19:03:11] [I] Load engine: resnet50.trt
[08/05/2021-19:03:11] [I] Inputs format: fp32:CHW
[08/05/2021-19:03:11] [I] Outputs format: fp32:CHW
[08/05/2021-19:03:11] [I] Input build shapes: model
[08/05/2021-19:03:11] [I] === System Options ===
[08/05/2021-19:03:11] [I] Device: 0
[08/05/2021-19:03:11] [I] DLACore:
[08/05/2021-19:03:11] [I] Plugins:
[08/05/2021-19:03:11] [I] === Inference Options ===
[08/05/2021-19:03:11] [I] Batch: 32
[08/05/2021-19:03:11] [I] Iterations: 10
[08/05/2021-19:03:11] [I] Duration: 3s (+ 200ms warm up)
[08/05/2021-19:03:11] [I] Sleep time: 0ms
[08/05/2021-19:03:11] [I] Streams: 1
[08/05/2021-19:03:11] [I] ExposeDMA: Disabled
[08/05/2021-19:03:11] [I] Spin-wait: Disabled
[08/05/2021-19:03:11] [I] Multithreading: Disabled
[08/05/2021-19:03:11] [I] CUDA Graph: Disabled
[08/05/2021-19:03:11] [I] Skip inference: Disabled
[08/05/2021-19:03:11] [I] Input inference shapes: model
[08/05/2021-19:03:11] [I] Inputs:
[08/05/2021-19:03:11] [I] === Reporting Options ===
[08/05/2021-19:03:11] [I] Verbose: Disabled
[08/05/2021-19:03:11] [I] Averages: 10 inferences
[08/05/2021-19:03:11] [I] Percentile: 99
[08/05/2021-19:03:11] [I] Dump output: Disabled
[08/05/2021-19:03:11] [I] Profile: Disabled
[08/05/2021-19:03:11] [I] Export timing to JSON file:
[08/05/2021-19:03:11] [I] Export output to JSON file:
[08/05/2021-19:03:11] [I] Export profile to JSON file:
[08/05/2021-19:03:11] [I]
[08/05/2021-19:03:15] [I] Warmup completed 2208 queries over 200 ms
[08/05/2021-19:03:15] [I] Timing trace has 45664 queries over 3.00502 s
[08/05/2021-19:03:15] [I] Trace averages of 10 runs:
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.53436 ms - Host latency: 4.04649 ms (end to end 4.87168 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08266 ms - Host latency: 3.59755 ms (end to end 3.90882 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08181 ms - Host latency: 3.59932 ms (end to end 3.9329 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08361 ms - Host latency: 3.60181 ms (end to end 3.90991 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.0859 ms - Host latency: 3.59798 ms (end to end 3.95728 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08119 ms - Host latency: 3.59527 ms (end to end 3.954 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07595 ms - Host latency: 3.58896 ms (end to end 3.94341 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07646 ms - Host latency: 3.58859 ms (end to end 3.94504 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07343 ms - Host latency: 3.58929 ms (end to end 3.95541 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07256 ms - Host latency: 3.58862 ms (end to end 3.88307 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06905 ms - Host latency: 3.58344 ms (end to end 3.85944 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06982 ms - Host latency: 3.58773 ms (end to end 3.90623 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07001 ms - Host latency: 3.57917 ms (end to end 3.90061 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07655 ms - Host latency: 3.58373 ms (end to end 3.90395 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07853 ms - Host latency: 3.58831 ms (end to end 3.9058 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07631 ms - Host latency: 3.58784 ms (end to end 3.90829 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08112 ms - Host latency: 3.59154 ms (end to end 3.91902 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08069 ms - Host latency: 3.59853 ms (end to end 3.92418 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07534 ms - Host latency: 3.59345 ms (end to end 3.91703 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06892 ms - Host latency: 3.58472 ms (end to end 3.88657 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.0684 ms - Host latency: 3.58553 ms (end to end 3.87766 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07126 ms - Host latency: 3.59153 ms (end to end 3.90643 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07008 ms - Host latency: 3.58796 ms (end to end 3.90218 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06745 ms - Host latency: 3.58542 ms (end to end 3.88456 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.17168 ms - Host latency: 3.68802 ms (end to end 4.02769 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.1178 ms - Host latency: 3.63817 ms (end to end 4.02917 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.068 ms - Host latency: 3.58591 ms (end to end 3.88401 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07126 ms - Host latency: 3.59105 ms (end to end 3.90474 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06775 ms - Host latency: 3.58568 ms (end to end 3.90659 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06849 ms - Host latency: 3.58615 ms (end to end 3.89694 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06934 ms - Host latency: 3.58459 ms (end to end 3.86191 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06837 ms - Host latency: 3.58687 ms (end to end 3.9057 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06765 ms - Host latency: 3.58049 ms (end to end 3.89189 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06915 ms - Host latency: 3.57844 ms (end to end 3.88886 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.0671 ms - Host latency: 3.57811 ms (end to end 3.88663 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06788 ms - Host latency: 3.57843 ms (end to end 3.88611 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06871 ms - Host latency: 3.58359 ms (end to end 3.90055 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.067 ms - Host latency: 3.5824 ms (end to end 3.86156 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06725 ms - Host latency: 3.58101 ms (end to end 3.91063 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06835 ms - Host latency: 3.58391 ms (end to end 3.87012 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06916 ms - Host latency: 3.58564 ms (end to end 3.90912 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06703 ms - Host latency: 3.58517 ms (end to end 3.9162 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06669 ms - Host latency: 3.58282 ms (end to end 3.9166 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06691 ms - Host latency: 3.58252 ms (end to end 3.8729 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.22102 ms - Host latency: 3.74155 ms (end to end 4.20087 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06812 ms - Host latency: 3.58162 ms (end to end 3.90349 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.0691 ms - Host latency: 3.57849 ms (end to end 3.89543 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06837 ms - Host latency: 3.57621 ms (end to end 3.903 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06873 ms - Host latency: 3.58269 ms (end to end 3.88623 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07047 ms - Host latency: 3.58979 ms (end to end 3.84941 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07338 ms - Host latency: 3.58607 ms (end to end 3.94023 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07474 ms - Host latency: 3.58573 ms (end to end 3.94072 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07509 ms - Host latency: 3.58824 ms (end to end 3.93969 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07688 ms - Host latency: 3.58815 ms (end to end 3.94366 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07437 ms - Host latency: 3.58877 ms (end to end 3.93783 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07208 ms - Host latency: 3.58466 ms (end to end 3.93403 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07495 ms - Host latency: 3.5886 ms (end to end 3.93893 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07366 ms - Host latency: 3.58656 ms (end to end 3.93948 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07583 ms - Host latency: 3.58754 ms (end to end 3.94219 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08022 ms - Host latency: 3.59531 ms (end to end 3.95045 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07533 ms - Host latency: 3.58809 ms (end to end 3.93776 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07561 ms - Host latency: 3.58906 ms (end to end 3.93855 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07683 ms - Host latency: 3.59023 ms (end to end 3.94022 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08062 ms - Host latency: 3.59453 ms (end to end 3.94686 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07983 ms - Host latency: 3.59514 ms (end to end 3.94866 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07177 ms - Host latency: 3.5875 ms (end to end 3.9473 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07145 ms - Host latency: 3.58569 ms (end to end 3.90929 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06757 ms - Host latency: 3.58076 ms (end to end 3.81578 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06818 ms - Host latency: 3.58544 ms (end to end 3.8675 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06815 ms - Host latency: 3.58741 ms (end to end 3.87761 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06488 ms - Host latency: 3.57692 ms (end to end 3.89135 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06948 ms - Host latency: 3.57888 ms (end to end 3.89121 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06791 ms - Host latency: 3.57843 ms (end to end 3.88979 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06749 ms - Host latency: 3.57751 ms (end to end 3.89247 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.0668 ms - Host latency: 3.57714 ms (end to end 3.89194 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06648 ms - Host latency: 3.57965 ms (end to end 3.88988 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06763 ms - Host latency: 3.57524 ms (end to end 3.90728 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06681 ms - Host latency: 3.57814 ms (end to end 3.90048 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06624 ms - Host latency: 3.57675 ms (end to end 3.89222 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06903 ms - Host latency: 3.58156 ms (end to end 3.90083 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06788 ms - Host latency: 3.57639 ms (end to end 3.91526 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07632 ms - Host latency: 3.59089 ms (end to end 3.98756 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.16801 ms - Host latency: 3.68024 ms (end to end 4.20354 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.11268 ms - Host latency: 3.62527 ms (end to end 4.06084 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07039 ms - Host latency: 3.5902 ms (end to end 3.95754 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07175 ms - Host latency: 3.58973 ms (end to end 3.96304 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06791 ms - Host latency: 3.58422 ms (end to end 3.95135 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07048 ms - Host latency: 3.58649 ms (end to end 3.95338 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07124 ms - Host latency: 3.58933 ms (end to end 3.94866 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06794 ms - Host latency: 3.58677 ms (end to end 3.96606 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07329 ms - Host latency: 3.59248 ms (end to end 3.96724 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07263 ms - Host latency: 3.59197 ms (end to end 3.96113 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07092 ms - Host latency: 3.58931 ms (end to end 3.9552 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07002 ms - Host latency: 3.5877 ms (end to end 3.96116 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07104 ms - Host latency: 3.5906 ms (end to end 3.95913 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.07126 ms - Host latency: 3.5925 ms (end to end 3.95891 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06873 ms - Host latency: 3.58472 ms (end to end 3.95713 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.06948 ms - Host latency: 3.58687 ms (end to end 3.95994 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08179 ms - Host latency: 3.6019 ms (end to end 3.95996 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.09592 ms - Host latency: 3.61252 ms (end to end 4.00115 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.10327 ms - Host latency: 3.61953 ms (end to end 4.02246 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.26594 ms - Host latency: 3.78328 ms (end to end 4.3501 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.09846 ms - Host latency: 3.621 ms (end to end 4.0095 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.09854 ms - Host latency: 3.61353 ms (end to end 3.9978 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.0887 ms - Host latency: 3.60793 ms (end to end 3.97942 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08633 ms - Host latency: 3.60313 ms (end to end 3.97515 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.0874 ms - Host latency: 3.60588 ms (end to end 3.97876 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08577 ms - Host latency: 3.60674 ms (end to end 3.97852 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08704 ms - Host latency: 3.60454 ms (end to end 3.97256 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08208 ms - Host latency: 3.59675 ms (end to end 3.9665 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08406 ms - Host latency: 3.60386 ms (end to end 3.97463 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08391 ms - Host latency: 3.60132 ms (end to end 3.9699 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08411 ms - Host latency: 3.60098 ms (end to end 3.97324 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08206 ms - Host latency: 3.599 ms (end to end 3.94602 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08167 ms - Host latency: 3.59746 ms (end to end 3.92168 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08154 ms - Host latency: 3.59548 ms (end to end 3.93716 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08105 ms - Host latency: 3.59124 ms (end to end 3.91655 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08333 ms - Host latency: 3.59395 ms (end to end 3.91631 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08108 ms - Host latency: 3.58828 ms (end to end 3.9259 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08381 ms - Host latency: 3.59189 ms (end to end 3.91384 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08342 ms - Host latency: 3.59255 ms (end to end 3.90391 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08418 ms - Host latency: 3.59443 ms (end to end 3.91169 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08398 ms - Host latency: 3.59431 ms (end to end 3.91343 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08318 ms - Host latency: 3.59141 ms (end to end 3.93123 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08208 ms - Host latency: 3.58875 ms (end to end 3.9291 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.0813 ms - Host latency: 3.58948 ms (end to end 3.93354 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08367 ms - Host latency: 3.59033 ms (end to end 3.93875 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08127 ms - Host latency: 3.5906 ms (end to end 3.9324 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08137 ms - Host latency: 3.58918 ms (end to end 3.92607 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08071 ms - Host latency: 3.58923 ms (end to end 3.91992 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08203 ms - Host latency: 3.59041 ms (end to end 3.91882 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08176 ms - Host latency: 3.59109 ms (end to end 3.92378 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08181 ms - Host latency: 3.59224 ms (end to end 3.91277 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08247 ms - Host latency: 3.59573 ms (end to end 3.89873 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08809 ms - Host latency: 3.59917 ms (end to end 3.96917 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.08618 ms - Host latency: 3.59473 ms (end to end 3.97898 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.09172 ms - Host latency: 3.6031 ms (end to end 4.0551 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.09915 ms - Host latency: 3.62112 ms (end to end 4.05015 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.0873 ms - Host latency: 3.6053 ms (end to end 4.01492 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.23391 ms - Host latency: 3.75632 ms (end to end 4.20364 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.13196 ms - Host latency: 3.64783 ms (end to end 3.97832 ms)
[08/05/2021-19:03:15] [I] Average on 10 runs - GPU latency: 2.09226 ms - Host latency: 3.60869 ms (end to end 3.98752 ms)
[08/05/2021-19:03:15] [I] Host latency
[08/05/2021-19:03:15] [I] min: 3.54727 ms (end to end 3.60052 ms)
[08/05/2021-19:03:15] [I] max: 4.39954 ms (end to end 5.49467 ms)
[08/05/2021-19:03:15] [I] mean: 3.59939 ms (end to end 3.94678 ms)
[08/05/2021-19:03:15] [I] median: 3.58875 ms (end to end 3.92859 ms)
[08/05/2021-19:03:15] [I] percentile: 4.10449 ms at 99% (end to end 4.55884 ms at 99%)
[08/05/2021-19:03:15] [I] throughput: 15195.9 qps
[08/05/2021-19:03:15] [I] walltime: 3.00502 s
[08/05/2021-19:03:15] [I] GPU Compute
[08/05/2021-19:03:15] [I] min: 2.04272 ms
[08/05/2021-19:03:15] [I] max: 2.88765 ms
[08/05/2021-19:03:15] [I] mean: 2.08505 ms
[08/05/2021-19:03:15] [I] median: 2.07471 ms
[08/05/2021-19:03:15] [I] percentile: 2.58887 ms at 99%
[08/05/2021-19:03:15] [I] total compute time: 2.97537 s
&&&& PASSED TensorRT.trtexec # trtexec --loadEngine=resnet50.trt --batch=32
Setting shape goes to,
&&&& RUNNING TensorRT.trtexec # trtexec --loadEngine=resnet50.trt --shapes=input:32x3x224x224 --explicitBatch
[08/05/2021-22:50:20] [I] === Model Options ===
[08/05/2021-22:50:20] [I] Format: *
[08/05/2021-22:50:20] [I] Model:
[08/05/2021-22:50:20] [I] Output:
[08/05/2021-22:50:20] [I] === Build Options ===
[08/05/2021-22:50:20] [I] Max batch: explicit
[08/05/2021-22:50:20] [I] Workspace: 16 MB
[08/05/2021-22:50:20] [I] minTiming: 1
[08/05/2021-22:50:20] [I] avgTiming: 8
[08/05/2021-22:50:20] [I] Precision: FP32
[08/05/2021-22:50:20] [I] Calibration:
[08/05/2021-22:50:20] [I] Safe mode: Disabled
[08/05/2021-22:50:20] [I] Save engine:
[08/05/2021-22:50:20] [I] Load engine: resnet50.trt
[08/05/2021-22:50:20] [I] Inputs format: fp32:CHW
[08/05/2021-22:50:20] [I] Outputs format: fp32:CHW
[08/05/2021-22:50:20] [I] Input build shape: input=32x3x224x224+32x3x224x224+32x3x224x224
[08/05/2021-22:50:20] [I] === System Options ===
[08/05/2021-22:50:20] [I] Device: 0
[08/05/2021-22:50:20] [I] DLACore:
[08/05/2021-22:50:20] [I] Plugins:
[08/05/2021-22:50:20] [I] === Inference Options ===
[08/05/2021-22:50:20] [I] Batch: Explicit
[08/05/2021-22:50:20] [I] Iterations: 10
[08/05/2021-22:50:20] [I] Duration: 3s (+ 200ms warm up)
[08/05/2021-22:50:20] [I] Sleep time: 0ms
[08/05/2021-22:50:20] [I] Streams: 1
[08/05/2021-22:50:20] [I] ExposeDMA: Disabled
[08/05/2021-22:50:20] [I] Spin-wait: Disabled
[08/05/2021-22:50:20] [I] Multithreading: Disabled
[08/05/2021-22:50:20] [I] CUDA Graph: Disabled
[08/05/2021-22:50:20] [I] Skip inference: Disabled
[08/05/2021-22:50:20] [I] Inputs:
[08/05/2021-22:50:20] [I] === Reporting Options ===
[08/05/2021-22:50:20] [I] Verbose: Disabled
[08/05/2021-22:50:20] [I] Averages: 10 inferences
[08/05/2021-22:50:20] [I] Percentile: 99
[08/05/2021-22:50:20] [I] Dump output: Disabled
[08/05/2021-22:50:20] [I] Profile: Disabled
[08/05/2021-22:50:20] [I] Export timing to JSON file:
[08/05/2021-22:50:20] [I] Export output to JSON file:
[08/05/2021-22:50:20] [I] Export profile to JSON file:
[08/05/2021-22:50:20] [I]
[08/05/2021-22:50:24] [I] Warmup completed 0 queries over 200 ms
[08/05/2021-22:50:24] [I] Timing trace has 0 queries over 3.00572 s
[08/05/2021-22:50:24] [I] Trace averages of 10 runs:
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.0458 ms - Host latency: 2.11439 ms (end to end 3.85903 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04544 ms - Host latency: 2.11686 ms (end to end 3.88474 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.05156 ms - Host latency: 2.12396 ms (end to end 3.89991 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.03818 ms - Host latency: 2.10934 ms (end to end 3.87325 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.03408 ms - Host latency: 2.1047 ms (end to end 3.86815 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.03502 ms - Host latency: 2.10625 ms (end to end 3.8654 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.03469 ms - Host latency: 2.10358 ms (end to end 3.87639 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.03475 ms - Host latency: 2.10411 ms (end to end 3.86059 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.03312 ms - Host latency: 2.10402 ms (end to end 3.85675 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.03482 ms - Host latency: 2.10825 ms (end to end 3.85409 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.03488 ms - Host latency: 2.10237 ms (end to end 3.58867 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04418 ms - Host latency: 2.11529 ms (end to end 3.68989 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04408 ms - Host latency: 2.11363 ms (end to end 3.82722 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.044 ms - Host latency: 2.1136 ms (end to end 3.83397 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04399 ms - Host latency: 2.11329 ms (end to end 3.82332 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04383 ms - Host latency: 2.11389 ms (end to end 3.83578 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.10689 ms - Host latency: 2.17693 ms (end to end 3.95345 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.13951 ms - Host latency: 2.20862 ms (end to end 4.03106 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.03262 ms - Host latency: 2.10308 ms (end to end 3.8084 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.0322 ms - Host latency: 2.10178 ms (end to end 3.81121 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.03254 ms - Host latency: 2.10217 ms (end to end 3.7957 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.03003 ms - Host latency: 2.09984 ms (end to end 3.79481 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.03001 ms - Host latency: 2.09912 ms (end to end 3.80732 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.03017 ms - Host latency: 2.10048 ms (end to end 3.80677 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.03036 ms - Host latency: 2.09966 ms (end to end 3.80651 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.02993 ms - Host latency: 2.09943 ms (end to end 3.80826 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.03138 ms - Host latency: 2.10139 ms (end to end 3.80778 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.03048 ms - Host latency: 2.09985 ms (end to end 3.81302 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.02998 ms - Host latency: 2.09961 ms (end to end 3.80854 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.03002 ms - Host latency: 2.09979 ms (end to end 3.81658 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.02988 ms - Host latency: 2.09923 ms (end to end 3.82995 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.03422 ms - Host latency: 2.09442 ms (end to end 3.87727 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.03401 ms - Host latency: 2.09267 ms (end to end 3.88882 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.03044 ms - Host latency: 2.10898 ms (end to end 3.80391 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.03192 ms - Host latency: 2.11232 ms (end to end 3.81103 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.03957 ms - Host latency: 2.10487 ms (end to end 3.89248 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.1524 ms - Host latency: 2.21779 ms (end to end 4.116 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.09506 ms - Host latency: 2.16051 ms (end to end 3.99609 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.03988 ms - Host latency: 2.10527 ms (end to end 3.888 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04702 ms - Host latency: 2.11706 ms (end to end 3.77853 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.0508 ms - Host latency: 2.11532 ms (end to end 3.3437 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.05797 ms - Host latency: 2.12361 ms (end to end 3.89656 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.0577 ms - Host latency: 2.12366 ms (end to end 3.89325 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.0589 ms - Host latency: 2.12579 ms (end to end 3.86136 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.05957 ms - Host latency: 2.12673 ms (end to end 3.86663 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.05664 ms - Host latency: 2.12208 ms (end to end 3.89711 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.05082 ms - Host latency: 2.11716 ms (end to end 3.87977 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.05026 ms - Host latency: 2.11613 ms (end to end 3.88085 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.05105 ms - Host latency: 2.11855 ms (end to end 3.84607 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04982 ms - Host latency: 2.11637 ms (end to end 3.85081 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04877 ms - Host latency: 2.11581 ms (end to end 3.84159 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04744 ms - Host latency: 2.11252 ms (end to end 3.87119 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04895 ms - Host latency: 2.11494 ms (end to end 3.8781 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04788 ms - Host latency: 2.1157 ms (end to end 3.86852 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.0446 ms - Host latency: 2.1248 ms (end to end 3.81759 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04434 ms - Host latency: 2.12197 ms (end to end 3.80798 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04429 ms - Host latency: 2.11367 ms (end to end 3.83337 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04393 ms - Host latency: 2.11294 ms (end to end 3.83149 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04427 ms - Host latency: 2.11372 ms (end to end 3.82933 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04376 ms - Host latency: 2.11338 ms (end to end 3.82817 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.0444 ms - Host latency: 2.11376 ms (end to end 3.83051 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04562 ms - Host latency: 2.11504 ms (end to end 3.81761 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04524 ms - Host latency: 2.11464 ms (end to end 3.82958 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04501 ms - Host latency: 2.11742 ms (end to end 3.80693 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04513 ms - Host latency: 2.11725 ms (end to end 3.83011 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04563 ms - Host latency: 2.1121 ms (end to end 3.89615 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.0458 ms - Host latency: 2.11254 ms (end to end 3.90043 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04554 ms - Host latency: 2.11169 ms (end to end 3.90334 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04775 ms - Host latency: 2.11431 ms (end to end 3.89227 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04629 ms - Host latency: 2.1144 ms (end to end 3.8303 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04988 ms - Host latency: 2.11603 ms (end to end 3.88124 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04877 ms - Host latency: 2.1151 ms (end to end 3.87844 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04924 ms - Host latency: 2.11691 ms (end to end 3.86573 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04619 ms - Host latency: 2.11234 ms (end to end 3.91251 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04504 ms - Host latency: 2.10919 ms (end to end 3.95609 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.20315 ms - Host latency: 2.2701 ms (end to end 4.21594 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.0455 ms - Host latency: 2.11116 ms (end to end 3.88937 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04528 ms - Host latency: 2.11138 ms (end to end 3.89127 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04618 ms - Host latency: 2.11251 ms (end to end 3.88793 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.05244 ms - Host latency: 2.11876 ms (end to end 3.88529 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.05143 ms - Host latency: 2.11759 ms (end to end 3.88224 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04764 ms - Host latency: 2.11517 ms (end to end 3.88208 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.048 ms - Host latency: 2.11478 ms (end to end 3.85712 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04673 ms - Host latency: 2.11396 ms (end to end 3.82443 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04697 ms - Host latency: 2.11327 ms (end to end 3.87886 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04768 ms - Host latency: 2.11478 ms (end to end 3.87354 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04998 ms - Host latency: 2.1158 ms (end to end 3.88672 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.05038 ms - Host latency: 2.11791 ms (end to end 3.85313 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04744 ms - Host latency: 2.1149 ms (end to end 3.84352 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04863 ms - Host latency: 2.1146 ms (end to end 3.88168 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04739 ms - Host latency: 2.11423 ms (end to end 3.87727 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04761 ms - Host latency: 2.11428 ms (end to end 3.87585 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04697 ms - Host latency: 2.11362 ms (end to end 3.86855 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04648 ms - Host latency: 2.10864 ms (end to end 3.93589 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.15076 ms - Host latency: 2.21209 ms (end to end 4.13518 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.08875 ms - Host latency: 2.15608 ms (end to end 3.96975 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04387 ms - Host latency: 2.11919 ms (end to end 3.84548 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04412 ms - Host latency: 2.11523 ms (end to end 3.87642 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04485 ms - Host latency: 2.1104 ms (end to end 3.93948 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04441 ms - Host latency: 2.10925 ms (end to end 3.93479 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04932 ms - Host latency: 2.11543 ms (end to end 3.90452 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04844 ms - Host latency: 2.11445 ms (end to end 3.87605 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.05312 ms - Host latency: 2.11907 ms (end to end 3.89158 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.05312 ms - Host latency: 2.11929 ms (end to end 3.8844 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.05447 ms - Host latency: 2.12109 ms (end to end 3.88955 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.05188 ms - Host latency: 2.11848 ms (end to end 3.88862 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04949 ms - Host latency: 2.11567 ms (end to end 3.86445 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.054 ms - Host latency: 2.12085 ms (end to end 3.85505 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.05093 ms - Host latency: 2.11697 ms (end to end 3.87581 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.05327 ms - Host latency: 2.11926 ms (end to end 3.88994 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.05115 ms - Host latency: 2.11689 ms (end to end 3.8812 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.05142 ms - Host latency: 2.11836 ms (end to end 3.86497 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.05132 ms - Host latency: 2.11785 ms (end to end 3.86853 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.05173 ms - Host latency: 2.11877 ms (end to end 3.85642 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.05078 ms - Host latency: 2.1165 ms (end to end 3.88088 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04622 ms - Host latency: 2.11724 ms (end to end 3.82698 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04524 ms - Host latency: 2.12048 ms (end to end 3.7918 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04707 ms - Host latency: 2.11807 ms (end to end 3.82341 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04644 ms - Host latency: 2.11714 ms (end to end 3.83191 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04353 ms - Host latency: 2.11272 ms (end to end 3.84133 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04524 ms - Host latency: 2.11389 ms (end to end 3.84019 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04341 ms - Host latency: 2.11328 ms (end to end 3.83916 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.0446 ms - Host latency: 2.11384 ms (end to end 3.84282 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04358 ms - Host latency: 2.11196 ms (end to end 3.83875 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.045 ms - Host latency: 2.11426 ms (end to end 3.84534 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04453 ms - Host latency: 2.11372 ms (end to end 3.84229 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04827 ms - Host latency: 2.11741 ms (end to end 3.85522 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04812 ms - Host latency: 2.11787 ms (end to end 3.8897 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04434 ms - Host latency: 2.11838 ms (end to end 3.84993 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04849 ms - Host latency: 2.11834 ms (end to end 3.87639 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04619 ms - Host latency: 2.11965 ms (end to end 3.85608 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04595 ms - Host latency: 2.11616 ms (end to end 3.83691 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04426 ms - Host latency: 2.11313 ms (end to end 3.83582 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.15349 ms - Host latency: 2.22197 ms (end to end 4.06038 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.09153 ms - Host latency: 2.16016 ms (end to end 3.93813 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04304 ms - Host latency: 2.1123 ms (end to end 3.83 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04685 ms - Host latency: 2.11592 ms (end to end 3.85623 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04377 ms - Host latency: 2.11948 ms (end to end 3.81934 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.0438 ms - Host latency: 2.11323 ms (end to end 3.83022 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04392 ms - Host latency: 2.11401 ms (end to end 3.83152 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04429 ms - Host latency: 2.11379 ms (end to end 3.82854 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.05359 ms - Host latency: 2.12224 ms (end to end 3.93667 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04436 ms - Host latency: 2.11411 ms (end to end 3.82173 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04451 ms - Host latency: 2.12656 ms (end to end 3.79387 ms)
[08/05/2021-22:50:24] [I] Average on 10 runs - GPU latency: 2.04575 ms - Host latency: 2.12356 ms (end to end 3.79424 ms)
[08/05/2021-22:50:24] [I] Host latency
[08/05/2021-22:50:24] [I] min: 2.08783 ms (end to end 2.10767 ms)
[08/05/2021-22:50:24] [I] max: 2.84589 ms (end to end 5.06152 ms)
[08/05/2021-22:50:24] [I] mean: 2.11868 ms (end to end 3.86127 ms)
[08/05/2021-22:50:24] [I] median: 2.11377 ms (end to end 3.86914 ms)
[08/05/2021-22:50:24] [I] percentile: 2.51477 ms at 99% (end to end 4.41522 ms at 99%)
[08/05/2021-22:50:24] [I] throughput: 0 qps
[08/05/2021-22:50:24] [I] walltime: 3.00572 s
[08/05/2021-22:50:24] [I] GPU Compute
[08/05/2021-22:50:24] [I] min: 2.02637 ms
[08/05/2021-22:50:24] [I] max: 2.78119 ms
[08/05/2021-22:50:24] [I] mean: 2.0501 ms
[08/05/2021-22:50:24] [I] median: 2.04526 ms
[08/05/2021-22:50:24] [I] percentile: 2.44788 ms at 99%
[08/05/2021-22:50:24] [I] total compute time: 2.99109 s
&&&& PASSED TensorRT.trtexec # trtexec --loadEngine=resnet50.trt --shapes=input:32x3x224x224 --explicitBatch