How are Benchmarks's FPS calculated

sudo python3 benchmark.py --model_name ssd-mobilenet-v1 --csv_file_path /home/eod/nfs/jetson_benchmarks/benchmark_csv/nx-benchmarks.csv --model_dir /home/eod/nfs/jetson_benchmarks/models/

show

--------------------------

Model Name: ssd-mobilenet-v1 
FPS:784.39 

--------------------------

but

 ./trtexec --onnx=/home/eod/nfs/jetson_benchmarks/models/ssd-mobilenet-v1-bs8.onnx --explicitBatch --int8 --workspace=2048 --saveEngine=/home/eod/nfs/jetson_benchmarks/models/ssd-mobilenet-v1_b8_ws2048_gpu.engine

./trtexec --onnx=/home/eod/nfs/jetson_benchmarks/models/ssd-mobilenet-v1-bs8.onnx --explicitBatch --int8 --workspace=2048 --avgRuns=100 --duration=180 --loadEngine=/home/eod/nfs/jetson_benchmarks/models/ssd-mobilenet-v1_b8_ws2048_gpu.engine

[04/08/2021-17:00:58] [I] Starting inference threads
[04/08/2021-17:03:58] [I] Warmup completed 0 queries over 200 ms
[04/08/2021-17:03:58] [I] Timing trace has 0 queries over 179.397 s
[04/08/2021-17:03:58] [I] Trace averages of 100 runs:
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.5291 ms - Host latency: 11.0219 ms (end to end 11.0332 ms, enqueue 2.0487 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.5514 ms - Host latency: 11.0461 ms (end to end 11.0576 ms, enqueue 2.1783 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.5812 ms - Host latency: 11.0771 ms (end to end 11.0883 ms, enqueue 2.22102 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.5617 ms - Host latency: 11.0543 ms (end to end 11.0657 ms, enqueue 2.12273 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.6209 ms - Host latency: 11.1166 ms (end to end 11.1289 ms, enqueue 2.26594 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.5643 ms - Host latency: 11.0559 ms (end to end 11.0675 ms, enqueue 2.16029 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.6062 ms - Host latency: 11.1017 ms (end to end 11.1126 ms, enqueue 2.18548 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.5899 ms - Host latency: 11.0846 ms (end to end 11.0961 ms, enqueue 2.14625 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.5658 ms - Host latency: 11.0598 ms (end to end 11.0707 ms, enqueue 2.08895 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.6279 ms - Host latency: 11.1243 ms (end to end 11.1358 ms, enqueue 2.25469 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.5788 ms - Host latency: 11.07 ms (end to end 11.0814 ms, enqueue 2.08209 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.6658 ms - Host latency: 11.1617 ms (end to end 11.1732 ms, enqueue 2.21092 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.6119 ms - Host latency: 11.1064 ms (end to end 11.1175 ms, enqueue 2.165 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.6492 ms - Host latency: 11.1416 ms (end to end 11.1531 ms, enqueue 2.11434 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.6725 ms - Host latency: 11.1702 ms (end to end 11.1819 ms, enqueue 2.18556 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.6237 ms - Host latency: 11.1159 ms (end to end 11.1276 ms, enqueue 2.19211 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.6785 ms - Host latency: 11.1753 ms (end to end 11.1876 ms, enqueue 2.1908 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.6262 ms - Host latency: 11.1192 ms (end to end 11.1307 ms, enqueue 2.20645 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.6949 ms - Host latency: 11.1889 ms (end to end 11.201 ms, enqueue 2.21525 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.7341 ms - Host latency: 11.2308 ms (end to end 11.2423 ms, enqueue 2.29701 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.651 ms - Host latency: 11.1436 ms (end to end 11.1548 ms, enqueue 2.12293 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.6953 ms - Host latency: 11.1918 ms (end to end 11.2039 ms, enqueue 2.25135 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.7114 ms - Host latency: 11.2041 ms (end to end 11.2153 ms, enqueue 2.20277 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.7073 ms - Host latency: 11.2024 ms (end to end 11.2137 ms, enqueue 2.30584 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.7455 ms - Host latency: 11.2416 ms (end to end 11.2529 ms, enqueue 2.26377 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.6901 ms - Host latency: 11.1817 ms (end to end 11.1939 ms, enqueue 2.11914 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.7216 ms - Host latency: 11.2192 ms (end to end 11.2305 ms, enqueue 2.31254 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.7407 ms - Host latency: 11.2335 ms (end to end 11.2447 ms, enqueue 2.34551 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.7601 ms - Host latency: 11.2544 ms (end to end 11.2663 ms, enqueue 2.22832 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.7883 ms - Host latency: 11.2845 ms (end to end 11.2956 ms, enqueue 2.33391 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.7177 ms - Host latency: 11.2093 ms (end to end 11.2207 ms, enqueue 2.12586 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.8166 ms - Host latency: 11.3128 ms (end to end 11.3244 ms, enqueue 2.32609 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.762 ms - Host latency: 11.2554 ms (end to end 11.2667 ms, enqueue 2.41578 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.7387 ms - Host latency: 11.2332 ms (end to end 11.2441 ms, enqueue 2.23996 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.8022 ms - Host latency: 11.298 ms (end to end 11.309 ms, enqueue 2.25531 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.7236 ms - Host latency: 11.2151 ms (end to end 11.2265 ms, enqueue 2.22082 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.8225 ms - Host latency: 11.3201 ms (end to end 11.3321 ms, enqueue 2.35359 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.7891 ms - Host latency: 11.2839 ms (end to end 11.2958 ms, enqueue 2.31238 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.7538 ms - Host latency: 11.2477 ms (end to end 11.2598 ms, enqueue 2.2677 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.8345 ms - Host latency: 11.3314 ms (end to end 11.3429 ms, enqueue 2.35578 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.7871 ms - Host latency: 11.28 ms (end to end 11.2924 ms, enqueue 2.26988 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.787 ms - Host latency: 11.2826 ms (end to end 11.2936 ms, enqueue 2.27187 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.8305 ms - Host latency: 11.3262 ms (end to end 11.3385 ms, enqueue 2.31297 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.774 ms - Host latency: 11.2668 ms (end to end 11.2788 ms, enqueue 2.2934 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.8906 ms - Host latency: 11.3871 ms (end to end 11.3982 ms, enqueue 2.42102 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.7898 ms - Host latency: 11.2832 ms (end to end 11.2943 ms, enqueue 2.3618 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.7773 ms - Host latency: 11.2724 ms (end to end 11.284 ms, enqueue 2.2643 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.8863 ms - Host latency: 11.382 ms (end to end 11.3932 ms, enqueue 2.35215 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.7651 ms - Host latency: 11.2569 ms (end to end 11.2684 ms, enqueue 2.25125 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.8723 ms - Host latency: 11.3692 ms (end to end 11.3807 ms, enqueue 2.28488 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.8838 ms - Host latency: 11.3791 ms (end to end 11.3903 ms, enqueue 2.31367 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.796 ms - Host latency: 11.2884 ms (end to end 11.3002 ms, enqueue 2.31566 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.891 ms - Host latency: 11.388 ms (end to end 11.3994 ms, enqueue 2.36895 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.8261 ms - Host latency: 11.3195 ms (end to end 11.3307 ms, enqueue 2.28328 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.8206 ms - Host latency: 11.3174 ms (end to end 11.3287 ms, enqueue 2.32215 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.8199 ms - Host latency: 11.3144 ms (end to end 11.3266 ms, enqueue 2.44367 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.7913 ms - Host latency: 11.2839 ms (end to end 11.2952 ms, enqueue 2.29539 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9056 ms - Host latency: 11.4027 ms (end to end 11.4143 ms, enqueue 2.34172 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.864 ms - Host latency: 11.3589 ms (end to end 11.3703 ms, enqueue 2.27195 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.8788 ms - Host latency: 11.3729 ms (end to end 11.3841 ms, enqueue 2.29289 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9362 ms - Host latency: 11.4327 ms (end to end 11.444 ms, enqueue 2.34 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.8088 ms - Host latency: 11.3003 ms (end to end 11.3129 ms, enqueue 2.29727 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.93 ms - Host latency: 11.4248 ms (end to end 11.4373 ms, enqueue 2.37555 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.8923 ms - Host latency: 11.3888 ms (end to end 11.4004 ms, enqueue 2.37633 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.8209 ms - Host latency: 11.3131 ms (end to end 11.3246 ms, enqueue 2.32953 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9456 ms - Host latency: 11.4428 ms (end to end 11.4539 ms, enqueue 2.42617 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.8998 ms - Host latency: 11.3941 ms (end to end 11.4072 ms, enqueue 2.41453 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9166 ms - Host latency: 11.4094 ms (end to end 11.4216 ms, enqueue 2.49422 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9475 ms - Host latency: 11.4444 ms (end to end 11.4555 ms, enqueue 2.37266 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.8657 ms - Host latency: 11.3577 ms (end to end 11.3693 ms, enqueue 2.25086 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.968 ms - Host latency: 11.4634 ms (end to end 11.4752 ms, enqueue 2.30641 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9391 ms - Host latency: 11.4358 ms (end to end 11.4471 ms, enqueue 2.47344 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.8372 ms - Host latency: 11.3296 ms (end to end 11.3413 ms, enqueue 2.34641 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9372 ms - Host latency: 11.4352 ms (end to end 11.4466 ms, enqueue 2.38969 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9263 ms - Host latency: 11.4202 ms (end to end 11.4316 ms, enqueue 2.29602 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9084 ms - Host latency: 11.4023 ms (end to end 11.4134 ms, enqueue 2.31492 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9577 ms - Host latency: 11.454 ms (end to end 11.4665 ms, enqueue 2.42328 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9237 ms - Host latency: 11.4172 ms (end to end 11.4292 ms, enqueue 2.34391 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.954 ms - Host latency: 11.4498 ms (end to end 11.4613 ms, enqueue 2.44734 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9557 ms - Host latency: 11.4516 ms (end to end 11.4635 ms, enqueue 2.3782 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9288 ms - Host latency: 11.4206 ms (end to end 11.4326 ms, enqueue 2.3125 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9669 ms - Host latency: 11.4634 ms (end to end 11.4749 ms, enqueue 2.39687 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9711 ms - Host latency: 11.4667 ms (end to end 11.4777 ms, enqueue 2.30461 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.858 ms - Host latency: 11.3505 ms (end to end 11.3612 ms, enqueue 2.25953 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0513 ms - Host latency: 11.5475 ms (end to end 11.5592 ms, enqueue 2.47539 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9416 ms - Host latency: 11.4376 ms (end to end 11.4485 ms, enqueue 2.32742 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.943 ms - Host latency: 11.4366 ms (end to end 11.448 ms, enqueue 2.32531 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.024 ms - Host latency: 11.5202 ms (end to end 11.5323 ms, enqueue 2.46187 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.8951 ms - Host latency: 11.3891 ms (end to end 11.4002 ms, enqueue 2.36789 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9723 ms - Host latency: 11.4668 ms (end to end 11.478 ms, enqueue 2.29375 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0363 ms - Host latency: 11.5327 ms (end to end 11.5441 ms, enqueue 2.41289 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.926 ms - Host latency: 11.4184 ms (end to end 11.4298 ms, enqueue 2.3268 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9913 ms - Host latency: 11.4871 ms (end to end 11.4993 ms, enqueue 2.36656 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9985 ms - Host latency: 11.4947 ms (end to end 11.5061 ms, enqueue 2.45992 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.879 ms - Host latency: 11.3721 ms (end to end 11.3834 ms, enqueue 2.27992 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.1148 ms - Host latency: 11.614 ms (end to end 11.6249 ms, enqueue 2.38156 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9553 ms - Host latency: 11.451 ms (end to end 11.4628 ms, enqueue 2.3193 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9278 ms - Host latency: 11.4207 ms (end to end 11.432 ms, enqueue 2.33766 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0599 ms - Host latency: 11.5563 ms (end to end 11.5676 ms, enqueue 2.44758 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9789 ms - Host latency: 11.473 ms (end to end 11.4844 ms, enqueue 2.50852 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0102 ms - Host latency: 11.5045 ms (end to end 11.5154 ms, enqueue 2.36727 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.081 ms - Host latency: 11.5771 ms (end to end 11.589 ms, enqueue 2.375 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9434 ms - Host latency: 11.4363 ms (end to end 11.4474 ms, enqueue 2.40969 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0215 ms - Host latency: 11.5142 ms (end to end 11.5271 ms, enqueue 2.35711 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.038 ms - Host latency: 11.5347 ms (end to end 11.5463 ms, enqueue 2.43344 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9727 ms - Host latency: 11.4643 ms (end to end 11.4769 ms, enqueue 2.33719 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0372 ms - Host latency: 11.533 ms (end to end 11.544 ms, enqueue 2.43859 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0472 ms - Host latency: 11.5423 ms (end to end 11.5541 ms, enqueue 2.45539 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9895 ms - Host latency: 11.4818 ms (end to end 11.4934 ms, enqueue 2.27141 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.1451 ms - Host latency: 11.6426 ms (end to end 11.6541 ms, enqueue 2.41672 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0328 ms - Host latency: 11.5283 ms (end to end 11.5398 ms, enqueue 2.38773 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9462 ms - Host latency: 11.4387 ms (end to end 11.4505 ms, enqueue 2.28781 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.1055 ms - Host latency: 11.6025 ms (end to end 11.6142 ms, enqueue 2.48891 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.073 ms - Host latency: 11.5672 ms (end to end 11.5793 ms, enqueue 2.39789 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0039 ms - Host latency: 11.4982 ms (end to end 11.5102 ms, enqueue 2.43359 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0989 ms - Host latency: 11.5939 ms (end to end 11.6062 ms, enqueue 2.47703 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9587 ms - Host latency: 11.453 ms (end to end 11.465 ms, enqueue 2.36359 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0523 ms - Host latency: 11.5455 ms (end to end 11.5578 ms, enqueue 2.32188 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.087 ms - Host latency: 11.5831 ms (end to end 11.5956 ms, enqueue 2.42688 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0486 ms - Host latency: 11.5423 ms (end to end 11.5536 ms, enqueue 2.47563 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.1209 ms - Host latency: 11.6144 ms (end to end 11.6277 ms, enqueue 2.31828 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.1072 ms - Host latency: 11.605 ms (end to end 11.6156 ms, enqueue 2.41922 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9747 ms - Host latency: 11.468 ms (end to end 11.4802 ms, enqueue 2.37812 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0925 ms - Host latency: 11.5886 ms (end to end 11.5989 ms, enqueue 2.39859 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0748 ms - Host latency: 11.5706 ms (end to end 11.583 ms, enqueue 2.43422 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9741 ms - Host latency: 11.468 ms (end to end 11.4791 ms, enqueue 2.32328 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.1158 ms - Host latency: 11.612 ms (end to end 11.6244 ms, enqueue 2.47094 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0703 ms - Host latency: 11.5658 ms (end to end 11.5769 ms, enqueue 2.39344 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.958 ms - Host latency: 11.4502 ms (end to end 11.463 ms, enqueue 2.32437 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.1042 ms - Host latency: 11.6002 ms (end to end 11.6111 ms, enqueue 2.46484 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0892 ms - Host latency: 11.5852 ms (end to end 11.5978 ms, enqueue 2.48031 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0273 ms - Host latency: 11.5214 ms (end to end 11.5325 ms, enqueue 2.33125 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.1233 ms - Host latency: 11.6211 ms (end to end 11.6327 ms, enqueue 2.40641 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.1184 ms - Host latency: 11.6134 ms (end to end 11.6256 ms, enqueue 2.33172 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0566 ms - Host latency: 11.5491 ms (end to end 11.56 ms, enqueue 2.27234 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0994 ms - Host latency: 11.5964 ms (end to end 11.6062 ms, enqueue 2.46312 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0427 ms - Host latency: 11.5375 ms (end to end 11.55 ms, enqueue 2.37812 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.065 ms - Host latency: 11.5589 ms (end to end 11.5708 ms, enqueue 2.33672 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.1122 ms - Host latency: 11.6086 ms (end to end 11.6223 ms, enqueue 2.38469 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0698 ms - Host latency: 11.5639 ms (end to end 11.5766 ms, enqueue 2.32516 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0514 ms - Host latency: 11.5463 ms (end to end 11.5589 ms, enqueue 2.42297 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.1159 ms - Host latency: 11.6145 ms (end to end 11.6252 ms, enqueue 2.43891 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0506 ms - Host latency: 11.543 ms (end to end 11.5553 ms, enqueue 2.45969 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0452 ms - Host latency: 11.5408 ms (end to end 11.5523 ms, enqueue 2.41609 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.1348 ms - Host latency: 11.6308 ms (end to end 11.6428 ms, enqueue 2.36719 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9603 ms - Host latency: 11.452 ms (end to end 11.4636 ms, enqueue 2.31297 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0578 ms - Host latency: 11.5525 ms (end to end 11.5652 ms, enqueue 2.32078 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.1567 ms - Host latency: 11.6531 ms (end to end 11.6658 ms, enqueue 2.40203 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9781 ms - Host latency: 11.4714 ms (end to end 11.4809 ms, enqueue 2.33016 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0995 ms - Host latency: 11.5961 ms (end to end 11.6081 ms, enqueue 2.36531 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.1097 ms - Host latency: 11.6067 ms (end to end 11.6177 ms, enqueue 2.51641 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.0084 ms - Host latency: 11.4994 ms (end to end 11.5106 ms, enqueue 2.35156 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.1292 ms - Host latency: 11.6263 ms (end to end 11.6384 ms, enqueue 2.42313 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.1453 ms - Host latency: 11.6416 ms (end to end 11.653 ms, enqueue 2.45156 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 10.9636 ms - Host latency: 11.4542 ms (end to end 11.4673 ms, enqueue 2.23172 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.1227 ms - Host latency: 11.6195 ms (end to end 11.6319 ms, enqueue 2.43203 ms)
[04/08/2021-17:03:58] [I] Average on 100 runs - GPU latency: 11.1158 ms - Host latency: 11.612 ms (end to end 11.6242 ms, enqueue 2.38125 ms)
[04/08/2021-17:03:58] [I] Host Latency
[04/08/2021-17:03:58] [I] min: 10.541 ms (end to end 10.5508 ms)
[04/08/2021-17:03:58] [I] max: 13.3281 ms (end to end 13.3594 ms)
[04/08/2021-17:03:58] [I] mean: 11.3989 ms (end to end 11.4105 ms)
[04/08/2021-17:03:58] [I] median: 11.2812 ms (end to end 11.2891 ms)
[04/08/2021-17:03:58] [I] percentile: 12.6719 ms at 99% (end to end 12.6875 ms at 99%)
[04/08/2021-17:03:58] [I] throughput: 0 qps
[04/08/2021-17:03:58] [I] walltime: 179.397 s
[04/08/2021-17:03:58] [I] Enqueue Time
[04/08/2021-17:03:58] [I] min: 1.22656 ms
[04/08/2021-17:03:58] [I] max: 11.6021 ms
[04/08/2021-17:03:58] [I] median: 2.20312 ms
[04/08/2021-17:03:58] [I] GPU Compute
[04/08/2021-17:03:58] [I] min: 10.0605 ms
[04/08/2021-17:03:58] [I] max: 12.8281 ms
[04/08/2021-17:03:58] [I] mean: 10.9041 ms
[04/08/2021-17:03:58] [I] median: 10.7832 ms
[04/08/2021-17:03:58] [I] percentile: 12.1719 ms at 99%
[04/08/2021-17:03:58] [I] total compute time: 171.424 s

it show model cost 10.0605 ms at least.
why the disparity between Benchmarks and my test ?

Hi,

The model takes ~10ms per run, 8x input (batchsize=8) per run.
The fps is roughly 800 fps.

Thanks.

1 Like