How to show every layer inference time in tensorRT?

mrsunqichang · March 27, 2021, 4:33pm

How to display every layer inference time in tensorRT?

I just read NVIDIA Deep Learning TensorRT Documentation and the link can not tell me how to display layers inference time,it’s just record the time in the example.

NVES · March 27, 2021, 4:37pm

Hi,
Request you to share the model, script, profiler and performance output if not shared already so that we can help you better.
Alternatively, you can try running your model with trtexec command.
https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec

While measuring the model performance, make sure you consider the latency and throughput of the network inference, excluding the data pre and post-processing overhead.
Please refer below link for more details:
https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt-722/best-practices/index.html#measure-performance

Thanks!

Topic		Replies	Views
Is it possible to know how much time each layer takes on TensorRT? TensorRT	3	898	April 27, 2022
TensorRT Latency measurement Jetson TX1	2	861	October 18, 2021
Performance of Neural Network on Jetson Jetson TX2	6	522	October 18, 2021
Print average model inference time using tensorrt executable TensorRT tensorrt	2	423	November 7, 2022
Slow inference UNet Industrial TF-TRT TensorRT tensorrt , tensorflow	1	458	July 2, 2023
Same inference speed with Resnet50 for int8 and fp16 Jetson Xavier NX jetson-inference	4	685	October 18, 2021
Inference Benchmarks - TensorRT Version ? TensorRT	1	2036	October 4, 2018
Ideas to maximize throughput using TensorRT TensorRT	1	359	November 20, 2020
Inference time speed up? TensorRT	1	467	August 3, 2021
Performance discrepancy using TensorRT engines TensorRT tensorrt	3	659	October 5, 2021