Is it possible to know how much time each layer takes on TensorRT?

AI & Data Science Deep Learning (Training & Inference) TensorRT
#1

Hello,

I am doing some analysis of how fast TensorRT works compared to others like TensorFlow.

I was wondering, is it possible to know how long it takes for each layer of a model to be computed by TensorRT?

#2

Hi,

Request you to share the model, script, profiler, and performance output if not shared already so that we can help you better.

Alternatively, you can try running your model with trtexec command.

While measuring the model performance, make sure you consider the latency and throughput of the network inference, excluding the data pre and post-processing overhead.
Please refer to the below links for more details:
https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt-803/best-practices/index.html#measure-performance

https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt-803/best-practices/index.html#model-accuracy

Thanks!

#4

I am using jkjung’s YOLOv3-tiny implementation and want to know the time it takes to execute each layer.

github/jkjung-avt/tensorrt_demos/tree/master/yolo

#5

Hi,

We can get to know using trtexec command. Please use --dumpProfile --separateProfileRun options along with trtexec to check per-layer performance.

Thank you.