Inference time speed up?

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

hi @Morganh

I have trained an object detection model using mmdet hrnet. after converting the mmdet .pth file to .trt file on NX device I am getting inference time for one image is like 0.3 seconds. Before conversion to trt it was like 2 seconds for single image. Using trt model can we speed up the inference time less than to 0.3 seconds ? Any idea ?
thanks

2

Hi,
Request you to share the model, script, profiler and performance output if not shared already so that we can help you better.
Alternatively, you can try running your model with trtexec command.
https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/trtexec

While measuring the model performance, make sure you consider the latency and throughput of the network inference, excluding the data pre and post-processing overhead.
Please refer below link for more details:
https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt-722/best-practices/index.html#measure-performance
https://docs.nvidia.com/deeplearning/tensorrt/best-practices/index.html#model-accuracy

Thanks!

1 Like