# Inferring detectnet\_v2 .trt model in python **URL:** \ **Category:** TAO Toolkit\ **Tags:** tensorrt\ **Created:** [January 7, 2021, 12:16pm UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403 "2021-01-07T12:16:27Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![jazeel.jk](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jazeel.jk/32/72899_2.png) [@jazeel.jk](https://forums.developer.nvidia.com/u/jazeel.jk)\ **Post date:** [January 18, 2021, 6:53am UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/22 "2021-01-18T06:53:35Z")
On giving this command **make nvinfer\_plugin -j$(nproc)** i could see two warnings. Apart from that installation was successful, and inside build/out folder, _libnvinfer\_plugin.so, libnvinfer\_plugin.so.7.0.0 , libnvinfer\_plugin.so.7.0.0.1_ these files got generated. these are the final few lines of the log, ``` /workspace/TensorRT/plugin/multilevelProposeROI/multilevelProposeROIPlugin.cpp: In member function ‘virtual int nvinfer1::plugin::MultilevelProposeROI::enqueue(int, const void* const*, void**, void*, cudaStream_t)’: /workspace/TensorRT/plugin/multilevelProposeROI/multilevelProposeROIPlugin.cpp:408:46: warning: pointer of type ‘void *’ used in arithmetic [-Wpointer-arith] mParam, proposal_ws, workspace + kernel_workspace_offset, ^ /workspace/TensorRT/plugin/multilevelProposeROI/multilevelProposeROIPlugin.cpp:425:29: warning: pointer of type ‘void *’ used in arithmetic [-Wpointer-arith] workspace + kernel_workspace_offset, ^ [60%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/detectionForward.cu.o [60%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/extractFgScores.cu.o [60%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/gatherTopDetections.cu.o [66%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/generateAnchors.cu.o [66%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/gridAnchorLayer.cu.o [66%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/maskRCNNKernels.cu.o [73%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/nmsLayer.cu.o [73%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/normalizeLayer.cu.o [73%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/permuteData.cu.o [73%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/priorBoxLayer.cu.o [80%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/proposalKernel.cu.o [80%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/proposalsForward.cu.o [80%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/regionForward.cu.o [86%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/reorgForward.cu.o [86%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/roiPooling.cu.o [86%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/rproiInferenceFused.cu.o /workspace/TensorRT/plugin/common/kernels/proposalKernel.cu(34): warning: variable "ALIGNMENT" was declared but never referenced [93%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/cudaDriverWrapper.cu.o [93%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/sortScoresPerImage.cu.o [93%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/sortScoresPerClass.cu.o [93%] Building CXX object plugin/CMakeFiles/nvinfer_plugin.dir/InferPlugin.cpp.o [100%] Building CXX object plugin/CMakeFiles/nvinfer_plugin.dir/__/samples/common/logger.cpp.o [100%] Linking CUDA device code CMakeFiles/nvinfer_plugin.dir/cmake_device_link.o [100%] Linking CXX shared library ../out/libnvinfer_plugin.so [100%] Built target nvinfer_plugin ``` After this succesfully copied _libnvinfer\_plugin.so.7.0.0.1_ file inside build/out folder to, _usr/lib/x86\_64-linux-gnu/_ folder in the name _libnvinfer\_plugin.so.7.0.0_ and still sudo ldconfig is not returning any output and while inferring, g_etPluginCreator could not find plugin BatchTilePlugin\_TRT version 1_ error is there ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 18, 2021, 7:00am UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/23 "2021-01-18T07:00:40Z")
> [@jazeel.jk](#): > > for sudo ldconfig it’s not returning any logs. Yes, there is no log. That’s expected. Please check again now with $ ls -sh /usr/lib/x86\_64-linux-gnu/libnvinfer\_plugin.so\* ---
**Author:** ![jazeel.jk](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jazeel.jk/32/72899_2.png) [@jazeel.jk](https://forums.developer.nvidia.com/u/jazeel.jk)\ **Post date:** [January 18, 2021, 7:02am UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/24 "2021-01-18T07:02:22Z")
`0 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so 0 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.7 4.7M /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.7.0.0` this is the output now… while inferring g _etPluginCreator could not find plugin BatchTilePlugin\_TRT version 1_ error is still there ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 18, 2021, 7:04am UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/25 "2021-01-18T07:04:40Z")
As mentioned above, please make sure you can get similar link and soft link under /usr/lib/x86\_64-linux-gnu/ nvidia@nvidia:~$ ll /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so\* lrwxrwxrwx 1 root root 26 Apr 27 08:53 /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so → libnvinfer\_plugin.so.7.1.0\* lrwxrwxrwx 1 root root 26 Apr 27 08:53 /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so.7 → libnvinfer\_plugin.so.7.1.0\* lrwxrwxrwx 1 root root 26 Apr 27 08:55 /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so.7.0.0 → libnvinfer\_plugin.so.7.1.0\* -rwxr-xr-x 1 root root 4652648 Apr 27 08:55 /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so.7.1.0\* ---
**Author:** ![jazeel.jk](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jazeel.jk/32/72899_2.png) [@jazeel.jk](https://forums.developer.nvidia.com/u/jazeel.jk)\ **Post date:** [January 18, 2021, 7:08am UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/26 "2021-01-18T07:08:54Z")
Yes, ``` root@f9f07db124db:/# ll /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so* lrwxrwxrwx 1 root root 26 Dec 17 2019 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so -> libnvinfer_plugin.so.7.0.0* lrwxrwxrwx 1 root root 26 Dec 17 2019 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.7 -> libnvinfer_plugin.so.7.0.0* -rwxr-xr-x 1 root root 4918136 Jan 18 06:42 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.7.0.0* ``` ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 18, 2021, 7:13am UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/27 "2021-01-18T07:13:45Z")
I am afraid it is not expected. Can you double check below step mentioned above? Please see below. Original: > $ ll -sh /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so\* > > 0 lrwxrwxrwx 1 root root 26 Apr 26 16:41 /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so → libnvinfer\_plugin.so.7.1.0 > 0 lrwxrwxrwx 1 root root 26 Apr 26 16:41 /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so.7 → libnvinfer\_plugin.so.7.1.0 > 4.5M -rw-r–r-- 1 root root 4.5M Apr 26 16:38 /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so.7.1.0 If > $ sudo mv /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so.7.1.0 ~/libnvinfer\_plugin.so.7.1.0.bak > > $ sudo cp {TRT\_SOURCE}/build/out/libnvinfer\_plugin.so.7.0.0.1 /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so.7.1.0 > > $ sudo ldconfig then > nvidia@nvidia:~$ ll /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so\* > lrwxrwxrwx 1 root root 26 Apr 27 08:53 /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so → libnvinfer\_plugin.so.7.1.0\* > lrwxrwxrwx 1 root root 26 Apr 27 08:53 /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so.7 → libnvinfer\_plugin.so.7.1.0\* > lrwxrwxrwx 1 root root 26 Apr 27 08:55 /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so.7.0.0 → libnvinfer\_plugin.so.7.1.0\* > -rwxr-xr-x 1 root root 4652648 Apr 27 08:55 /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so.7.1.0\* ---
**Author:** ![jazeel.jk](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jazeel.jk/32/72899_2.png) [@jazeel.jk](https://forums.developer.nvidia.com/u/jazeel.jk)\ **Post date:** [January 18, 2021, 7:44am UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/28 "2021-01-18T07:44:59Z")
Initially, ``` root@4c0b40b94840:/workspace/TensorRT/build# ll -sh /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so* 0 lrwxrwxrwx 1 root root 26 Dec 17 2019 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so -> libnvinfer_plugin.so.7.0.0 0 lrwxrwxrwx 1 root root 26 Dec 17 2019 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.7 -> libnvinfer_plugin.so.7.0.0 15M -rw-r--r-- 1 root root 15M Aug 3 16:18 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.7.0.0 ``` then , sudo mv /usr/lib/x86\_64-linux-gnu/libnvinfer\_plugin.so.7.0.0 ~/libnvinfer\_plugin.so.7.0.0.bak sudo cp `pwd`/out/libnvinfer\_plugin.so.7.0.0.1 /usr/lib/x86\_64-linux-gnu/libnvinfer\_plugin.so.7.0.0 sudo ldconfig after that, ``` root@4c0b40b94840:/workspace/TensorRT/build# ll -sh /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so* 0 lrwxrwxrwx 1 root root 26 Dec 17 2019 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so -> libnvinfer_plugin.so.7.0.0* 0 lrwxrwxrwx 1 root root 26 Dec 17 2019 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.7 -> libnvinfer_plugin.so.7.0.0* 4.7M -rwxr-xr-x 1 root root 4.7M Jan 18 07:36 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.7.0.0* ``` So this is the issue, I am not receiving anything like the third line in the log you shared, something like /usr/lib/x86\_64-linux-gnu/libnvinfer\_plugin.so.7.0.0 → libnvinfer\_plugin.so.7.0.0\* is not coming for me… ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 18, 2021, 9:22am UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/29 "2021-01-18T09:22:03Z")
I will try to run your steps on my side. Please correct me if any. 1. Trigger 2.0\_py3 tlt docker 2. Build TRT OSS 3. replace the plugin ---
**Author:** ![jazeel.jk](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jazeel.jk/32/72899_2.png) [@jazeel.jk](https://forums.developer.nvidia.com/u/jazeel.jk)\ **Post date:** [January 18, 2021, 9:30am UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/30 "2021-01-18T09:30:17Z")
before bulding trt oss, cmake was installed… wget [https://github.com/Kitware/CMake/releases/download/v3.13.5/cmake-3.13.5.tar.gz](https://github.com/Kitware/CMake/releases/download/v3.13.5/cmake-3.13.5.tar.gz) ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 18, 2021, 9:41am UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/31 "2021-01-18T09:41:29Z")
One extra question: Is it a must for you to run inference in the tlt container? Actually after you trained an etlt model, you can copy this etlt model to Jetson devices(such as Nano, NX, Xavier, TX2, etc), then run tlt-converter jetpack version to generate trt engine. Then build the TRT OSS in the Jetson device and replace the plugin. Then run inference in the Jetson device. ---
**Author:** ![jazeel.jk](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jazeel.jk/32/72899_2.png) [@jazeel.jk](https://forums.developer.nvidia.com/u/jazeel.jk)\ **Post date:** [January 18, 2021, 9:48am UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/32 "2021-01-18T09:48:00Z")
I was trying to run it in my system. Outside docker i tried to install tensorrt, while importing there are some issues which is yet to solve. But inside the container, tensorrt is already installed and i could infer detectnet\_v2 made trt engine, successfully with my python code… But for infering trt engine made with yolo, these plugins need to be installed… there I am facing these issues… ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 18, 2021, 9:50am UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/33 "2021-01-18T09:50:34Z")
Got it. So your request is that how to run inference successfully in tlt docker with the existing yolo trt engine. ---
**Author:** ![jazeel.jk](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jazeel.jk/32/72899_2.png) [@jazeel.jk](https://forums.developer.nvidia.com/u/jazeel.jk)\ **Post date:** [January 18, 2021, 9:52am UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/34 "2021-01-18T09:52:47Z")
Yes. True… And in near future i will be running this in Jetson, but currently yes… I am stuck inferring yolo trt engine inside tlt docker, with a standalone python script… ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 18, 2021, 9:55am UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/35 "2021-01-18T09:55:50Z")
Thanks for the info. ---
**Author:** ![jazeel.jk](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jazeel.jk/32/72899_2.png) [@jazeel.jk](https://forums.developer.nvidia.com/u/jazeel.jk)\ **Post date:** [January 18, 2021, 10:03am UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/36 "2021-01-18T10:03:03Z")
I really appreciate the help you are doing… ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 18, 2021, 10:10am UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/37 "2021-01-18T10:10:11Z")
Have you run tlt-infer successfully against the same yolo trt engine? ---
**Author:** ![jazeel.jk](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jazeel.jk/32/72899_2.png) [@jazeel.jk](https://forums.developer.nvidia.com/u/jazeel.jk)\ **Post date:** [January 18, 2021, 10:12am UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/38 "2021-01-18T10:12:54Z")
Yes I did… ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 18, 2021, 10:14am UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/39 "2021-01-18T10:14:53Z")
Can you share your standalone inference script here? ---
**Author:** ![jazeel.jk](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jazeel.jk/32/72899_2.png) [@jazeel.jk](https://forums.developer.nvidia.com/u/jazeel.jk)\ **Post date:** [January 18, 2021, 10:20am UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/40 "2021-01-18T10:20:16Z")
[trt\_loader.py](https://forums.developer.nvidia.com/uploads/short-url/tUFNUGYeaJTcfiE5QZ6Bz7iZP7O.py) (9.0 KB) [main.py](https://forums.developer.nvidia.com/uploads/short-url/uWu678VwzeF0dJaJeUZTJmuESpE.py) (2.2 KB) main.py is the one i run… it will load trt\_loader… works fine for detectnet\_v2 ---
**Author:** ![jazeel.jk](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jazeel.jk/32/72899_2.png) [@jazeel.jk](https://forums.developer.nvidia.com/u/jazeel.jk)\ **Post date:** [January 18, 2021, 10:29am UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/41 "2021-01-18T10:29:47Z")
Same methods as you suggested… > [@Inferring detectnet\_v2 .trt model in python](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/3): > > @jazeel.jk Please refer to [Run PeopleNet with tensorrt - #21 by carlos.alvarez](https://forums.developer.nvidia.com/t/run-peoplenet-with-tensorrt/128000/21) [Previous page](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403.md?page=1) [Next page](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403.md?page=3)