# Inferring detectnet\_v2 .trt model in python **URL:** \ **Category:** TAO Toolkit\ **Tags:** tensorrt\ **Created:** [January 7, 2021, 12:16pm UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403 "2021-01-07T12:16:27Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![jazeel.jk](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jazeel.jk/32/72899_2.png) [@jazeel.jk](https://forums.developer.nvidia.com/u/jazeel.jk)\ **Post date:** [January 7, 2021, 12:16pm UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/1 "2021-01-07T12:16:27Z")
I have trained a detectnet\_v2 model and then converted it to .trt engine… I am able to infer it using tlt-infer inside the docker. But how can i use this .trt file outside the docker in my own python application ? ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 7, 2021, 1:17pm UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/3 "2021-01-07T13:17:51Z")
@jazeel.jk Please refer to [Run PeopleNet with tensorrt - #21 by carlos.alvarez](https://forums.developer.nvidia.com/t/run-peoplenet-with-tensorrt/128000/21) ---
**Author:** ![jazeel.jk](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jazeel.jk/32/72899_2.png) [@jazeel.jk](https://forums.developer.nvidia.com/u/jazeel.jk)\ **Post date:** [January 13, 2021, 10:12am UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/4 "2021-01-13T10:12:10Z")
Hi @Morganh, Thanks, it worked for detectnet\_v2 made .trt file inference. But now i created a .engine file using yolo. And i am getting the following error. I am running it inside the container itself with my code. ``` [TensorRT] ERROR: INVALID_ARGUMENT: getPluginCreator could not find plugin BatchTilePlugin_TRT version 1 [TensorRT] ERROR: safeDeserializationUtils.cpp (293) - Serialization Error in load: 0 (Cannot deserialize plugin since corresponding IPluginCreator not found in Plugin Registry) [TensorRT] ERROR: INVALID_STATE: std::exception [TensorRT] ERROR: INVALID_CONFIG: Deserialize the cuda engine failed. Traceback (most recent call last): File "main.py", line 36, in context = trt_engine.create_execution_context() AttributeError: 'NoneType' object has no attribute 'create_execution_context' root@605c1edbf876:/workspace/tlt-experiments/box_jeston# ``` could you please help me if you have a solution? Thanks… ---
**Author:** ![jazeel.jk](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jazeel.jk/32/72899_2.png) [@jazeel.jk](https://forums.developer.nvidia.com/u/jazeel.jk)\ **Post date:** [January 13, 2021, 12:14pm UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/5 "2021-01-13T12:14:05Z")
Should i build TensorRT OSS inside the container ? i read these plugins are required for yolov3, batchTilePlugin, resizeNearestPlugin and batchedNMSPlugin. But can i build TensorRT OSS inside nvidia tlt container ? ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 13, 2021, 12:15pm UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/6 "2021-01-13T12:15:09Z")
Please refer to github [GitHub - NVIDIA-AI-IOT/deepstream\_tao\_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream](https://github.com/NVIDIA-AI-IOT/deepstream_tlt_apps) mentioned in [https://docs.nvidia.com/metropolis/TLT/tlt-getting-started-guide/text/deploying\_to\_deepstream.html#generating-an-engine-using-tlt-converter](https://docs.nvidia.com/metropolis/TLT/tlt-getting-started-guide/text/deploying_to_deepstream.html#generating-an-engine-using-tlt-converter) > This is **ONLY** needed when running _SSD_ , _DSSD_ , _RetinaNet_ , _YOLOV3_ and _MaskRCNN_ models because some TRT plugins such as BatchTilePlugin required by these models is not supported by TensorRT7.x native package. ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 13, 2021, 12:15pm UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/7 "2021-01-13T12:15:16Z")
Moving this topic into TLT forum. ---
**Author:** ![jazeel.jk](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jazeel.jk/32/72899_2.png) [@jazeel.jk](https://forums.developer.nvidia.com/u/jazeel.jk)\ **Post date:** [January 13, 2021, 12:27pm UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/8 "2021-01-13T12:27:56Z")
Yes. I saw that. " TensorRT OSS on x86"… But i want to run the code inside the nvidia tlt container itself. Is it possible to build TensorRT OSS inside the container? ---
**Author:** ![jazeel.jk](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jazeel.jk/32/72899_2.png) [@jazeel.jk](https://forums.developer.nvidia.com/u/jazeel.jk)\ **Post date:** [January 13, 2021, 12:30pm UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/9 "2021-01-13T12:30:13Z")
And inside the nvidia tlt container tensorrt version is 7.0.0.11, should i downgrade it for batchTilePlugin to support? ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 13, 2021, 1:13pm UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/10 "2021-01-13T13:13:45Z")
See [Invalid device function error when export .tlt file to .etlt - #16 by Morganh](https://forums.developer.nvidia.com/t/invalid-device-function-error-when-export-tlt-file-to-etlt/123482/16) , it is possible to build OSS inside the docker. > And inside the nvidia tlt container tensorrt version is 7.0.0.11, should i downgrade it for batchTilePlugin to support? git clone -b **release/7.0** [GitHub - NVIDIA/TensorRT: NVIDIA® TensorRT™, an SDK for high-performance deep learning inference, includes a deep learning inference optimizer and runtime that delivers low latency and high throughput for inference applications.](https://github.com/nvidia/TensorRT) TensorRT && \ ---
**Author:** ![jazeel.jk](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jazeel.jk/32/72899_2.png) [@jazeel.jk](https://forums.developer.nvidia.com/u/jazeel.jk)\ **Post date:** [January 13, 2021, 1:38pm UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/11 "2021-01-13T13:38:10Z")
Thank you so much… And how i can determine DGPU\_ARCHS variable for my GPU? My GPU is GeForce GTX 1660 Ti/PCIe/SSE2 ---
**Author:** ![jazeel.jk](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jazeel.jk/32/72899_2.png) [@jazeel.jk](https://forums.developer.nvidia.com/u/jazeel.jk)\ **Post date:** [January 13, 2021, 1:59pm UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/12 "2021-01-13T13:59:30Z")
I was not able to get it from deviceQuery.cpp. This is my nividia-smi output, ``` +-----------------------------------------------------------------------------+ | NVIDIA-SMI 440.100 Driver Version: 440.100 CUDA Version: 10.2 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 GeForce GTX 166... Off | 00000000:01:00.0 On | N/A | | N/A 49C P8 8W / N/A | 1159MiB / 5941MiB | 32% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| +-----------------------------------------------------------------------------+ ``` ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 13, 2021, 3:00pm UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/13 "2021-01-13T15:00:47Z")
Reference: > **[NVIDIA GeForce GTX 16 Series Graphics Card](https://www.nvidia.com/en-us/geforce/graphics-cards/16-series/)** > > Get 1.4X Power Efficiency Over Previous Generation for a Faster, Cooler, Quieter Gaming Experience. > **[NVIDIA CUDA GPU Compute Capability](https://developer.nvidia.com/cuda/gpus)** > > Find the compute capability for your GPU. > **[Support Matrix — NVIDIA TensorRT](https://docs.nvidia.com/deeplearning/tensorrt/latest/getting-started/support-matrix.html)** > **[deepstream\_tao\_apps/TRT-OSS/Jetson at master · NVIDIA-AI-IOT/deepstream\_tao\_apps](https://github.com/NVIDIA-AI-IOT/deepstream_tao_apps/tree/master/TRT-OSS/Jetson)** > > Sample apps to demonstrate how to deploy models trained with TAO on DeepStream - NVIDIA-AI-IOT/deepstream\_tao\_apps > [@compute capability ?](https://forums.developer.nvidia.com/t/compute-capability/110091): > > hello, question are all silly: what does Nvidia mean by compute capability? , because the Titan RTX, GTX 10xO and RTX 20x0 have the same score, but are of different technologies and the benchmark is not explained Your card 's compute capability is 7.5 . Please set GPU\_ARCHS to 75. ---
**Author:** ![jazeel.jk](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jazeel.jk/32/72899_2.png) [@jazeel.jk](https://forums.developer.nvidia.com/u/jazeel.jk)\ **Post date:** [January 13, 2021, 4:56pm UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/14 "2021-01-13T16:56:14Z")
Thanks @Morganh… I did the full steps you mentioned to for Installing TRT OSS to the base docker. But still the same error remains. After installing TRT OSS, it should get those required plugins right? But still “could not find plugin BatchTilePlugin\_TRT version 1” error is there… ``` [TensorRT] ERROR: INVALID_ARGUMENT: getPluginCreator could not find plugin BatchTilePlugin_TRT version 1 [TensorRT] ERROR: safeDeserializationUtils.cpp (293) - Serialization Error in load: 0 (Cannot deserialize plugin since corresponding IPluginCreator not found in Plugin Registry) [TensorRT] ERROR: INVALID_STATE: std::exception [TensorRT] ERROR: INVALID_CONFIG: Deserialize the cuda engine failed. Traceback (most recent call last): File "main.py", line 36, in context = trt_engine.create_execution_context() AttributeError: 'NoneType' object has no attribute 'create_execution_context' root@605c1edbf876:/workspace/tlt-experiments/box_jeston# ``` ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 14, 2021, 3:37pm UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/15 "2021-01-14T15:37:41Z")
Have you replaced the plugin “libnvinfer\_plugin.so\*”? See more in [deepstream\_tao\_apps/TRT-OSS/Jetson at master · NVIDIA-AI-IOT/deepstream\_tao\_apps · GitHub](https://github.com/NVIDIA-AI-IOT/deepstream_tlt_apps/tree/master/TRT-OSS/Jetson) ---
**Author:** ![jazeel.jk](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jazeel.jk/32/72899_2.png) [@jazeel.jk](https://forums.developer.nvidia.com/u/jazeel.jk)\ **Post date:** [January 14, 2021, 4:23pm UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/16 "2021-01-14T16:23:33Z")
Yes i did these steps, ``` cp out/libnvinfer_plugin.so.7.0.0.1 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.7.0.0 && \ cp out/libnvinfer_plugin_static.a /usr/lib/x86_64-linux-gnu/libnvinfer_plugin_static.a && \ cd ../../../ && \ rm -rf trt_oss_src ``` ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 14, 2021, 4:26pm UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/17 "2021-01-14T16:26:07Z")
> [@jazeel.jk](#): > > `/usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so` Can you share below result? $ ll -sh /usr/lib/x86\_64-linux-gnu/libnvinfer\_plugin.so\* ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 14, 2021, 4:29pm UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/18 "2021-01-14T16:29:05Z")
Please take a look at my comment shared in below topic. > [@Failling in building sample from TLT-DEEPSTREAM](https://forums.developer.nvidia.com/t/failling-in-building-sample-from-tlt-deepstream/127038/15): > > Please see below. Original: $ ll -sh /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so\* 0 lrwxrwxrwx 1 root root 26 Apr 26 16:41 /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so → libnvinfer\_plugin.so.7.1.0 0 lrwxrwxrwx 1 root root 26 Apr 26 16:41 /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so.7 → libnvinfer\_plugin.so.7.1.0 4.5M -rw-r–r-- 1 root root 4.5M Apr 26 16:38 /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so.7.1.0 If $ sudo mv /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so.7.… ---
**Author:** ![jazeel.jk](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jazeel.jk/32/72899_2.png) [@jazeel.jk](https://forums.developer.nvidia.com/u/jazeel.jk)\ **Post date:** [January 14, 2021, 4:30pm UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/19 "2021-01-14T16:30:26Z")
this is the output for ll -sh /usr/lib/x86\_64-linux-gnu/libnvinfer\_plugin.so\*, ``` 0 lrwxrwxrwx 1 root root 26 Dec 17 2019 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so -> libnvinfer_plugin.so.7.0.0 0 lrwxrwxrwx 1 root root 26 Dec 17 2019 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.7 -> libnvinfer_plugin.so.7.0.0 4.1M -rw-r--r-- 1 root root 4.1M Jan 13 16:47 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.7.0.0 ``` ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 14, 2021, 4:33pm UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/20 "2021-01-14T16:33:15Z")
Please check my comment in above link. Please see below. Please follow the exact step shared in the github. > Original: > > > $ ll -sh /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so\* > > > > 0 lrwxrwxrwx 1 root root 26 Apr 26 16:41 /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so → libnvinfer\_plugin.so.7.1.0 > > 0 lrwxrwxrwx 1 root root 26 Apr 26 16:41 /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so.7 → libnvinfer\_plugin.so.7.1.0 > > 4.5M -rw-r–r-- 1 root root 4.5M Apr 26 16:38 /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so.7.1.0 > > If > > > $ sudo mv /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so.7.1.0 ~/libnvinfer\_plugin.so.7.1.0.bak > > > > $ sudo cp {TRT\_SOURCE}/build/out/libnvinfer\_plugin.so.7.0.0.1 /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so.7.1.0 > > > > $ sudo ldconfig > > then > > > nvidia@nvidia:~$ ll /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so\* > > lrwxrwxrwx 1 root root 26 Apr 27 08:53 /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so → libnvinfer\_plugin.so.7.1.0\* > > lrwxrwxrwx 1 root root 26 Apr 27 08:53 /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so.7 → libnvinfer\_plugin.so.7.1.0\* > > lrwxrwxrwx 1 root root 26 Apr 27 08:55 /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so.7.0.0 → libnvinfer\_plugin.so.7.1.0\* > > -rwxr-xr-x 1 root root 4652648 Apr 27 08:55 /usr/lib/aarch64-linux-gnu/libnvinfer\_plugin.so.7.1.0\* > > That’s the expected now. ---
**Author:** ![jazeel.jk](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jazeel.jk/32/72899_2.png) [@jazeel.jk](https://forums.developer.nvidia.com/u/jazeel.jk)\ **Post date:** [January 18, 2021, 6:48am UTC](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403/21 "2021-01-18T06:48:21Z")
Thanks… I followed the steps given in the github repo… Installation was successful. But for sudo ldconfig it’s not returning any logs. [Next page](https://forums.developer.nvidia.com/t/inferring-detectnet-v2-trt-model-in-python/165403.md?page=2)