Inferring detectnet_v2 .trt model in python

jazeel.jk · January 18, 2021, 7:01am

On giving this command make nvinfer_plugin -j$(nproc) i could see two warnings. Apart from that installation was successful, and inside build/out folder, libnvinfer_plugin.so, libnvinfer_plugin.so.7.0.0 , libnvinfer_plugin.so.7.0.0.1 these files got generated.

these are the final few lines of the log,

/workspace/TensorRT/plugin/multilevelProposeROI/multilevelProposeROIPlugin.cpp: In member function ‘virtual int nvinfer1::plugin::MultilevelProposeROI::enqueue(int, const void* const*, void**, void*, cudaStream_t)’:
/workspace/TensorRT/plugin/multilevelProposeROI/multilevelProposeROIPlugin.cpp:408:46: warning: pointer of type ‘void *’ used in arithmetic [-Wpointer-arith]
             mParam, proposal_ws, workspace + kernel_workspace_offset,
                                              ^
/workspace/TensorRT/plugin/multilevelProposeROI/multilevelProposeROIPlugin.cpp:425:29: warning: pointer of type ‘void *’ used in arithmetic [-Wpointer-arith]
                 workspace + kernel_workspace_offset,
                             ^
[ 60%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/detectionForward.cu.o
[ 60%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/extractFgScores.cu.o
[ 60%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/gatherTopDetections.cu.o
[ 66%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/generateAnchors.cu.o
[ 66%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/gridAnchorLayer.cu.o
[ 66%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/maskRCNNKernels.cu.o
[ 73%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/nmsLayer.cu.o
[ 73%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/normalizeLayer.cu.o
[ 73%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/permuteData.cu.o
[ 73%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/priorBoxLayer.cu.o
[ 80%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/proposalKernel.cu.o
[ 80%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/proposalsForward.cu.o
[ 80%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/regionForward.cu.o
[ 86%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/reorgForward.cu.o
[ 86%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/roiPooling.cu.o
[ 86%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/rproiInferenceFused.cu.o
/workspace/TensorRT/plugin/common/kernels/proposalKernel.cu(34): warning: variable "ALIGNMENT" was declared but never referenced

[ 93%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/cudaDriverWrapper.cu.o
[ 93%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/sortScoresPerImage.cu.o
[ 93%] Building CUDA object plugin/CMakeFiles/nvinfer_plugin.dir/common/kernels/sortScoresPerClass.cu.o
[ 93%] Building CXX object plugin/CMakeFiles/nvinfer_plugin.dir/InferPlugin.cpp.o
[100%] Building CXX object plugin/CMakeFiles/nvinfer_plugin.dir/__/samples/common/logger.cpp.o
[100%] Linking CUDA device code CMakeFiles/nvinfer_plugin.dir/cmake_device_link.o
[100%] Linking CXX shared library ../out/libnvinfer_plugin.so
[100%] Built target nvinfer_plugin

After this succesfully copied libnvinfer_plugin.so.7.0.0.1 file inside build/out folder to, usr/lib/x86_64-linux-gnu/ folder in the name libnvinfer_plugin.so.7.0.0

and still sudo ldconfig is not returning any output and while inferring, getPluginCreator could not find plugin BatchTilePlugin_TRT version 1 error is there

Morganh · January 18, 2021, 7:00am

Yes, there is no log. That’s expected.
Please check again now with
$ ls -sh /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so*

jazeel.jk · January 18, 2021, 7:02am

0 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so 0 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.7 4.7M /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.7.0.0

this is the output now…
while inferring g etPluginCreator could not find plugin BatchTilePlugin_TRT version 1 error is still there

Morganh · January 18, 2021, 7:05am

As mentioned above, please make sure you can get similar link and soft link under /usr/lib/x86_64-linux-gnu/

nvidia@nvidia:~$ ll /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so*
lrwxrwxrwx 1 root root 26 Apr 27 08:53 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so -> libnvinfer_plugin.so.7.1.0*
lrwxrwxrwx 1 root root 26 Apr 27 08:53 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7 -> libnvinfer_plugin.so.7.1.0*
lrwxrwxrwx 1 root root 26 Apr 27 08:55 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7.0.0 -> libnvinfer_plugin.so.7.1.0*
-rwxr-xr-x 1 root root 4652648 Apr 27 08:55 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7.1.0*

jazeel.jk · January 18, 2021, 7:08am

Yes,

root@f9f07db124db:/# ll /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so*
lrwxrwxrwx 1 root root      26 Dec 17  2019 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so -> libnvinfer_plugin.so.7.0.0*
lrwxrwxrwx 1 root root      26 Dec 17  2019 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.7 -> libnvinfer_plugin.so.7.0.0*
-rwxr-xr-x 1 root root 4918136 Jan 18 06:42 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.7.0.0*

Morganh · January 18, 2021, 7:13am

I am afraid it is not expected.
Can you double check below step mentioned above?

Please see below.

Original:

$ ll -sh /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so*

0 lrwxrwxrwx 1 root root 26 Apr 26 16:41 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so -> libnvinfer_plugin.so.7.1.0
0 lrwxrwxrwx 1 root root 26 Apr 26 16:41 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7 -> libnvinfer_plugin.so.7.1.0
4.5M -rw-r–r-- 1 root root 4.5M Apr 26 16:38 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7.1.0

If

$ sudo mv /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7.1.0 ~/libnvinfer_plugin.so.7.1.0.bak

$ sudo cp {TRT_SOURCE}/build/out/libnvinfer_plugin.so.7.0.0.1 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7.1.0

$ sudo ldconfig

then

nvidia@nvidia:~$ ll /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so*
lrwxrwxrwx 1 root root 26 Apr 27 08:53 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so -> libnvinfer_plugin.so.7.1.0*
lrwxrwxrwx 1 root root 26 Apr 27 08:53 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7 -> libnvinfer_plugin.so.7.1.0*
lrwxrwxrwx 1 root root 26 Apr 27 08:55 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7.0.0 -> libnvinfer_plugin.so.7.1.0*
-rwxr-xr-x 1 root root 4652648 Apr 27 08:55 /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7.1.0*

jazeel.jk · January 18, 2021, 7:45am

Initially,

root@4c0b40b94840:/workspace/TensorRT/build# ll -sh /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so*
      0 lrwxrwxrwx 1 root root  26 Dec 17  2019 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so -> libnvinfer_plugin.so.7.0.0
      0 lrwxrwxrwx 1 root root  26 Dec 17  2019 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.7 -> libnvinfer_plugin.so.7.0.0
    15M -rw-r--r-- 1 root root 15M Aug  3 16:18 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.7.0.0

then ,
sudo mv /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.7.0.0 ~/libnvinfer_plugin.so.7.0.0.bak
sudo cp pwd/out/libnvinfer_plugin.so.7.0.0.1 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.7.0.0
sudo ldconfig

after that,

root@4c0b40b94840:/workspace/TensorRT/build# ll -sh /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so*
   0 lrwxrwxrwx 1 root root   26 Dec 17  2019 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so -> libnvinfer_plugin.so.7.0.0*
   0 lrwxrwxrwx 1 root root   26 Dec 17  2019 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.7 -> libnvinfer_plugin.so.7.0.0*
4.7M -rwxr-xr-x 1 root root 4.7M Jan 18 07:36 /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.7.0.0*

So this is the issue,
I am not receiving anything like the third line in the log you shared, something like /usr/lib/x86_64-linux-gnu/libnvinfer_plugin.so.7.0.0 -> libnvinfer_plugin.so.7.0.0* is not coming for me…

Morganh · January 18, 2021, 9:22am

I will try to run your steps on my side. Please correct me if any.

Trigger 2.0_py3 tlt docker
Build TRT OSS
replace the plugin

jazeel.jk · January 18, 2021, 9:30am

before bulding trt oss, cmake was installed… wget https://github.com/Kitware/CMake/releases/download/v3.13.5/cmake-3.13.5.tar.gz

Morganh · January 18, 2021, 9:41am

One extra question: Is it a must for you to run inference in the tlt container?
Actually after you trained an etlt model, you can copy this etlt model to Jetson devices(such as Nano, NX, Xavier, TX2, etc), then run tlt-converter jetpack version to generate trt engine.
Then build the TRT OSS in the Jetson device and replace the plugin. Then run inference in the Jetson device.

jazeel.jk · January 18, 2021, 9:48am

I was trying to run it in my system. Outside docker i tried to install tensorrt, while importing there are some issues which is yet to solve. But inside the container, tensorrt is already installed and i could infer detectnet_v2 made trt engine, successfully with my python code… But for infering trt engine made with yolo, these plugins need to be installed… there I am facing these issues…

Morganh · January 18, 2021, 9:50am

Got it. So your request is that how to run inference successfully in tlt docker with the existing yolo trt engine.

jazeel.jk · January 18, 2021, 9:52am

Yes. True… And in near future i will be running this in Jetson, but currently yes… I am stuck inferring yolo trt engine inside tlt docker, with a standalone python script…

Morganh · January 18, 2021, 9:55am

Thanks for the info.

jazeel.jk · January 18, 2021, 10:03am

I really appreciate the help you are doing…

Morganh · January 18, 2021, 10:10am

Have you run tlt-infer successfully against the same yolo trt engine?

jazeel.jk · January 18, 2021, 10:12am

Yes I did…

Morganh · January 18, 2021, 10:14am

Can you share your standalone inference script here?

jazeel.jk · January 18, 2021, 10:20am

trt_loader.py (9.0 KB) main.py (2.2 KB)

main.py is the one i run… it will load trt_loader… works fine for detectnet_v2

jazeel.jk · January 18, 2021, 10:29am

Same methods as you suggested…