Deploy TRT Object Detection model (Mobilenetv2) with Deepstream Error:"Failed to parse bboxes"

[DemolationPeter]

Thanks for the quick reply @AastaLLL .

You solution works but I had to more than just copy the lines in the .config file.

Under “Workflow” everyone can find my way to the working model. The workflow might be hard to understand and to follow on some parts especially because I used Deepstream5.0 in an Docker environment. Therefor the workflow can change if you don’t use Docker.
I hope that my post can help others to get Mobilenetv2 also working in Deepstream and TensorRT. Also you can find some files in the post. I’m aware that you do the conversion prettier but it is working for me.

Important

I use the Deepstream5.0 Docker Container (deepstream:5.0.1-20.09-triton) and Tensorflow1.15

---

Workflow

1. Download the Mobilenetv2 Model from the Tensorflow 1 Model Zoo
2. I created 2 Python Files for the conversion of the frozen_inference_graph.pb into a TensorRT Model with the UFF Parser (See Update 2 please!!!)

For the conversion into an UFF Model
First File: uff_converter.py (2.9 KB)

For the conversion I used the Deepstream5.0 Docker Container but you need to install Tensorflow1 (I used 1.15.5) inside the container. Therefore you have to install pip for python3. Than you can install tensorflow-gpu==1.15.5 for example. With Tensorflow you get the graphsurgeon and uff libary and you can convert the .pb model to .uff with the .py script.

You can also use the TensorRT Docker Container but careful you have to use TensorRT7.0.0 Container because the Deepstream5.0 Container uses Tensor7.0.0. If you don’t use the same TensorRT Version you get errors in Deepstream. For the conversion you have to run a script in the Docker Container which installs Tensorflow1.15.5 and other libarys and programs. You can find the script under /opt/tensorrt/python and then run . python_setup.sh. Then you can also do the conversion.

UFF into TRT
Second File: trt_engine_build_from_uff.py (1.0 KB)

After the uff Conversion you can use this script. These script transforms the uff model into a TRT Model.
My TRT Model of Mobilenetv2 with 90 classes is roughly 80Mb. (FP32)

3. Know you need to create .config files to run the TRT Model with Deepstream5.0.

Therefore I used the .config files from the samples and source folder in Deepstream5.0. You can find them under /opt/nvidia/deepstream/deepstream-5.0/ in the Docker Container. I
I used the .configs from the /opt/nvidia/deepstream/deepstream-5.0/sources/objectDetector_SSD. You need the whole directory. These are the config files I used for my inference. You have to change the path for your project. Also you need a label.txt file.

config_infer_primary_ssd.txt (3.3 KB)
source_1080p_dec_infer_mobilenetv2_tf.txt (4.3 KB)
labels.txt (702 Bytes)

4. Compile the Makefile in the objectDetector_SSD/nvdsinfer_custom_impl_ssd folder

The Makefile in the folder creates a file (libnvdsinfer_custom_impl_ssd.so). Know Deepstream can use the special detection parser for the SSD Model and knows what to do with “NMS” Output layer you created in the uff conversion . If you don’t have these file an the correct path to it you get also errors.

Important
I had to do changes in the Makefile otherwise I encountered also errors. For the Deepstream5.0 Container I used these Makefile.
Makefile (1.9 KB)
The important changes are:

• CUDA_VER:=10.2
• CFLAGS+= -I…/…/includes -I/usr/local/cuda-$(CUDA_VER)/include -I/opt/nvidia/deepstream/deepstream-5.0/sources/includes
• LIBS:= -lnvinfer -lnvparsers -L/usr/local/cuda-$(CUDA_VER)/lib64 -lcudart -lcublas -L/opt/nvidia/deepstream/deepstream-5.0/sources/libs

Otherwise you get errors that the .cpp files can’t find the libraries (include “NvInferPlugin.h”
and include “nvdsinfer_custom_impl.h”)

5. You need to copy the .uff Model, TRT Model, and config Models in a folder on your host which the Deepstream Container can use.

I copied the /opt/nvidia/deepstream/deepstream-5.0/deepstream folder with the /samples and /sources directory into a directory on my host which the host and the Docker Container have access too. Than you have to check the the config files for the correct path to the uff, TRT Model, config files, label.txt and the libnvdsinfer_custom_impl_ssd file.

6. You can start the inference with “deepstream-app -c deepstream_app_config_ssd.txt”

---

Sources:

For the .config files I used these sources:

Optimize NVIDIA GPU performance for efficient model inference

Google Colaboratory

jkjung-avt/tensorrt_demos/blob/master/ssd/build_engine.py

GitHub - AastaNV/TRT_object_detection: Python sample for referencing object...

Changes to the Makefile: (careful for docker the paths are different!!!)

Error while running SSD provided in Deepstream

Hi, The error comes from the missing header. Please add the TensorRT path into the include parameter. And the path should be end in …/lib or …/inlcude. For example CUDA_VER:=10.0 CC:= g++ CFLAGS:= -Wall -Werror -std=c++11 -shared -fPIC CFLAGS+= -I../../includes -I/usr/local/cuda-$(CUDA_VER)/include <b>-I/home/nvidia/TensorRT-5.1.3.6/include</b> LIBS:= -lnvinfer -lnvparsers -L/usr/local/cuda-$(CUDA_VER)/lib64 -lcudart -lcublas <b>-L/home/nvidia/TensorRT-5.1.3.6/lib</b> LFLAGS:= -Wl,--start…

---

What is actual the performance for the Mobilenetv2 TRT Model?
I got for FP32 roughly 30FPS. I thought it would be faster.
Can you confirm the speed roughly @AastaLLL for Mobilenetv2?

Update
If I change inside the “source_…” file the option “sync” under the part “sink0” from 1 to 0 I get 60 FPS.

Update 2

If you trained Mobilenetv2 with Tf1.15 you can’t transform it with the UFF Parser because TensorRT can’t handle BatchNorm/FusedBatchNormV3. FusedBatchNorm was implemented in tensorflow with TF1.15. Maybe using TF1.14 can solve this problem.