Little to no detection using TLT Faster-RCNN trained model on Deepstream-App

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
nvidia GPU
• DeepStream Version
5.0.1
• JetPack Version (valid for Jetson only)
• TensorRT Version
7.0.0
• NVIDIA GPU Driver Version (valid for GPU only)
460.39
• Issue Type( questions, new requirements, bugs)
question
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
follow the steps I describe
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Hello everyone.
I trained a Faster-RCNN nn using TLT’s example notebook (I added more epochs and made another minor changes, but the core and the dataset are just as the examples data)
I can train the netwonr, prune it, retrain it, test it, and deploy it to etlt with no problem.
The network works well when I do inference on the notebook, I can see good results even on new images from the internet
When I run it on deepstream-app, it’s another story. The app generates the .engine and runs successfully, but there are almost no detection on the sample videos.

I am using TLT and Deepstream on docker. Images are:
nvcr.io/nvidia/tlt-streamanalytics:v3.0-dp-py3
nvcr.io/nvidia/deepstream:5.0.1-20.09-devel

These are the files I use.
On the specs changes are done to the batches sizes and folder directories
default_spec_resnet18.txt (4.1 KB)
default_spec_resnet18_retrain_spec.txt (4.4 KB)

deepstream_app_config_fasterRCNN.txt (3.1 KB) config_infer_primary_frcnn.txt (2.2 KB)
frcnn_labels.txt (29 Bytes)

And these are my kitty outputs.

 00_000_000024.txt
::::::::::::::
Car 0.0 0 0.0 29.439560 420.078003 329.627258 918.770020 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.968405
::::::::::::::
00_000_000025.txt
::::::::::::::
Car 0.0 0 0.0 17.687958 412.115479 331.289398 926.380310 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.699041
::::::::::::::
00_000_000026.txt
::::::::::::::
Car 0.0 0 0.0 41.543549 440.329987 330.368134 907.080933 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.858691
::::::::::::::
00_000_000027.txt
::::::::::::::
::::::::::::::
00_000_000028.txt
::::::::::::::
::::::::::::::
00_000_000029.txt
::::::::::::::
::::::::::::::
00_000_000030.txt
::::::::::::::
::::::::::::::
00_000_000031.txt

I can’t figure out what could I be missing for not having much detection.
Is there a problem on my config files?
Is there a known issue with the parser?
Thank you!

3

Hi,

Since your model is Faster-RCNN, would you mind to follow below sample to deploy it with Deepstream again?
The repository has the configure file for Faster-RCNN, so you don’t need to modified one from resnet18.

Thanks.

4

Hello @AastaLLL, thank you for your answer.
I followed all the steps, but I got no improvement.
Detection keeps being almost the same with the plugin provided on that git.

5

Moving this topic from DS forum into TLT forum.

8

Please double check your config file. You can refer to https://github.com/NVIDIA-AI-IOT/deepstream_tlt_apps/blob/master/configs/frcnn_tlt/pgie_frcnn_tlt_config.txt

Especially below parameter is different from yours.

parse-bbox-func-name=NvDsInferParseCustomNMSTLT
custom-lib-path=…/…/post_processor/libnvds_infercustomparser_tlt.so

9

Hello @Morganh, thanks for your help.
@AastaLLL’s solution was to compile and use the parser from that very GitHub repository.
As I replied to him, it did not make a difference.
Here is the config infer of that second try.

[property]
gpu-id=0
net-scale-factor=1.0
offsets=103.939;116.779;123.68
model-color-format=1
labelfile-path=frcnn_labels.txt
tlt-encoded-model=../models/frcnn_kitti_resnet18_retrain.etlt
tlt-model-key=<can't share the key>
model-engine-file=../models/frcnn_kitti_resnet18_retrain.etlt_b1_gpu0_fp32.engine
infer-dims=3;544;960
uff-input-order=0
uff-input-blob-name=input_image
batch-size=1
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=0
num-detected-classes=4
interval=0
gie-unique-id=1
is-classifier=0
#network-type=0
output-blob-names=NMS
cluster-mode=2
parse-bbox-func-name=NvDsInferParseCustomNMSTLT
custom-lib-path=/home/user/dev/nvidia/proyectos/ejemplos/deepstream_tlt_apps/post_processor/libnvds_infercustomparser_tlt.so

[class-attrs-all]
pre-cluster-threshold=0.01
roi-top-offset=0
roi-bottom-offset=0
detected-min-w=0
detected-min-h=0
detected-max-w=0
detected-max-h=0
10

To narrow down, can you download the sample faster_rcnn model and check if it can work?
See below step in GitHub - NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream

2. Download Models

cd deepstream_tlt_apps/
wget https://nvidia.box.com/shared/static/i1cer4s3ox4v8svbfkuj5js8yqm3yazo.zip -O models.zip
unzip models.zip
rm models.zip
11

With the downloaded model detection improves but not that much.
It doesn’t detects any vehicle and sometimes will detect and track a person.
It is still less than desirable.

12

If you were using the sample faster_rcnn model, please make sure you were using https://github.com/NVIDIA-AI-IOT/deepstream_tlt_apps/blob/master/configs/frcnn_tlt/frcnn_labels.txt and https://github.com/NVIDIA-AI-IOT/deepstream_tlt_apps/blob/master/configs/frcnn_tlt/pgie_frcnn_tlt_config.txt

13

I am using those exact same files.
I’m running the command with

deepstream-app -c deepstream_app_config_fasterRCNN.txt

my deepstream_app_config_fasterRCNN.txt file is

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=1
gie-kitti-output-dir=./

[tiled-display]
enable=0
rows=1
columns=1
width=1280
height=720
gpu-id=0
nvbuf-memory-type=0

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=3
num-sources=1
uri=file:/home/user/dev/nvidia/samples/streams/sample_1080p_h264.mp4
gpu-id=0
cudadec-memtype=0

[streammux]
gpu-id=0
batch-size=1
batched-push-timeout=-1
## Set muxer output width and height
width=1920
height=1080
nvbuf-memory-type=0

[sink2]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=1
bitrate=3000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
# set below properties in case of RTSPStreaming  
rtsp-port=8555
udp-port=5400

[osd]
enable=1
gpu-id=0
border-width=3
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[primary-gie]
enable=1
gpu-id=0
batch-size=1
gie-unique-id=1
interval=1
config-file=config_infer_primary_frcnn_infer_0_Example.txt

nvbuf-memory-type=0

my config_infer_primary_frcnn_infer_0_Example.txt is the same as you pusted, except for paths.

[property]
gpu-id=0
net-scale-factor=1.0
offsets=103.939;116.779;123.68
model-color-format=1
labelfile-path=/home/user/dev/nvidia/proyectos/ejemplos/deepstream_tlt_apps/configs/frcnn_tlt/frcnn_labels.txt
tlt-encoded-model=/home/user/dev/nvidia/proyectos/ejemplos/deepstream_tlt_apps/models/frcnn/faster_rcnn_resnet18.etlt
tlt-model-key=nvidia_tlt
model-engine-file=/home/user/dev/nvidia/proyectos/ejemplos/deepstream_tlt_apps/models/frcnn/faster_rcnn_resnet18.etlt_b1_gpu0_fp16.engine

infer-dims=3;544;960
uff-input-order=0
uff-input-blob-name=input_image
batch-size=1
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=2
num-detected-classes=5
interval=0
gie-unique-id=1
is-classifier=0
#network-type=0
output-blob-names=NMS
cluster-mode=2
parse-bbox-func-name=NvDsInferParseCustomNMSTLT
custom-lib-path=/home/user/dev/nvidia/proyectos/ejemplos/deepstream_tlt_apps/post_processor/libnvds_infercustomparser_tlt.so

[class-attrs-all]
pre-cluster-threshold=0.6
roi-top-offset=0
roi-bottom-offset=0
detected-min-w=0
detected-min-h=0
detected-max-w=0
detected-max-h=0

and finnaly, frcnn_labels.txt is

Bicycle
Car
Person
Roadsign
background

I double checked every path with ls and more commands.
The sample video is one of the provided by deepstream.

14

So, according to you description, even with pretrained model provided by Nvidia, you still cannot get correct detection, right?

Please follow github GitHub - NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream to run , and run with below.

For detection model:
Usage: ds-tlt -c pgie_config_file -i <H264 or JPEG filename> [-b BATCH] [-d]

You need not deepstream_app_config_fasterRCNN.txt.

15

Hello @Morganh , @AastaLLL .
I found out what step I was missing
On motropolis’s docs, there is a final step on Tensor OSS’s install asking to replace the original libnvinfer_plugin.so with the generated at the end of Tensor OSS’s compilation.
After doing that I got the expected results.

sudo mv /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7.x.y ${HOME}/libnvinfer_plugin.so.7.x.y.bak   // backup original libnvinfer_plugin.so.x.y
sudo cp `pwd`/out/libnvinfer_plugin.so.7.m.n  /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.7.x.y
sudo ldconfig

Thank you again for all your help.
I got Faster RCNN and YOLO_V4 running propperly on my deepstream app.

1 Like
16

Thanks for the info. Great job!