Error on tags while running deepstream-app with a classifier as primary-gie

Hello everyone.
I just made a config file for deepstream-app to run with a classifier as primary (and only) gie.
The problem is that everything is listed as background

Any help on this will be apreciated!

=================== Things I tried ======================

  • Using a rtsp camera and capturing myself, a bottle, among others
  • Using a rtsp camera and capturing an internet photo of a plane shown on my cell
  • Using a rtsp camera and capturing an image on validation dataset shown on my computer
  • Using a rtsp camera and capturing an image on test dataset shown on my computer
  • Giving this video as input instead of a rtsp camera: https://www.youtube.com/watch?v=G0usamnRrMM

================== Some Outputs ========================

=================== About the model =======================
I trained the model using the classification example on tlt docker. I ran the full 80 epochs, pruned, and retrained again 80 epochs.
I’m exporting on FP32 and deepstream-app is generating the engine file.
I dont know where the engine file is generated, it is not on the folder I ask it to be.

Here it is the confusion matrix of the model and the classification report.

              precision    recall  f1-score   support

   aeroplane       0.74      0.74      0.74       134
     bicycle       0.36      0.57      0.44       111
        bird       0.74      0.56      0.64       153
        boat       0.52      0.76      0.62       102
      bottle       0.20      0.19      0.19       142
         bus       0.47      0.49      0.48        85
         car       0.53      0.45      0.48       233
         cat       0.67      0.64      0.66       216
       chair       0.20      0.17      0.18       224
         cow       0.62      0.48      0.54        61
 diningtable       0.14      0.18      0.16       108
         dog       0.57      0.46      0.51       258
       horse       0.45      0.40      0.43        97
   motorbike       0.42      0.36      0.39       106
      person       0.37      0.35      0.36       818
 pottedplant       0.22      0.23      0.22       106
       sheep       0.47      0.71      0.57        65
        sofa       0.18      0.20      0.19       102
       train       0.58      0.85      0.69       109
   tvmonitor       0.32      0.37      0.34       115

   micro avg       0.43      0.43      0.43      3345
   macro avg       0.44      0.46      0.44      3345
weighted avg       0.43      0.43      0.43      3345

=============== Labels ================
labels.txt looks like this:

aeroplane
bicycle 
bird
boat
bottle
bus 
car
cat
chair
cow 
diningtable
dog
horse
motorbike       
person
pottedplant  
sheep     
sofa  
train    
tvmonitor 

background is not even included here.

=============== Config Files ================

Deepstream-app Config File

# PR. GIE CON RED DE CLASIFICACION EN VEZ DE DETECCION
# SOURCES -> MUX -> OSD -> PR. GIE -> TRCKER -> SINKS -> TILED

# Copyright (c) 2019-2020 NVIDIA Corporation.  All rights reserved.
#
# NVIDIA Corporation and its licensors retain all intellectual property
# and proprietary rights in and to this software, related documentation
# and any modifications thereto.  Any use, reproduction, disclosure or
# distribution of this software and related documentation without an express
# license agreement from NVIDIA Corporation is strictly prohibited.

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=1
rows=1
columns=1
width=1240
height=980
gpu-id=0
#(0): nvbuf-mem-default - Default memory allocated, specific to particular platform
#(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory, applicable for Tesla
#(2): nvbuf-mem-cuda-device - Allocate Device cuda memory, applicable for Tesla
#(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory, applicable for Tesla
#(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
nvbuf-memory-type=0

[source0]
enable=0
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=4
uri=rtsp://<My Camera>
#drop-frame-interval=2
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[source1]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP 
type=3
uri=file://../streams/planes.mp4
#uri=file://../streams/sample_1080p_h265.mp4
#uri=file://../streams/sample_720p.mp4
num-sources=1
#drop-frame-interval=4
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[sink0]
enable=0
#Type - 1=FakeSink 2=EglSink 3=File
type=5
sync=1
source-id=0
gpu-id=0
qos=0
nvbuf-memory-type=0
overlay-id=1

[sink1]
enable=0
type=3
#1=mp4 2=mkv
container=1
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=0
#iframeinterval=10
bitrate=2000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
output-file=out.mp4
source-id=0

[sink2]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
type=4
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=1
bitrate=3000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
# set below properties in case of RTSPStreaming  
rtsp-port=8555
udp-port=5401

[osd]
enable=1
gpu-id=0
border-width=5
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=1
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=0
batch-size=8
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
## Set muxer output width and height
width=1920
height=1080
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0
## If set to TRUE, system timestamp will be attached as ntp timestamp
## If set to FALSE, ntp timestamp from rtspsrc, if available, will be attached
# attach-sys-ts-as-ntp=1
# config-file property is mandatory for any gie section.
# Other properties are optional and if set will override the properties set in
# the infer config file.

[primary-gie]
enable=1
gpu-id=0
model-engine-file=../models/Controlflow_tlt_classification_full/final_model_full_labels.etlt.engine
batch-size=1
#Required by the app for OSD, not a plugin property
## 0=FP32, 1=INT8, 2=FP16 mode
interval=0
gie-unique-id=1
nvbuf-memory-type=0
config-file=config_infer_secondary_controlflow_as_primary.txt

[tests]
file-loop=0

Model’s Config File

# Copyright (c) 2020 NVIDIA Corporation.  All rights reserved.
#
# NVIDIA Corporation and its licensors retain all intellectual property
# and proprietary rights in and to this software, related documentation
# and any modifications thereto.  Any use, reproduction, disclosure or
# distribution of this software and related documentation without an express
# license agreement from NVIDIA Corporation is strictly prohibited.

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
tlt-model-key=<My Key>
tlt-encoded-model=../models/Controlflow_tlt_classification_full/final_model_full_labels.etlt
labelfile-path=../models/Controlflow_tlt_classification/labels.txt
#int8-calib-file=../models/Controlflow_tlt/dashcamnet_int8.txt
model-engine-file=../models/Controlflow_tlt_classification_full/final_model_full_labels.etlt.engine
#input-dims=3;544;960;0
input-dims=3;224;224;0
uff-input-blob-name=input_1
batch-size= 1 #8 #3
process-mode=1
model-color-format=0
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=0
network-type=1
#num-detected-classes=7
interval=2
gie-unique-id=1
output-blob-names=predictions/Softmax
classifier-threshold=0.2

Thanks!

Move to TLT forum because it is related to TLT.

Could you modify part of your config as below and retry? Reference: Integrating TAO Models into DeepStream — TAO Toolkit 3.22.05 documentation

net-scale-factor=1.0
offsets=123.67;116.28;103.53
model-color-format=1

Hello there. Sorry for the delay, I think we might live on opposite sides of the globe.
I did what you said, and something changed, but every plane is still labeled as background.

What’s new now, is that sometimes the background tag disappears.
When running this in the camera I get no label at all.
When running the net on classification_test_video.mp4 video saple I don’t get any output neither.

I’ve checked the documentation, tlt and pugin manual > nvinfer, and I would like to ask something.
How do you get to those values on net scale factor and offsets?

Thank you!

I dont know where the engine file is generated, it is not on the folder I ask it to be.

engine file is generated in local file, which name is like: model_b1_gpu0_fp32.engine

model-engine-file=…/models/Controlflow_tlt_classification_full/final_model_full_labels.etlt.engine

this is actually used to tell DeepStream where to load the existing TRT engine instead of where to save the TRT engine.

labelfile-path=…/models/Controlflow_tlt_classification/labels.txt

Is the content of this labet file what you pasted above?

Thanks!

Hello, thanks for your answer!

this is actually used to tell DeepStream where to load the existing TRT engine instead of where to save the TRT engine.

This is clarifying, thank you. Is there a way to tell deepstream-app where to create the model engine file (I’ll be looking myself for this too)?

Is the content of this label file what you pasted above?

Yep, that is a literal copy-paste of my labels.txt file. I writed it myself based on the following.
Integrating TAO Models into DeepStream — TAO Toolkit 3.22.05 documentation

And choosed the values based on this output

Something that calls my attention is that if values of labels.txt were wrongly sorted, I should check different wrong tags on every object, but I get the same “background” tag on everything.

I just saw a typo on the path, I was referencing another folder.
Thanks for pointing that out!

so, is the issue gone?

yes, I thought I marked your answer as solution.
I did it now, thank you!

Actually I re-opened the discussion since a new test failed.
(Was that some kind of jerk move to you? It is not intended if so, please tell me if I should make a new question instead)

I connected back the camera and no tag was displayed, but then I took a screenshoot of the sink’s output and gave it to the net on the tlt docker’s jupyter, where I got labeled as “person”.


Screenshot from 2020-06-19 13-28-32

I tryed changing model-color-format back to 0 with no luck.
the camera gives me a video on rtsp with this video settings.

You know what could it be?
If source is a .mp4 stream there is no trouble.

Again, thank you a lot, and sorry if gave u any trouble.

Here I made another test. I configured a sink to save the output on a mp4 file, and then gave that mp4 as input.
No detections were made nor on the camera stream, nor on the saved mp4 file.
Detections are made, however, on the aeroplanes viedo.

@ai12
Could you please clarify more for your latest result?
You mentioned that

If source is a .mp4 stream there is no trouble.

Does it mean for this case, you can run deepstream-app successfully and the classifier tag can be shown?