I am currently working on a project where cars and then their licence plates are detected. My previous back-to-back detection worked fine on Jetson nano with the deepstream sdk 4.0. But so far it failed to run on deepstream 5.0. However, the cars detection and licence plates detection are working without problems on deepstream 5.0. when standalone (see the figure 1 for the licence plate detection).
There were two solutions tested as back-to-back detection. The first one was based on the deepstream-test2 example deepstream 5.0, where the sgie was a detector instead of a classifier . The bounding boxes are presented on figure 2. The second solution was implemented with deepstream-test5 example of deepstream 5.0. The result is in figure 3.
These results were very similar. Licence plates tended to be on the left side of the cropped frames. It seemed that cropped frames after the first detector somehow were directed to the second detector rotated 90 degrees. Мaybe they were mixed up somewhere with the width and the height of the cropped frame inside of deepstream 5.0.
Fig. 1 Licence plate detection with deepstream 5.0:
Fig. 2 Back to back detection based on the deepstream-test2 example deepstream 5.0![|382x259]
Fig. 3 Back to back detection based on the deepstream-test5 example deepstream 5.0
The following are the config files of the back to back detection based on the deepstream-test5 example deepstream 5.0
test5_config_file_src_infer
[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
[tiled-display]
enable=1
rows=1
columns=1
width=960
height=540
gpu-id=0
nvbuf-memory-type=0
[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=3
uri=file:///media/microsd/streams/traffic2.mp4
num-sources=1
gpu-id=0
nvbuf-memory-type=0
[source1]
enable=0
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=3
uri=file:///media/microsd/streams/traffic2.mp4
num-sources=2
gpu-id=0
nvbuf-memory-type=0
[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File
type=2
sync=1
source-id=0
gpu-id=0
nvbuf-memory-type=0
[sink1]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=UDPSink 5=nvoverlaysink 6=MsgConvBroker
type=6
msg-conv-config=dstest5_msgconv_sample_config.txt
msg-conv-payload-type=0
msg-broker-proto-lib=/opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_amqp_proto.so
#Optional:
msg-broker-config=/home/dlinano/mvp_its/deepstream5/deepstream-test5/configs/cfg_amqp.txt
[osd]
enable=1
gpu-id=0
display-text=0
border-width=1
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Arial
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0
[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=0
batch-size=1
batched-push-timeout=40000
## Set muxer output width and height
width=1920
height=1080
enable-padding=0
nvbuf-memory-type=0
[primary-gie]
enable=1
gpu-id=0
batch-size=1
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode = 2
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;1;1;1
bbox-border-color3=0;1;0;1
nvbuf-memory-type=0
interval=0
gie-unique-id=1
config-file=/home/dlinano/mvp_its/deepstream5/deepstream-test5/configs/config_primary_detector.txt
[tracker]
enable=0
tracker-width=640
tracker-height=352
ll-lib-file=/opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_mot_klt.so
#enable-batch-process applicable to DCF only
enable-batch-process=0
[secondary-gie0]
enable=1
gpu-id=0
network-type = 0
batch-size=16
gie-unique-id=2
operate-on-gie-id=1
#operate-on-class-ids=0;
config-file=/home/dlinano/mvp_its/deepstream5/deepstream-test5/configs/config_secondary_detector.txt
[tests]
file-loop=0
config_primary_detector
[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
model-engine-file=/home/dlinano/mvp_its/models/cars_4types_p2.trt
labelfile-path=/home/dlinano/mvp_its/models/car_labels.txt
force-implicit-batch-dim=1
batch-size=1
process-mode=1
model-color-format=0
network-mode=2
num-detected-classes=4
interval=0
gie-unique-id=1
output-blob-names=output_cov/Sigmoid;output_bbox/BiasAdd
## 0=Group Rectangles, 1=DBSCAN, 2=NMS, 3= DBSCAN+NMS Hybrid, 4 = None(No clustering)
cluster-mode=1
## Per class configuration
[class-attrs-0]
pre-cluster-threshold=0.1
eps=0.6
minBoxes=3
detected-min-w=150
detected-min-h=150
roi-top-offset=0
roi-bottom-offset=10
[class-attrs-1]
pre-cluster-threshold=0.3
eps=0.6
minBoxes=3
detected-min-w=250
detected-min-h=250
roi-top-offset=0
roi-bottom-offset=10
[class-attrs-2]
pre-cluster-threshold=0.25
eps=0.6
minBoxes=3
detected-min-w=250
detected-min-h=250
roi-top-offset=0
roi-bottom-offset=10
[class-attrs-3]
pre-cluster-threshold=0.15
eps=0.6
minBoxes=3
detected-min-w=250
detected-min-h=250
roi-top-offset=0
roi-bottom-offset=10
config_secondary_detector
[property]
gpu-id=0
batch-size=16
gie-unique-id=2
operate-on-gie-id=1
net-scale-factor=1
model-engine-file=/home/dlinano/mvp_its/models/plate.trt
labelfile-path=/home/dlinano/mvp_its/models/plate_labels.txt
process-mode=2
#model-color-format=0
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=2
num-detected-classes=1
interval=0
output-blob-names=output_cov/Sigmoid;output_bbox/BiasAdd
force-implicit-batch-dim=1
## 0=Group Rectangles, 1=DBSCAN, 2=NMS, 3= DBSCAN+NMS Hybrid, 4 = None(No clustering)
cluster-mode=1
network-type = 0
[class-attrs-all]
pre-cluster-threshold=0.03
group-threshold=1
eps=0.5
minBoxes=3
detected-min-w=0
detected-min-h=0
#detected-max-w=0
#detected-max-h=0
Please share your thoughts on the issue.
It would be nice to get some replies on the following questions.
- Is back-to-back detection supported in deepstream 5.0 (i.e. both pgie and sgie are detectors)?
- If it is not supported when are you expecting to implement back-to-back detection in deepstream 5 (taking into account a lot of use cases based on this feature)?
- If it is supported can the deepstream-test5 example be used for back-to-back detection, or I need to build it from multiple instances of “nvinfer” element like in deepstream-test2?
- In case deepstream-test5 example supports the back-to-back detection, where am I wrong in my config files above?
Thank you in advance
