Yolov3_tiny is used to track vehicle detection problems

hi:
I would like to ask two questions. One is that the Primary_Detector_Nano model in deepstream is poorly identified at night. Is this just a model use case? Do not provide accurate vehicle tracking, we need to develop it ourselves? There is another problem, we use the yolov3_tiny model to track and detect the vehicle, the source stream is 1080p, I am modifying the streammux wight=1280 height=720 in the configuration file deepstream_app_config_yolov3_tiny.txt, when using deepstream-app, Error:

root@uu-desktop:/opt/nvidia/deepstream/deepstream-4.0/sources/objectDetector_Yolo# deepstream-app -c deepstream_app_config_yoloV3_tiny.txt 
** WARN: <parse_gie:940>: Unknown key 'raw_output_directory' for group [primary-gie]

Using winsys: x11 
Opening in BLOCKING MODE 
Creating LL OSD context new
Creating LL OSD context new
0:00:00.671765941  9241     0x10ac3520 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:initialize(): Trying to create engine from model files
Loading pre-trained weights...
Loading complete!
Total Number of weights read : 8858734
      layer               inp_size            out_size       weightPtr
(1)   conv-bn-leaky     3 x 416 x 416      16 x 416 x 416    496   
(2)   maxpool          16 x 416 x 416      16 x 208 x 208    496   
(3)   conv-bn-leaky    16 x 208 x 208      32 x 208 x 208    5232  
(4)   maxpool          32 x 208 x 208      32 x 104 x 104    5232  
(5)   conv-bn-leaky    32 x 104 x 104      64 x 104 x 104    23920 
(6)   maxpool          64 x 104 x 104      64 x  52 x  52    23920 
(7)   conv-bn-leaky    64 x  52 x  52     128 x  52 x  52    98160 
(8)   maxpool         128 x  52 x  52     128 x  26 x  26    98160 
(9)   conv-bn-leaky   128 x  26 x  26     256 x  26 x  26    394096
(10)  maxpool         256 x  26 x  26     256 x  13 x  13    394096
(11)  conv-bn-leaky   256 x  13 x  13     512 x  13 x  13    1575792
(12)  maxpool         512 x  13 x  13     512 x  13 x  13    1575792
(13)  conv-bn-leaky   512 x  13 x  13    1024 x  13 x  13    6298480
(14)  conv-bn-leaky  1024 x  13 x  13     256 x  13 x  13    6561648
(15)  conv-bn-leaky   256 x  13 x  13     512 x  13 x  13    7743344
(16)  conv-linear     512 x  13 x  13     255 x  13 x  13    7874159
(17)  yolo            255 x  13 x  13     255 x  13 x  13    7874159
(18)  route                  -            256 x  13 x  13    7874159
(19)  conv-bn-leaky   256 x  13 x  13     128 x  13 x  13    7907439
(20)  upsample        128 x  13 x  13     128 x  26 x  26        - 
(21)  route                  -            384 x  26 x  26    7907439
(22)  conv-bn-leaky   384 x  26 x  26     256 x  26 x  26    8793199
(23)  conv-linear     256 x  26 x  26     255 x  26 x  26    8858734
(24)  yolo            255 x  26 x  26     255 x  26 x  26    8858734
Output blob names :
yolo_17
yolo_24
Total number of layers: 50
Total number of layers on DLA: 0
Building the TensorRT Engine...
Building complete!
0:00:32.850000647  9241     0x10ac3520 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:generateTRTModel(): Storing the serialized cuda engine to file at /opt/nvidia/deepstream/deepstream-4.0/sources/objectDetector_Yolo/model_b1_fp32.engine
Deserialize yoloLayerV3 plugin: yolo_17
Deserialize yoloLayerV3 plugin: yolo_24

Runtime commands:
	h: Print this help
	q: Quit

	p: Pause
	r: Resume


**PERF: FPS 0 (Avg)	FPS 1 (Avg)	
**PERF: 0.00 (0.00)	0.00 (0.00)	
** INFO: <bus_callback:163>: Pipeline ready

Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
NvMMLiteBlockCreate : Block : BlockType = 261 
** INFO: <bus_callback:149>: Pipeline running

Creating LL OSD context new
Creating LL OSD context new
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
H264: Profile = 66, Level = 0 
ERROR from sink_sub_bin_mux1: Could not multiplex stream.
Debug info: gstqtmux.c(4561): gst_qt_mux_add_buffer (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin1/GstQTMux:sink_sub_bin_mux1:
Buffer has no PTS.
Quitting
App run failed

配置文件:

################################################################################
# Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved.
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
# DEALINGS IN THE SOFTWARE.
################################################################################

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=0
rows=1
columns=2
width=1920
height=1080
gpu-id=1
#(0): nvbuf-mem-default - Default memory allocated, specific to particular platform
#(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory, applicable for Tesla
#(2): nvbuf-mem-cuda-device - Allocate Device cuda memory, applicable for Tesla
#(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory, applicable for Tesla
#(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
nvbuf-memory-type=0

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=3
uri=file://../../samples/streams/test.mp4
num-sources=1
gpu-id=1
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[source1]
enable=0
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=3
uri=file://../../samples/streams/test.mp4
num-sources=1
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[source2]
enable=0
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=3
uri=file://../../samples/streams/test.mp4
num-sources=1
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[source3]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=3
#uri=file://../../samples/streams/test.mp4
uri=file://20191010111000.mp4

num-sources=1
gpu-id=1
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File
type=3
container=1
codec=1
sync=0
source-id=0
gpu-id=0
qos=0
nvbuf-memory-type=0
overlay-id=1
output-file=out.mp4

#[sink0]
#enable=0
##Type - 1=FakeSink 2=EglSink 3=File
#type=3
#sync=0
#source-id=1
#gpu-id=1
#nvbuf-memory-type=0

[sink1]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File
type=2
sync=0
source-id=1
gpu-id=1
nvbuf-memory-type=0

[sink2]
enable=0
#Type - 1=FakeSink 2=EglSink 3=File
type=2
sync=0
source-id=2
gpu-id=0
nvbuf-memory-type=0

[sink3]
enable=0
#Type - 1=FakeSink 2=EglSink 3=File
type=2
sync=0
source-id=3
gpu-id=0
nvbuf-memory-type=0

[osd]
enable=1
gpu-id=0
border-width=1
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=0
batch-size=1

##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
## Set muxer output width and height
width=1280
height=720
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0

# config-file property is mandatory for any gie section.
# Other properties are optional and if set will override the properties set in
# the infer config file.
[primary-gie]
enable=1
gpu-id=0
#model-engine-file=model_b1_fp32.engine
labelfile-path=labels.txt
batch-size=1
#Required by the app for OSD, not a plugin property
#bbox-border-color0=1;0;0;1
#bbox-border-color1=0;1;1;1
#bbox-border-color2=0;0;1;1
#bbox-border-color3=0;1;0;1
gie-unique-id=1
nvbuf-memory-type=0
config-file=config_infer_primary_yoloV3_tiny.txt
raw_output_directory=1234
[tests]
file-loop=0

Hi,

1. To improve the detection quality at night, you can re-train the model with more low light database.
We also have a toolkit to help user refine the model:
https://developer.nvidia.com/transfer-learning-toolkit

2. I can run YOLO sample with 1280x720 streammux output.
Would you mind to give the following config a try?

################################################################################
# Copyright (c) 2019, NVIDIA CORPORATION. All rights reserved.
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
# DEALINGS IN THE SOFTWARE.
################################################################################

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=1
rows=1
columns=1
width=1280
height=720
gpu-id=0
#(0): nvbuf-mem-default - Default memory allocated, specific to particular platform
#(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory, applicable for Tesla
#(2): nvbuf-mem-cuda-device - Allocate Device cuda memory, applicable for Tesla
#(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory, applicable for Tesla
#(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
nvbuf-memory-type=0

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=3
uri=file://../../samples/streams/sample_1080p_h264.mp4
num-sources=1
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File
type=2
sync=0
source-id=0
gpu-id=0
nvbuf-memory-type=0

[osd]
enable=1
gpu-id=0
border-width=1
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=0
batch-size=1
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
## Set muxer output width and height
width=1280
height=720
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0

# config-file property is mandatory for any gie section.
# Other properties are optional and if set will override the properties set in
# the infer config file.
[primary-gie]
enable=1
gpu-id=0
model-engine-file=model_b1_fp32.engine
labelfile-path=labels.txt
batch-size=1
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
gie-unique-id=1
interval=2
nvbuf-memory-type=0
config-file=config_infer_primary_yoloV3_tiny.txt

[tests]
file-loop=0

Thanks.

hi AastaLLL:
Thank you for your reply. I want to draw the detected area in the video stream for vehicle detection. Is there a corresponding example to provide it, or provide a solution? Thank you.

Hi,

Deepstream only draw the bounding box boundary.
Are you looking for rendering approach with overlay like this:
https://github.com/dusty-nv/jetson-inference/blob/master/docs/detectnet-console-2.md

Thanks.

hi AastaLLL:
Hello, thank you for your reply. We now have the darknet algorithm. The caffee algorithm provided in deepstream is not standard. How to convert the caffee algorithm into an algorithm that can be applied in deepstream after trt acceleration.

Hi,

In Deepstream, we have a sample to convert darknet model into TensorRT:
/opt/nvidia/deepstream/deepstream-4.0/sources/objectDetector_Yolo

You can start from this sample directly.
Thanks.

hi AastaLLL:
I have seen this Darknet model. Is there a similar Darknet model for license plate detection or body color detection or vehicle type detection? I am researching using yolo interface to implement on test2.

Hi,

We don’t have pre-trained darknet model of your requirement.
Our darknet example use the default YOLOv2(tiny), YOLOv3(tiny) released by the author.

For the license plate detection, you can give OpenALPR a try.
https://github.com/openalpr/deepstream_jetson

Although the GitHub is a little bit out-of-date(DeepStream1.5), you can still start from it.

We don’t have color detector and type detector.
But there are some color/type classifier defined in the Caffe format inside the Deepstream.
It’s used with the a car detector beforehand.
/opt/nvidia/deepstream/deepstream-4.0/samples/models/

Thanks