DeepSORT tracker not working on multiple streams (erases detections in some frames)

DeepSORT tracker fails to track objects in case of multiple input streams. Further investigation revealed that it erases object metadata of detected objects in roughly half of the processed frames, which means that for 2 streams there appear to be detections only in the first one (detections from the second one are apparently erased, so not visible in the output). If you reproduce by the commands below, you will get two output videos where the first looks ok, but second one has no detections.

By even further investigation we found that this problem is temporarily solved by increasing batch size in streammux. However if input streams differ highly in fps (like 3fps vs 30fps) then we get the faulty behaviour again.

Hardware Platform: GPU
DeepStream Version: 6.1.1
TensorRT Version: 8.4.1
NVIDIA GPU Driver Version: 515.65.01
Issue Type: Bug
How to reproduce the issue in docker image nvcr.io/nvidia/deepstream:6.1.1-devel

cd /opt/nvidia/deepstream/deepstream/bin
deepstream-app -c /opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app/source2_1080p_dec_infer-resnet_demux_int8.txt

Content of /opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app/source2_1080p_dec_infer-resnet_demux_int8.txt:

################################################################################
# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
# DEALINGS IN THE SOFTWARE.
################################################################################

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
uri=file://../../streams/sample_1080p_h264.mp4
num-sources=1
#drop-frame-interval=2
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[source1]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=3
uri=file://../../streams/sample_1080p_h264.mp4
num-sources=1
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[sink0]
#source0 output as filesink
enable=1
type=3
#1=mp4 2=mkv
container=1
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=1
#iframeinterval=10
bitrate=2000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
output-file=out_source0.mp4
source-id=0

[sink1]
#source1 output as filesink
enable=1
type=3
#1=mp4 2=mkv
container=1
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=1
#iframeinterval=10
bitrate=2000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
output-file=out_source1.mp4
source-id=1

#[sink0]
#enable=0
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
#type=4
#1=h264 2=h265
#codec=1
#encoder type 0=Hardware 1=Software
#enc-type=0
#sync=1
#bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
#profile=0
# set below properties in case of RTSPStreaming
#rtsp-port=8554
#udp-port=5400
#source-id=0

[sink1]
#enable=0
#Type - 1=FakeSink 2=EglSink 3=File
#type=2
#sync=1
#source-id=1
#gpu-id=0
#nvbuf-memory-type=0

[osd]
enable=1
gpu-id=0
border-width=1
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=0
batch-size=1
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=33000
## Set muxer output width and height
width=1920
height=1080
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0
## If set to TRUE, system timestamp will be attached as ntp timestamp
## If set to FALSE, ntp timestamp from rtspsrc, if available, will be attached
# attach-sys-ts-as-ntp=1

# config-file property is mandatory for any gie section.
# Other properties are optional and if set will override the properties set in
# the infer config file.
[primary-gie]
enable=1
gpu-id=0
model-engine-file=../../models/Primary_Detector/resnet10.caffemodel_b1gpu0_int8.engine
batch-size=1
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
interval=0
gie-unique-id=1
nvbuf-memory-type=0
config-file=config_infer_primary.txt

[tracker]
enable=1
# For NvDCF and DeepSORT tracker, tracker-width and tracker-height must be a multiple of 32, respectively
tracker-width=640
tracker-height=384
ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
# ll-config-file required to set different tracker types
# ll-config-file=config_tracker_IOU.yml
# ll-config-file=config_tracker_NvDCF_perf.yml
# ll-config-file=config_tracker_NvDCF_accuracy.yml
ll-config-file=config_tracker_DeepSORT.yml
gpu-id=0
enable-batch-process=1
enable-past-frame=1
display-tracking-id=1

[tests]
file-loop=0
1 Like

Can you reproduce the issue with below configure? Only change the tracker config to DeepSORT.yml
source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.yml

Yes.
If I change batch-size in streammux from 4 to 1, I get similar faulty behaviour. I expect it would be the same with batch-size=4 and differing fps in input streams, as described above.

Content of the config is precisely this:

################################################################################
# Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
# DEALINGS IN THE SOFTWARE.
################################################################################

application:
  enable-perf-measurement: 1
  perf-measurement-interval-sec: 5
  ##gie-kitti-output-dir=streamscl

tiled-display:
  enable: 1
  rows: 2
  columns: 2
  width: 1280
  height: 720
  gpu-id: 0
  #(0): nvbuf-mem-default - Default memory allocated, specific to particular platform
  #(1): nvbuf-mem-cuda-pinned - Allocate Pinned/Host cuda memory, applicable for Tesla
  #(2): nvbuf-mem-cuda-device - Allocate Device cuda memory, applicable for Tesla
  #(3): nvbuf-mem-cuda-unified - Allocate Unified cuda memory, applicable for Tesla
  #(4): nvbuf-mem-surface-array - Allocate Surface Array memory, applicable for Jetson
  nvbuf-memory-type: 0

source:
  csv-file-path: sources_4.csv

sink0:
  enable: 0
  #Type - 1=FakeSink 2=EglSink 3=File 7=nv3dsink (Jetson only)
  type: 2
  sync: 1
  source-id: 0
  gpu-id: 0
  nvbuf-memory-type: 0

sink1:
  enable: 1
  type: 3
  #1=mp4 2=mkv
  container: 1
  #1=h264 2=h265
  codec: 1
  #encoder type 0=Hardware 1=Software
  enc-type: 0
  sync: 0
  #iframeinterval=10
  bitrate: 2000000
  #H264 Profile - 0=Baseline 2=Main 4=High
  #H265 Profile - 0=Main 1=Main10
  profile: 0
  output-file: out.mp4
  source-id: 0

sink2:
  enable: 0
  #Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming
  type: 4
  #1=h264 2=h265
  codec: 1
  #encoder type 0=Hardware 1=Software
  enc-type: 0
  sync: 0
  bitrate: 4000000
  #H264 Profile - 0=Baseline 2=Main 4=High
  #H265 Profile - 0=Main 1=Main10
  profile: 0
  # set below properties in case of RTSPStreaming
  rtsp-port: 8554
  udp-port: 5400

osd:
  enable: 1
  gpu-id: 0
  border-width: 1
  text-size: 15
  #value changed
  text-color: 1;1;1;1
  text-bg-color: 0.3;0.3;0.3;1
  font: Serif
  show-clock: 0
  clock-x-offset: 800
  clock-y-offset: 820
  clock-text-size: 12
  clock-color: 1;0;0;0
  nvbuf-memory-type: 0

streammux:
  gpu-id: 0
  ##Boolean property to inform muxer that sources are live
  live-source: 0
  buffer-pool-size: 4
  batch-size: 1
  ##time out in usec, to wait after the first buffer is available
  ##to push the batch even if the complete batch is not formed
  batched-push-timeout: 40000
  ## Set muxer output width and height
  width: 1920
  height: 1080
  ##Enable to maintain aspect ratio wrt source, and allow black borders, works
  ##along with width, height properties
  enable-padding: 0
  nvbuf-memory-type: 0

primary-gie:
  enable: 1
  gpu-id: 0
  model-engine-file: ../../models/Primary_Detector/resnet10.caffemodel_b4_gpu0_int8.engine
  batch-size: 4
  #Required by the app for OSD, not a plugin property
  bbox-border-color0: 1;0;0;1
  bbox-border-color1: 0;1;1;1
  bbox-border-color2: 0;0;1;1
  bbox-border-color3: 0;1;0;1
  interval: 0
  gie-unique-id: 1
  nvbuf-memory-type: 0
  config-file: config_infer_primary.yml

tracker:
  enable: 1
  # For NvDCF and DeepSORT tracker, tracker-width and tracker-height must be a multiple of 32, respectively
  tracker-width: 640
  tracker-height: 384
  ll-lib-file: /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
  # ll-config-file required to set different tracker types
  # ll-config-file=config_tracker_IOU.yml
  # ll-config-file: config_tracker_NvDCF_perf.yml
  # ll-config-file=config_tracker_NvDCF_accuracy.yml
  ll-config-file: config_tracker_DeepSORT.yml
  gpu-id: 0
  enable-batch-process: 1
  enable-past-frame: 1
  display-tracking-id: 1

secondary-gie0:
  enable: 1
  model-engine-file: ../../models/Secondary_VehicleTypes/resnet18.caffemodel_b16_gpu0_int8.engine
  gpu-id: 0
  batch-size: 16
  gie-unique-id: 4
  operate-on-gie-id: 1
  operate-on-class-ids: 0
  config-file: config_infer_secondary_vehicletypes.txt

secondary-gie1:
  enable: 1
  model-engine-file: ../../models/Secondary_CarColor/resnet18.caffemodel_b16_gpu0_int8.engine
  batch-size: 16
  gpu-id: 0
  gie-unique-id: 5
  operate-on-gie-id: 1
  operate-on-class-ids: 0
  config-file: config_infer_secondary_carcolor.txt

secondary-gie2:
  enable: 1
  model-engine-file: ../../models/Secondary_CarMake/resnet18.caffemodel_b16_gpu0_int8.engine
  batch-size: 16
  gpu-id: 0
  gie-unique-id: 6
  operate-on-gie-id: 1
  operate-on-class-ids: 0
  config-file: config_infer_secondary_carmake.txt


tests:
  file-loop: 0

Do you see the same behavior when nvDCF and IOU tracker? Or only DeepSORT have this kind of behavior?

I tested both nvDCF and IOU by using config_tracker_NvDCF_perf.yml or config_tracker_IOU.yml config. None displayed the issue we observe with DeepSORT.

1 Like