How to smooth the bbox detections with the DCF tracking in DS5.0 GA?

I’m using the DCF tracker with the sample config file provided here: /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/tracker_config.yml.

With this file the bboxes are very flickery and its not nice to watch. How do we adjust the settings to have a smooth movement of the bboxes - which is actually the main intention of using a tracker?

Copying in the config file here for reference:

%YAML:1.0
################################################################################
# Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
# DEALINGS IN THE SOFTWARE.
################################################################################

NvDCF:
  # [General]
  useUniqueID: 1    # Use 64-bit long Unique ID when assignining tracker ID. Default is [true]
  maxTargetsPerStream: 20 # Max number of targets to track per stream. Recommended to set >10. Note: this value should account for the targets being tracked in shadow mode as well. Max value depends on the GPU memory capacity

  # [Feature Extraction]
  useColorNames: 1     # Use ColorNames feature
  useHog: 0            # Use Histogram-of-Oriented-Gradient (HOG) feature
  useHighPrecisionFeature: 0   # Use high-precision in feature extraction. Default is [true]

  # [DCF]
  filterLr: 0.15 # learning rate for DCF filter in exponential moving average. Valid Range: [0.0, 1.0]
  filterChannelWeightsLr: 0.22 # learning rate for the channel weights among feature channels. Valid Range: [0.0, 1.0]
  gaussianSigma: 0.75 # Standard deviation for Gaussian for desired response when creating DCF filter [pixels]
  featureImgSizeLevel: 3 # Size of a feature image. Valid range: {1, 2, 3, 4, 5}, from the smallest to the largest
  SearchRegionPaddingScale: 1 # Search region size. Determines how large the search region should be scaled from the target bbox.  Valid range: {1, 2, 3}, from the smallest to the largest

  # [MOT] [False Alarm Handling]
  maxShadowTrackingAge: 30  # Max length of shadow tracking (the shadow tracking age is incremented when (1) there's detector input yet no match or (2) tracker confidence is lower than minTrackerConfidence). Once reached, the tracker will be terminated.
  probationAge: 3           # Once the tracker age (incremented at every frame) reaches this, the tracker is considered to be valid
  earlyTerminationAge: 1    # Early termination age (in terms of shadow tracking age) during the probation period. If reached during the probation period, the tracker will be terminated prematurely.

  # [Tracker Creation Policy] [Target Candidacy]
  minDetectorConfidence: -1  # If the confidence of a detector bbox is lower than this, then it won't be considered for tracking
  minTrackerConfidence: 0.7  # If the confidence of an object tracker is lower than this on the fly, then it will be tracked in shadow mode. Valid Range: [0.0, 1.0]
  minTargetBboxSize: 10      # If the width or height of the bbox size gets smaller than this threshold, the target will be terminated.
  minDetectorBboxVisibilityTobeTracked: 0.0  # If the detector-provided bbox's visibility (i.e., IOU with image) is lower than this, it won't be considered.
  minVisibiilty4Tracking: 0.0  # If the visibility of the tracked object (i.e., IOU with image) is lower than this, it will be terminated immediately, assuming it is going out of scene.

  # [Tracker Termination Policy]
  targetDuplicateRunInterval: 5 # The interval in which the duplicate target detection removal is carried out. A Negative value indicates indefinite interval. Unit: [frames]
  minIou4TargetDuplicate: 0.9 # If the IOU of two target bboxes are higher than this, the newer target tracker will be terminated.

  # [Data Association] Matching method
  useGlobalMatching: 0   # If true, enable a global matching algorithm (i.e., Hungarian method). Otherwise, a greedy algorithm wll be used.

  # [Data Association] Thresholds in matching scores to be considered as a valid candidate for matching
  minMatchingScore4Overall: 0.0   # Min total score
  minMatchingScore4SizeSimilarity: 0.5    # Min bbox size similarity score
  minMatchingScore4Iou: 0.1       # Min IOU score
  minMatchingScore4VisualSimilarity: 0.2    # Min visual similarity score
  minTrackingConfidenceDuringInactive: 1.0  # Min tracking confidence during INACTIVE period. If tracking confidence is higher than this, then tracker will still output results until next detection 

  # [Data Association] Weights for each matching score term
  matchingScoreWeight4VisualSimilarity: 0.8  # Weight for the visual similarity (in terms of correlation response ratio)
  matchingScoreWeight4SizeSimilarity: 0.0    # Weight for the Size-similarity score
  matchingScoreWeight4Iou: 0.1               # Weight for the IOU score
  matchingScoreWeight4Age: 0.1               # Weight for the tracker age

  # [State Estimator]
  useTrackSmoothing: 1    # Use a state estimator
  stateEstimatorType: 1   # The type of state estimator among { moving_avg:1, kalman_filter:2 }

  # [State Estimator] [MovingAvgEstimator]
  trackExponentialSmoothingLr_loc: 0.5       # Learning rate for new location
  trackExponentialSmoothingLr_scale: 0.3     # Learning rate for new scale
  trackExponentialSmoothingLr_velocity: 0.05  # Learning rate for new velocity

  # [State Estimator] [Kalman Filter]
  kfProcessNoiseVar4Loc: 0.1   # Process noise variance for location in Kalman filter
  kfProcessNoiseVar4Scale: 0.04   # Process noise variance for scale in Kalman filter
  kfProcessNoiseVar4Vel: 0.04   # Process noise variance for velocity in Kalman filter
  kfMeasurementNoiseVar4Trk: 9   # Measurement noise variance for tracker's detection in Kalman filter
  kfMeasurementNoiseVar4Det: 9   # Measurement noise variance for detector's detection in Kalman filter

  # [Past-frame Data]
  useBufferedOutput: 0   # Enable storing of past-frame data in a buffer and report it back

  # [Instance-awareness]
  useInstanceAwareness: 0 # Use instance-awareness for multi-object tracking
  lambda_ia: 2            # Regularlization factor for each instance
  maxInstanceNum_ia: 4    # The number of nearby object instances to use for instance-awareness


Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) Nano and Xavier NX
• DeepStream Version 5.0 GA

Hey Jason,
Would you mind to disable the Tracker or use KLT tracker to see if the bbox flickery?
If the issue only persist when use DCF, pls share your setup with me, I will try to tune it locally.

Its not flickery with KLT. However its not as good a tracker so KLT loses the bbox A LOT.

DCF is a very good tracker but its flickering continuously which suggests to me the config is not right or there is a bug…?

There are just too many parameters for me to try and work it out. I could spend an entire year adjusting all those DCF parameters and still no nothing. ;-)

Can you check the config and maybe see what the cause is? OR suggest improvements?

I have my nvinfer interval set to 1. But this should not matter. It should track well with interval set to 6 or more even.

Could you improve the featureImgSizeLevel = 5 and give a try?
Also you may need to set the nvinfer interval > 1.

Will be better if you can share your setup with me since we don’t observe the issue in our internal SQA side.

I have increased the pgie interval to 4.

I notice that not only are the bboxes still flickering all the time but I also sometimes get zombie detections after a person has walked out of the camera scene. i.e. a bbox that just hangs around for a long time after the person is long gone. This zombie bbox slowly gets bigger and bigger till it its the size of the whole frame if I just leave it running!

Increasing featureImgSizeLevel = 5 from 3 seemed to make no difference.

It looks to me as if the bbox is only drawn when the pgie makes a detection and its missing when its the DCF tracker generating the bbox.??

ok I have found this new DS5.0GA parameter: minTrackingConfidenceDuringInactive. It is set to 1.0 in the sample config file provided with deepstream.

See this blog post where they are talking about zombie/lingering detections. They suggest increasing minTrackingConfidenceDuringInactive to 99.

However if I actually DECREASE this value to 0.5 then the flickering bboxes goes away and they move nice and smooth - like they should for a tracker.
But this is going to mean I get those lingering zombie bboxes right?

So it seems that when minTrackingConfidenceDuringInactive is 1.0 or greater you will only see bboxes drawn when the pgie makes a detection - which is why they flicker. It would probably be better if you set pgie interval to 0 to help reduce the flicker.
So if pgie interval is > 1 then you need to reduce the minTrackingConfidenceDuringInactive value to be less than 1.0. (I have just tested with 0.5) to stop the flickering.

I will play around with these minTrackingConfidenceDuringInactive values some more to find a suitable value. But that leave the open question of how to stop these lingering zombie bboxes…

Please look at these sample mp4 files to see the lingering/zombie bbox issue. Its really bad and renders the DCF tracker totally useless. It must be a bug that it can go for so long with zero detections from pgie.

Video 1: Shows a person detected and leaving the scene. Notice how the bbox still lingers and does not disappear: https://www.dropbox.com/s/dtcopqtco8blsoq/nano1_2_1597313386000.MP4?dl=0

Video 2: This is from 15 more seconds after the last clip finished recording. The lingering bbox is still there: https://www.dropbox.com/s/s158qkkthe4eh3h/nano1_2_1597313415000.MP4?dl=0

Video 3: This final video is from another 1.5 minutes after the last video finished. The lingering bbox is still there but at least it is shrinking in size now and extinguishes itself: https://www.dropbox.com/s/s3bztoaxbqwhu9u/nano1_2_1597313523000.MP4?dl=0

How can the dcf tracker keep lingering bboxes around for over a minute or two since the last detection by pgie?

Something else to note - in the third video you can see the zombie bbox starts to shring (and it actually disappers afterwards). However sometimes the zombie bbxo does the reverse and actually grows and grows till it takes up the entire frame and never goes away unless I stop the program!

ps. You can ignore the yellow shading. I simply have a probe which colors the bbox yellow if it stationary.

Hey
So what are the current issues after you tuned the few parameters, flickery + lingering or just lingering?
could you share your config file corresponding to the 3 videos?
Your input is a camera, right? Have you tried local video input? What I’m asking is want to repro your issue locally. Yeah, it’s really weird per your videos. I would suggest to share all your setup with me, you can send me a private msg if you have any concern, or you can just write a simple demo or other way to repro your issue using your configs.

Have responded in a direct message with more details… Will report back here with results once we find a resolution for anyone that may be watching.

Thanks , I will try to repro it using deepstream-app, also is it possible to send me your nvinfer config file, just need to confirm the interval is 4, right? I will use the detection model in DS package to repro. Not sure have you tried this once using deepstream-app. using pipeline …->pgie->tracker->… . Meanwhile I will try to repro using this pipeline with the input stream provided in DS

I’ve checked with all different intervals from 0 to 4.

When minTrackingConfidenceDuringInactive is 1.0 or higher it looks like the bbox is only drawn when the pgie finds an object - so the higher the pgie interval the longer the flickering duration of the bboxes.

Note that the pgie config here will show 0 because I overwrite the property in my program. My program works with AWS IoT so I set config options in the Thing Shadow. My program reads these and overwrites the gstreamer element settings at runtime.

pgie config:

# Copyright (c) 2018 NVIDIA Corporation.  All rights reserved.
#
# NVIDIA Corporation and its licensors retain all intellectual property
# and proprietary rights in and to this software, related documentation
# and any modifications thereto.  Any use, reproduction, disclosure or
# distribution of this software and related documentation without an express
# license agreement from NVIDIA Corporation is strictly prohibited.

# Following properties are mandatory when engine files are not specified:
#   int8-calib-file(Only in INT8)
#   Caffemodel mandatory properties: model-file, proto-file, output-blob-names
#   UFF: uff-file, input-dims, uff-input-blob-name, output-blob-names
#   ONNX: onnx-file
#
# Mandatory properties for detectors:
#   num-detected-classes
#
# Optional properties for detectors:
#   enable-dbscan(Default=false), interval(Primary mode only, Default=0)
#   custom-lib-path,
#   parse-bbox-func-name
#
# Mandatory properties for classifiers:
#   classifier-threshold, is-classifier
#
# Optional properties for classifiers:
#   classifier-async-mode(Secondary mode only, Default=false)
#
# Optional properties in secondary mode:
#   operate-on-gie-id(Default=0), operate-on-class-ids(Defaults to all classes),
#   input-object-min-width, input-object-min-height, input-object-max-width,
#   input-object-max-height
#
# Following properties are always recommended:
#   batch-size(Default=1)
#
# Other optional properties:
#   net-scale-factor(Default=1), network-mode(Default=0 i.e FP32),
#   model-color-format(Default=0 i.e. RGB) model-engine-file, labelfile-path,
#   mean-file, gie-unique-id(Default=0), offsets, gie-mode (Default=1 i.e. primary),
#   custom-lib-path, network-mode(Default=0 i.e FP32)
#
# The values in the config file are overridden by values set through GObject
# properties.

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
model-file=/opt/smartguardian/models/Primary_Detector_Nano/resnet10.caffemodel
proto-file=/opt/smartguardian/models/Primary_Detector_Nano/resnet10.prototxt
labelfile-path=/opt/smartguardian/models/Primary_Detector_Nano/labels.txt
batch-size=8
process-mode=1
model-color-format=0
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=2
num-detected-classes=4
# ignore the Roadsign (3) class
filter-out-class-ids=3
interval=0
gie-unique-id=1
output-blob-names=conv2d_bbox;conv2d_cov/Sigmoid
force-implicit-batch-dim=1
#parse-bbox-func-name=NvDsInferParseCustomResnet
#custom-lib-path=/path/to/libnvdsparsebbox.so
## 0=Group Rectangles, 1=DBSCAN, 2=NMS, 3= DBSCAN+NMS Hybrid, 4 = None(No clustering) 
cluster-mode=1
#scaling-filter=0
#scaling-compute-hw=0

[class-attrs-all]
# # threshold has been deprecated - use pre-cluster-threshold instead
pre-cluster-threshold=0.2
#pre-cluster-threshold=0.3
#group-threshold=1
## Set eps=0.7 and minBoxes for cluster-mode=1(DBSCAN)
#eps=0.2
eps=0.7
minBoxes=1
roi-top-offset=0
roi-bottom-offset=0
detected-min-w=0
detected-min-h=0
detected-max-w=0
detected-max-h=0

# Per class configuration : Car = 0, Bicycle = 1, Person = 2, Roadsign = 3
[class-attrs-2]
pre-cluster-threshold=0.5
eps=0.7
minBoxes=1
##group-threshold=1
#roi-top-offset=20
#roi-bottom-offset=10
#detected-min-w=40
#detected-min-h=40
#detected-max-w=400
#detected-max-h=800

[class-attrs-0]
pre-cluster-threshold=0.4
eps=0.7
minBoxes=1
##group-threshold=1
#roi-top-offset=20
#roi-bottom-offset=10
#detected-min-w=40
#detected-min-h=40
#detected-max-w=400
#detected-max-h=800

Cool, let me try and give you response when get update

Something else to add here - I’m finding that when my pipeline includes the dcf tracker and has been running for a while non-stop it slows right down. When I walk in front of a camera it can take 2 minutes till I see this detection in my probe function (which is tapped in after the tracker).
It can cause other anomalies as well - I have smart record with a max recording duration of 30 seconds - yet strangely the recorded files can sometimes be 5 or 6 minutes long and are frozen on one frame for most of that time.

I have output the size of my queues and they are not filling up so I’m not sure what the issue could be.

Note: This does not happen when using either of the IOU or KLT trackers.

Hey Jason,
I cannot repro your issue using deepstream-app with your config files and input stream is a local mp4 file in DS package. Could you help to repro your issue using deepstream-app?

I’ll try with deepstream app. Just note that you need to leave it run all day before you get the weird behaviour.

Hello Jason,

Generally speaking, the bbox flickering issue during tracking can be mitigated if you lower minTrackerConfidence. The zombie bbox issue in case of no detection can be mitigated by increasing minTrackingConfidenceDuringInactive.

I would recommend you to set pgie interval = 0 and fix minTrackingConfidenceDuringInactive to 99 and try adjusting minTrackerConfidence first. After that, you can try adjusting pgie interval as you like while fine-tuning the two tracker params.

Please try that and report us your observations. Also, from your video, it’s not clear if there’s frequent ID changes or not because your bbox label doesn’t show ID. So, please try displaying object ID as well.

In the meantime, @bcao please reproduce the issue locally.

1 Like

Should I create a new topic for this issue then:

Is the DCF tracker suitable for the nano or does it requires something more powerful like the NX or more?

What could cause the slowing down of the pipeline by using the DCF tracker (I set the maxTargetsPerStream to a low value of 20 as well - maybe I should try with 3).

I do notice the number of buffers in the queue before the nvinfer component increasing over a few hours… When it gets to around 4 or 5 buffers in that queue however I start seeing strange results. Not sure why as the default value for a queue is to allow 200 buffers.

When I use other trackers like IoU or KLT I never see the buffers in the queue before nvinfer increase above 0 so the DCF tracker definitely puts the system under quite some load.

Yeah, pls create a new topic.
Let’s discuss the previous DCF tracker issue(flickering zombie ) on this topic

Hey @jasonpgf2a

If you saw my earlier comment in this post, disregard it.

However i suggest you change the stateEstimatorType=2 (Kalman Filter {“Sort” uses this}), useBufferedOutput=1, minMatchingscoreOverall=0.2, useGlobalMatching=1, minVisibility4Tracking=0.1, useHOG=1, and minTrackingConfidenceDuringInactive=0.7

To speed it up, featureImgSizeLevel=1 && SearchRegionPaddingScale=2

You can also grab all the related tracking information: (Ghost IDS, Active IDS, etc)

Little Recomendation :

You should create your own tracker, a winning combination i found was KLT with a Kalman Filter… Its a great starting spot to make a more advanced tracker (multi camera, multi object) and once you’re done, you’ll have a tracker that is much more accurate and faster than NVDCF.

1 Like

@rsc44 have you made the code for your custom tracker available ? I’d be interested to see how you went about adding a Kaplan filter to the klt tracker… ;-)