Tuning NvDCF tracker for low FPS

Hello,

I am trying to use the NvDCF tracker to track objects in videos that have low a low frame rate (say 2-5 FPS).I understand that the low FPS will make tracking more difficult, however, I believe that as of now I am encountering a few issues that can probably be solved with proper tuning.

I am using a Tesla T4 and the Deepstream container 6.0.1.

In order to debug my pipeline I am saving the output of that tracker to a video using a probe. I am using Yolo v5 as PGIE so we can safely assume it is capable of detecting the majority of the objects. The output video shows bounding boxes around the objects detected as well as their tracker id. Unfortunately, as of now I can’t share the video for privacy reasons. I will be able to share the video with you privately if you let me know how.

The following are the issues that I am encountering:

  • Some objects are not detected: to be more precise, I think the PGIE is detecting them, but then the tracker is deleting them. I believe that this was caused by probationAge: 3. Since most of the objects not reported were moving really fast, I imagine it is possible that the tracker had issues tracking them for more than 3 frames. Therefore, I set probationAge: 0 and minIouDiff4NewTarget: 1. It is extremely important for me that no object will ever be cancelled by the tracker (I am willing to tolerate false positive). Is there are any other setting that can cause a detection to be deleted/discarded by the tracker?
  • Objects that are moving fast easily change tracking id. Example: I have cars moving from left to right. When they move fast, the tracker often starts a new track every frame (i.e. the tracker gives the car a new tracking id each frame). This is unexpected as the car sometime stills overlap with its bounding box in the previous frame
  • Objects getting closer to the border gets new tracking ids (as in the previous bullet point). Example: when a car appears close the left border (and the car is not yet entirely visible because part of it it’s out of the camera range) it receives a tracking id 1. As soon as the entire car is visible, it receives tracking id 2. When the car reaches the right border of the image, and the car starts to disappear, the car receives the new tracking id 3. I suspect this is caused by the fact that close to borders the car is partially occluded.

These are the settings that I am using. I used as starting point the settings I found you Python examples.

%YAML:1.0
################################################################################
# SPDX-FileCopyrightText: Copyright (c) 2019-2021 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################

BaseConfig:
  minDetectorConfidence: 0   # If the confidence of a detector bbox is lower than this, then it won't be considered for tracking

TargetManagement:
  enableBboxUnClipping: 1   # In case the bbox is likely to be clipped by image border, unclip bbox
  maxTargetsPerStream: 150  # Max number of targets to track per stream. Recommended to set >10. Note: this value should account for the targets being tracked in shadow mode as well. Max value depends on the GPU memory capacity

  # [Creation & Termination Policy]
  minIouDiff4NewTarget: 0.5   # If the IOU between the newly detected object and any of the existing targets is higher than this threshold, this newly detected object will be discarded.
  minTrackerConfidence: 0.2   # If the confidence of an object tracker is lower than this on the fly, then it will be tracked in shadow mode. Valid Range: [0.0, 1.0]
  probationAge: 0 # If the target's age exceeds this, the target will be considered to be valid.
  maxShadowTrackingAge: 30  # Max length of shadow tracking. If the shadowTrackingAge exceeds this limit, the tracker will be terminated.
  earlyTerminationAge: 1   # If the shadowTrackingAge reaches this threshold while in TENTATIVE period, the target will be terminated prematurely.

  # NvDCF TargetManagementsettings
  # searchRegionPaddingScale  # Search region size

TrajectoryManagement:
  useUniqueID: 0   # Use 64-bit long Unique ID when assignining tracker ID. Default is [true]
  enableReAssoc: 1    # Enable Re-Assoc

  # [Re-Assoc: Motion-based]
  minTrajectoryLength4Projection: 20  # min trajectory length required to make projected trajectory
  prepLength4TrajectoryProjection: 10 # the length of the trajectory during which the state estimator is updated to make projections
  trajectoryProjectionLength: 90      # the length of the projected trajectory

  # [Re-Assoc: Trajectory Similarity]
  minTrackletMatchingScore: 0.5       # min tracklet similarity score for matching in terms of average IOU between tracklets
  maxAngle4TrackletMatching: 30       # max angle difference for tracklet matching [degree]
  minSpeedSimilarity4TrackletMatching: 0.2    # min speed similarity for tracklet matching
  minBboxSizeSimilarity4TrackletMatching: 0.6 # min bbox size similarity for tracklet matching
  maxTrackletMatchingTimeSearchRange: 20      # the search space in time for max tracklet similarity

DataAssociator:
  dataAssociatorType: 0 # the type of data associator among { DEFAULT= 0 }  # NOTE: never found this param in the doc!
  associationMatcherType: 0 # the type of matching algorithm among { GREEDY=0, GLOBAL=1 }
  checkClassMatch: 1  # If checked, only the same-class objects are associated with each other. Default: true

  # [Association Metric: Thresholds for valid candidates]
  minMatchingScore4Overall: 0.0   # Min total score
  minMatchingScore4SizeSimilarity: 0.6  # Min bbox size similarity score
  minMatchingScore4Iou: 0.0       # Min IOU score
  minMatchingScore4VisualSimilarity: 0.7  # Min visual similarity score

  # [Association Metric: Weights]
  matchingScoreWeight4VisualSimilarity: 0.6  # Weight for the visual similarity (in terms of correlation response ratio)
  matchingScoreWeight4SizeSimilarity: 0.0    # Weight for the Size-similarity score
  matchingScoreWeight4Iou: 0.4   # Weight for the IOU score

StateEstimator:
  stateEstimatorType: 1  # the type of state estimator among { DUMMY=0, SIMPLE=1, REGULAR=2 }

  # [Dynamics Modeling]
  processNoiseVar4Loc: 2.0    # Process noise variance for bbox center
  processNoiseVar4Size: 1.0   # Process noise variance for bbox size
  processNoiseVar4Vel: 0.1    # Process noise variance for velocity
  measurementNoiseVar4Detector: 4.0    # Measurement noise variance for detector's detection
  measurementNoiseVar4Tracker: 16.0    # Measurement noise variance for tracker's localization

  # noiseWeightVar4Loc  # Noise covariance weight for bbox location; if set, location noise will be proportional to box height
  # noiseWeightVar4Vel  # Noise covariance weight for bbox velocity; if set, location noise will be proportional to box height
  # useAspectRatio  # Use aspect ratio in Kalman Filter’s states

VisualTracker:
  visualTrackerType: 1 # the type of visual tracker among { DUMMY=0, NvDCF=1 }

  # [NvDCF: Feature Extraction]
  useColorNames: 1     # Use ColorNames feature
  useHog: 0            # Use Histogram-of-Oriented-Gradient (HOG) feature
  featureImgSizeLevel: 2  # Size of a feature image. Valid range: {1, 2, 3, 4, 5}, from the smallest to the largest
  featureFocusOffsetFactor_y: -0.2 # The offset for the center of hanning window relative to the feature height. The center of hanning window would move by (featureFocusOffsetFactor_y*featureMatSize.height) in vertical direction

  # [NvDCF: Correlation Filter]
  filterLr: 0.075 # learning rate for DCF filter in exponential moving average. Valid Range: [0.0, 1.0]
  filterChannelWeightsLr: 0.1 # learning rate for the channel weights among feature channels. Valid Range: [0.0, 1.0]
  gaussianSigma: 0.75 # Standard deviation for Gaussian for desired response when creating DCF filter [pixels]

Thank you so much for your help

Related to the topic above, I found the sample settings for the IoU config into the Deepstream 6.0.1 container in the file /opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/config_tracker_IOU.yml:

%YAML:1.0
################################################################################
# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
# DEALINGS IN THE SOFTWARE.
################################################################################

BaseConfig:
  minDetectorConfidence: 0   # If the confidence of a detector bbox is lower than this, then it won't be considered for tracking

TargetManagement:
  maxTargetsPerStream: 150  # Max number of targets to track per stream. Recommended to set >10. Note: this value should account for the targets being tracked in shadow mode as well. Max value depends on the memory capacity  

  # [Creation & Termination Policy]
  minIouDiff4NewTarget: 0.5   # If the IOU between the newly detected object and any of the existing targets is higher than this threshold, this newly detected object will be discarded.
  probationAge: 4   # If the target's age exceeds this, the target will be considered to be valid.
  maxShadowTrackingAge: 38 # Max length of shadow tracking. If the shadowTrackingAge exceeds this limit, the tracker will be terminated.
  earlyTerminationAge: 1 # If the shadowTrackingAge reaches this threshold while in TENTATIVE period, the target will be terminated prematurely.

TrajectoryManagement:
  useUniqueID: 0   # Use 64-bit long Unique ID when assignining tracker ID.

DataAssociator:
  dataAssociatorType: 0 # the type of data associator among { DEFAULT= 0 }
  associationMatcherType: 0 # the type of matching algorithm among { GREEDY=0, GLOBAL=1 }
  checkClassMatch: 1  # If checked, only the same-class objects are associated with each other. Default: true

  # [Association Metric: Thresholds for valid candidates]
  minMatchingScore4Overall: 0.0   # Min total score
  minMatchingScore4SizeSimilarity: 0.0    # Min bbox size similarity score
  minMatchingScore4Iou: 0.0       # Min IOU score  

  # [Association Metric: Weights]  
  matchingScoreWeight4SizeSimilarity: 0.4    # Weight for the Size-similarity score
  matchingScoreWeight4Iou: 0.6               # Weight for the IOU score

I see that all the thresholds are set to 0: minDetectorConfidence, minMatchingScore4Overall, minMatchingScore4SizeSimilarity, minMatchingScore4Iou. I am wondering: if there is no threshold set, how can the tracker decides if two objects are the same or not? Is there another threshold not specified?

Sorry for the late response, our team will do the investigation and provide suggestions soon. Thanks

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

Can you share your video with private channel?