In 5.0 NVDCF the bounding boxes have a tendency to randomly start shrinking while they are on a person. I assume it has something to do with the confidence level of the tracked object. This did not happen in DS 4.2.
Is there anyway to prevent this ? What setting should i change in config_tracker.yml to mitigate this from happening ?
It should be noted that this happens regardless of the model i choose to use.
SSD is my go to, but it happens with resnet and yolo aswell.
Any insight would be appreciated. I’ll try to screenshot the BBOX shrinkage next time i encounter the issue.
Thanks
**• Hardware Platform Jetson ** • DeepStream Version 5.0 • JetPack Version 4.4 DP • TensorRT Version 7.1.0.16
There is also an issue with bounding boxes staying alive with the tracking Id of the last object to leave the frame.
Can anyone at nvidia explain the major differences between nvdcf from 4.0 to 5.0.
As I did not see these issues with DS 4.0
I am well aware that there are comments next to the settings in the config_tracker.yml file but they are not very helpful, as I do not understand what scenarios the settings should be adjusted for.
Hello, here is a screenshot of the persistent bounding box, it appears when i leave the frame, it does not happen every time. The persistant bounding box appears where i leave the frame, clearing the room of all other objects does not fix the issue.
Also it does not happen if at least 1 other person is in the video feed.
I have set interval=2 in my SSD config file.
Here is my config file for tracker_config.yml
indent preformatted text by 4 spaces
%YAML:1.0
NvDCF:
# [General]
useUniqueID: 1 # Use 64-bit long Unique ID when assignining tracker ID. Default is [true]
maxTargetsPerStream: 10 # Max number of targets to track per stream. Recommended to set >10. Note: this
value should account for the targets being tracked in shadow mode as well. Max value depends on the GPU
memory capacity
# [Feature Extraction]
useColorNames: 1 # Use ColorNames feature
useHog: 1 # Use Histogram-of-Oriented-Gradient (HOG) feature
useHighPrecisionFeature: 0 # Use high-precision in feature extraction. Default is [true]
# [DCF]
filterLr: 0.071 # learning rate for DCF filter in exponential moving average. Valid Range: [0.0, 1.0]
filterChannelWeightsLr: 0.42 # learning rate for the channel weights among feature channels. Valid Range: [0.0,
1.0]
gaussianSigma: 0.75 # Standard deviation for Gaussian for desired response when creating DCF filter [pixels]
featureImgSizeLevel: 4 # Size of a feature image. Valid range: {1, 2, 3, 4, 5}, from the smallest to the largest
SearchRegionPaddingScale: 3 # Search region size. Determines how large the search region should be scaled
from the target bbox. Valid range: {1, 2, 3}, from the smallest to the largest
# [MOT] [False Alarm Handling]
maxShadowTrackingAge: 9 # Max length of shadow tracking (the shadow tracking age is incremented when (1)
there's detector input yet no match or (2) tracker confidence is lower than minTrackerConfidence). Once
reached, the tracker will be terminated.
probationAge: 12 # Once the tracker age (incremented at every frame) reaches this, the tracker is
considered to be valid
earlyTerminationAge: 2 # Early termination age (in terms of shadow tracking age) during the probation period.
If reached during the probation period, the tracker will be terminated prematurely.
# [Tracker Creation Policy] [Target Candidacy]
minDetectorConfidence: 0.0
minTrackerConfidence: 0.5
minTargetBboxSize: 60
minDetectorBboxVisibilityTobeTracked: 0.1
minVisibiilty4Tracking: 0.1
targetDuplicateRunInterval: 2
minIou4TargetDuplicate: 0.9
# [Data Association] Matching method
useGlobalMatching: 1
minMatchingScore4Overall: 0.0
minMatchingScore4SizeSimilarity: 0.5
minMatchingScore4Iou: 0.1
minMatchingScore4VisualSimilarity: 0.4
matchingScoreWeight4VisualSimilarity: 0.8
matchingScoreWeight4SizeSimilarity: 0.1
matchingScoreWeight4Iou: 0.1
matchingScoreWeight4Age: 0.1
bboxPaddingScaleForAssociation: 0.0
visibilityRoiFactor: 0.00
# [State Estimator]
useTrackSmoothing: 1 # Use a state estimator
stateEstimatorType: 2 # The type of state estimator among { moving_avg:1, kalman_filter:2 }
# [State Estimator] [MovingAvgEstimator]
trackExponentialSmoothingLr_loc: 0.5 # Learning rate for new location
trackExponentialSmoothingLr_scale: 0.3 # Learning rate for new scale
trackExponentialSmoothingLr_velocity: 0.05 # Learning rate for new velocity
# [State Estimator] [Kalman Filter]
kfProcessNoiseVar4Loc: 0.1 # Process noise variance for location in Kalman filter
kfProcessNoiseVar4Scale: 0.04 # Process noise variance for scale in Kalman filter
kfProcessNoiseVar4Vel: 0.04 # Process noise variance for velocity in Kalman filter
kfMeasurementNoiseVar4Trk: 9 # Measurement noise variance for tracker's detection in Kalman filter
kfMeasurementNoiseVar4Det: 9 # Measurement noise variance for detector's detection in Kalman filter
# [Past-frame Data]
useBufferedOutput: 1 # Enable storing of past-frame data in a buffer and report it back
# [Instance-awareness]
useInstanceAwareness: 1 # Use instance-awareness for multi-object tracking
lambda_ia: 10 # Regularlization factor for each instance
maxInstanceNum_ia: 20 # The number of nearby object instances to use for instance-awareness
Yes if i raise the minTrackerConfidence to 0.7 the issue goes away. However the tracker starts to switch id’s more frequently. So it is not a solution, more of a band-aid. I’m not sure why the tracking ID just lingers there, shouldn’t the visual similarity score or kalman filter aid in getting rid of the lingering id ?
Some background :
I am designing a post-processing tool, that should help us with persistent tracking ID’s but, if i were to better understand what was happening under the hood of nvdcf it would take out some of the guess work of the design process.
My Goal:
I need to understand how i can configure the tracker to see less frequent ID switches what do you recommend i change in my tracker_config.yml ? Also can you explain to me the visual simularity score ?
What features are being extracted from the ROI ? is it just the color of different regions of the BBOX ? Are textures/patterns being compared ?
Am i able to extract the shadow tracking ID’s from the metadata ?
Am i able to extract the visual similarity from the metadata ?
The lingering problem will be resolved if you add the following field in tracker_config.yml This additional param will be added when we release DS 5.0 GA.
minTrackingConfidenceDuringInactive: 99
NvDCF tracker color features (e.g., ColorNames) and directional edge features (e.g., HOG) and uses correlation filter to localize the target. The visual similarity score reflects how much the tracker believes the target is at a particular location. When the background is cluttered and has similar features, it is possible that the tracker can yield a high confidence on a non-target location. By setting minTrackingConfidenceDuringInactive value high >1.0, we can avoid tracking bogus targets when detector doesn’t detect.
When a target is not confirmed by detector, the tracker still keeps track of the target yet in the shadow tracking mode. In a few frames later, if it is associated with a detector’s bbox, then it will re-activated with the persistent ID.
Please let me know if you have any further questions.
Thank you for explaining how the feature extraction works and why it’s possible for these tracking ids stay in “active tracking mode”. Your insight has really helped.
Am I able to pull the nvtracker meta data from NvDsBatchMeta ? I want to pull all the tracking meta such as tracking ids, tracking confidence levels, and tracking Id age. So I can account for trackers that are in shadow mode as well as the “active” ones.
If i pull object_id from object meta i only can recieve the “active tracking id” and the confidence from nvinfer.
I’ve checked the dev guides, but I’m not sure what the meta class is for NvMOTTrackerMeta.
I understand object meta. In a probe I I through the frame meta, then to object meta list, and finally grab all the object meta in the frame .
Psuedo code looks a little like this
(Loop)
frame_meta → object_meta_list; l_obj → data.
class_id = obj_meta.class_id
…
I dont understand where it is in the metadata struct tho. I’ve tried to search for tracking_id and tracking_confidence. It doesnt exist in frame_meta or object_meta. So how can I retrieve it ?
According to this document, there is a data struct that I could potentially use to grab nvtracker meta.
Did resolve the lingering issue, but it has a strange visual side effect by making the bounding box flash repeatedly (not an issue , just reporting what im seeing).
flashing effect is an expected behavior if you set minTrackingConfidenceDuringInactive: 99. It is because tracker is not put into Active mode whenever there’s no detector bbox associated with, no matter how confident the tracker has. So, bboxes will be shown only when there’s detector bboxes associated with, while doing shadow-tracking during the frames where no detector bbox is associated.
You can play with minTrackingConfidenceDuringInactive value and see which one works best for you. I would try a value between 0.5 to 1.0.
Regarding the metadata, please refer to an reference application source code in apps/sample_apps/deepstream-app/deepstream-app.c.
If you look at the source code, there’s a function called write_kitti_track_output . You can see how the metadata is accessed there regarding tracker info.
Hey pshin, I have followed the above and included the minTrackingConfidenceDuringInactive: 99 and still have issues with lingering detections until another person enters the frame. Here is my tracker config:
%YAML:1.0
NvDCF:
[General]
useUniqueID: 1 # Use 64-bit long Unique ID when assignining tracker ID. Default is [true]
maxTargetsPerStream: 99 # Max number of targets to track per stream. Recommended to set >10. Note: this value should account for the targets being tracked in shadow mode as well. Max value depends on the GPU memory capacity
# [Feature Extraction]
useColorNames: 1 # Use ColorNames feature
useHog: 1 # Use Histogram-of-Oriented-Gradient (HOG) feature
useHighPrecisionFeature: 1 # Use high-precision in feature extraction. Default is [true]
# [DCF]
filterLr: 0.15 # learning rate for DCF filter in exponential moving average. Valid Range: [0.0, 1.0]
filterChannelWeightsLr: 0.22 # learning rate for the channel weights among feature channels. Valid Range: [0.0, 1.0]
gaussianSigma: 0.75 # Standard deviation for Gaussian for desired response when creating DCF filter [pixels]
featureImgSizeLevel: 2 # Size of a feature image. Valid range: {1, 2, 3, 4, 5}, from the smallest to the largest
SearchRegionPaddingScale: 1 # Search region size. Determines how large the search region should be scaled from the target bbox. Valid range: {1, 2, 3}, from the smallest to the largest
# [MOT] [False Alarm Handling]
maxShadowTrackingAge: 30 # Max length of shadow tracking (the shadow tracking age is incremented when (1) there's detector input yet no match or (2) tracker confidence is lower than minTrackerConfidence). Once reached, the tracker will be terminated.
probationAge: 3 # Once the tracker age (incremented at every frame) reaches this, the tracker is considered to be valid
earlyTerminationAge: 1 # Early termination age (in terms of shadow tracking age) during the probation period. If reached during the probation period, the tracker will be terminated prematurely.
# [Tracker Creation Policy] [Target Candidacy]
minDetectorConfidence: -1 # If the confidence of a detector bbox is lower than this, then it won't be considered for tracking
minTrackerConfidence: 0.7 # If the confidence of an object tracker is lower than this on the fly, then it will be tracked in shadow mode. Valid Range: [0.0, 1.0]
minTargetBboxSize: 10 # If the width or height of the bbox size gets smaller than this threshold, the target will be terminated.
minDetectorBboxVisibilityTobeTracked: 0.0 # If the detector-provided bbox's visibility (i.e., IOU with image) is lower than this, it won't be considered.
minVisibiilty4Tracking: 0.0 # If the visibility of the tracked object (i.e., IOU with image) is lower than this, it will be terminated immediately, assuming it is going out of scene.
# [Tracker Termination Policy]
targetDuplicateRunInterval: 5 # The interval in which the duplicate target detection removal is carried out. A Negative value indicates indefinite interval. Unit: [frames]
minIou4TargetDuplicate: 0.9 # If the IOU of two target bboxes are higher than this, the newer target tracker will be terminated.
# [Data Association] Matching method
useGlobalMatching: 0 # If true, enable a global matching algorithm (i.e., Hungarian method). Otherwise, a greedy algorithm wll be used.
# [Data Association] Thresholds in matching scores to be considered as a valid candidate for matching
minMatchingScore4Overall: 0.0 # Min total score
minMatchingScore4SizeSimilarity: 0.5 # Min bbox size similarity score
minMatchingScore4Iou: 0.1 # Min IOU score
minMatchingScore4VisualSimilarity: 0.2 # Min visual similarity score
minTrackingConfidenceDuringInactive: 0.8 #1.0 # Min tracking confidence during INACTIVE period. If tracking confidence is higher than this, then tracker will still output results until next detection
# [Data Association] Weights for each matching score term
matchingScoreWeight4VisualSimilarity: 0.8 # Weight for the visual similarity (in terms of correlation response ratio)
matchingScoreWeight4SizeSimilarity: 0.0 # Weight for the Size-similarity score
matchingScoreWeight4Iou: 0.1 # Weight for the IOU score
matchingScoreWeight4Age: 0.1 # Weight for the tracker age
# [State Estimator]
useTrackSmoothing: 1 # Use a state estimator
stateEstimatorType: 1 # The type of state estimator among { moving_avg:1, kalman_filter:2 }
# [State Estimator] [MovingAvgEstimator]
trackExponentialSmoothingLr_loc: 0.5 # Learning rate for new location
trackExponentialSmoothingLr_scale: 0.3 # Learning rate for new scale
trackExponentialSmoothingLr_velocity: 0.05 # Learning rate for new velocity
# [State Estimator] [Kalman Filter]
kfProcessNoiseVar4Loc: 0.1 # Process noise variance for location in Kalman filter
kfProcessNoiseVar4Scale: 0.04 # Process noise variance for scale in Kalman filter
kfProcessNoiseVar4Vel: 0.04 # Process noise variance for velocity in Kalman filter
kfMeasurementNoiseVar4Trk: 9 # Measurement noise variance for tracker's detection in Kalman filter
kfMeasurementNoiseVar4Det: 9 # Measurement noise variance for detector's detection in Kalman filter
# [Past-frame Data]
useBufferedOutput: 0 # Enable storing of past-frame data in a buffer and report it back
# [Instance-awareness]
useInstanceAwareness: 0 # Use instance-awareness for multi-object tracking
lambda_ia: 2 # Regularlization factor for each instance
maxInstanceNum_ia: 4 # The number of nearby object instances to use for instance-awareness
minTrackingConfidenceDuringInactive: 99 #gets rid of the lingering big boxes
@gabe_ddi You have “minconfidenceduringinactive” set twice.
First at 0.8 and second at 99.
Set the first one to 0.98 and delete the second entry.
0.98 works well. If it doesnt, set it to something above 1.
You should probably also set the minvisualsimilarityscore higher the 0.2.
I recommend 0.5
By the way, Nvinfer has a bug and does not output detection confidence levels correctly. You should patch it so that it correctly output detection confidence. Then you can set minDetectionConfidence to 0.2 in the .yml and it works much much better.
You now became an expert on NvDCF tracker :)
Thanks for helping out.
Hello gabesj7in,
I second to rsc44. If you set minTrackingConfidenceDuringInactive larger than 1, like 99, you are not supposed to see any lingering issue. If you still see it, then please check PGIE-only pipeline and see if your detector generates bboxes on non-target regions.
