NvDCF maxShadowTrackingAge issue

Accelerated Computing Intelligent Video Analytics DeepStream SDK
#1

I suspect there’s some bug in minTrackerConfidence, maxShadowTrackingAge or probationAge of the Nvdcf tracker.

We have a reproducible test case with Deepstream 5.0.1, where we run our app (based on the reference app architecture) from the same video in 2 modes, with nvdcf tracker enabled and no tracker.

Configuration was:

  • probationAge: 11
  • earlyTerminationAge
  • minTrackingConfidenceDuringInactive: 0.92
  • maxShadowTrackingAge: 10

Attaching 2 dumps of deepstream metadata written to CSV for one particular object.

When running without the tracker, the object was detected once by the primary detector in 1 frame (5864). No other detection was made, and no other objects were in this frames, 14 frames before, and some 100 frames after.

When running with the tracker, the same object was output in frame 5875 (11 frames later) and lingered until frame 6028.

Two potentially critical issues here:

  • why did the object get out of temporary mode after 11 frame, if there were no detections for 10 frames after it was first detected
  • why the object wasn’t terminated within maxShadowTrackingAge but was tracked for another 43 frames

@detector_only.txt (271 Bytes) detector_tracker.txt (6.9 KB)

1 Like
#3

Hey, what’s the interval value of your pgie detector?
There are complex termination logic inside DCF, could you refer https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_troubleshooting.html#nvdcf-tracker-parameter-tuning to make the termination policy more aggressive

#4

PGIE interval was set to 0.

Yes, setting minTrackingConfidenceDuringInactive above 1 made the object disappear but, I’ve read https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_troubleshooting.html#nvdcf-tracker-parameter-tuning and https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvtracker.html and still don’t understand a number of issues.

  1. The documentation explicitly states that minTrackingConfidenceDuringInactive is a float 0.0 to 1.0
  2. with just one (false) detection, probationAge of 11 and nothing else around for several frames the object should never have been activated.
  3. For another 43 frames, there was no detector box to match, so why the tracker didn’t switch into shadow mode and terminate the object. Is there any specific metadata we can use to debug this?

In addition, the tracker was running with Kalman Filter as state estimator:

# [State Estimator]
useTrackSmoothing: 1  # Use a state estimator
stateEstimatorType: 2   # The type of state estimator among { moving_avg:1, kalman_filter:2 }
# [State Estimator] [MovingAvgEstimator]
trackExponentialSmoothingLr_loc: 0.5     # Learning rate for new location
trackExponentialSmoothingLr_scale: 0.5  # Learning rate for new scale
trackExponentialSmoothingLr_velocity: 0.5  # Learning rate for new velocity
# [State Estimator] [Kalman Filter]
kfProcessNoiseVar4Loc: 0.78  # Process noise variance for location in Kalman filter
kfProcessNoiseVar4Scale: 0.61  # Process noise variance for scale in Kalman filter
kfProcessNoiseVar4Vel: 0.48000000000000004   # Process noise variance for velocity in Kalman filter
kfMeasurementNoiseVar4Trk: 4.37 # Measurement noise variance for tracker's detection in Kalman filter
kfMeasurementNoiseVar4Det: 0.29000000000000004 # Measurement noise variance for detector's detection in Kalman filter
#5

Hey, would you mind to share all your configs with us and how to repro your issue?

#6

Hi I can share all the deepstream config, but the app, model and video we used are confidential and cannot be shared publicly, so how can we proceed?

#7

Is it possible to capture a very small video can be used to repro the issue, you can only share the video include the frame 5800 to 6000 just like what you mentioned?

#8

Or maybe you can try to repro it using the videos inside deepstream and share me with the repro steps?

BTW, didn’t you use the deepstream-app?

#9

No. The app architecture (component sequence) is based on the reference app, and we are using our own proprietary model

#10

@bcao to further debug it I am trying to look at NvDsPastFrameObjBatch but for some reason numFilled is always 0 even though I have:

  • enable-batch-process - true in Gstreamer element
  • enable-past-frame - true in Gstreamer element
  • useBufferedOutput - 1 in nvdcf config file

any suggestions?

#11

ok, ignore my last message.

This is a formatted dump from NvDsPastFrameObj after I rerun the the code with additional metadata.

type,object_id,tracker_confidence,age
tracker_past_object,5865,1.0000,2
tracker_past_object,5866,1.0000,3
tracker_past_object,5867,0.9870,4
tracker_past_object,5868,1.0000,5
tracker_past_object,5869,1.0000,6
tracker_past_object,5870,0.9494,7
tracker_past_object,5871,0.9087,8
tracker_past_object,5872,0.9214,9
tracker_past_object,5873,0.9780,10
tracker_past_object,5874,0.9486,11

Indeed the tracker was tracking the object for 11 frames with confidence close to 1 even though there were no detections.

#14

it’s difficult to debug where is wrong per the dumped info.
Could you help us the config/streams(maybe some public videos) how to repro the issue, so we look into more.