I am running Deepstream 6.0 onboard a quadcopter on a Jetson Xavier NX. Due to such configuration, there is a lot of movement from the camera during flight from orientation and positional change of the camera. While the target on the ground is moving, its motion is negligible compared to the motion of the camera.
My GStreamer pipeline (CPP source code) involves 2 single-class models in series followed by an NvTracker element. I use an interval=3 on both the NvInfer elements to allow for “real-time” processing with the NvTracker giving a throughput of ~12-15 fps. Given any image, there should only be a maximum of 2 detections (1 from NvInfer1, 1 from NvInfer2). The problem is that there is bounding box “lag” from the motion, similar to what was posted here Deepstream Nvtracker, bounding boxes issues - #5 by Rik. I have tried the suggested config files:
Since I am using the detections (bbox center) coupled with depth (from a stereo camera) to calculate 3D position of the target, any substantional motion lag/error on the bounding box is causing me issues. Any help tuning/providing a config file for the NvTracker would be appreciated
The bbox lag issue often occurs when there’s a fast moving object or there’s jumpy movements due to low fps or camera movement. To mitigate this issue, the state estimator can be tuned to adapt to the newer measurement faster. Please refer to Gst-nvtracker — DeepStream 6.1.1 Release documentation where processNoiseVar and measurementNoiseVar params were explained.
I would first recommend to disable the state estimator by setting stateEstimatorType: 0. Then the tracker would use the detector’s bbox without any filtering, so there should be no lag, yet no noise would be reduced due to the absence of the state estimator. Once you verify there’s no lag, you can enable again the state estimator, but set much higher processNoiseVar and lower measurementNoiseVar than what is in config_tracker_NvDCF_accuracy.yml to allow the state estimator adapt to the new measurements faster. And then gradually try adjusting those params.
I setstateEstimatorType : 0 and immediately am getting no lag with “perfect” bounding boxes around my targets! I did try your second part of your suggestion to tune processNoiseVar and measurementNoiseVar but with less success. I had to set prcessNoise4Loc: 150.0, processNoiseVar4Size : 50.0, processNoiseVar4Vel: 150.0, measurementNoiseVar4Detector : 2.0, and measurementNoiseVar4Tracker: 15.0 to start seeing results close to settings stateEstimatorType : 0. But even then, the boxes were less frequent (more false negatives) and still had some lag.
For my project, stateEstimatorType : 0 is exactly what I need. Regardless, I am still curious what would be the benefit of tuning the other parameters above if the bounding boxes didn’t have any error and it seems like turning on the StateEstimator introduces lag, something I can’t work with? Thanks again
Even with state estimator enabled, if you set the values of all processNoiseVar params really high, and set the values of all measurementNoiseVar really low, it would behave like state estimator disabled. 150.0 may not high enough.