VPI KLT Tracker not robust to image rotation

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
1

I’m having an issue where when I call vpiSubmitKLTBoundingBox(), bounding boxes are being marked invalid (trackingStatus = 1) even with very minor changes in the image sequence. Here are the parameters used:

KLT params Value
num iterations scaling 20
ncc threshold update 0.8
ncc threshold kill 0.6
ncc threshold stop 1.0
max scale change 0.2
max translation change 1.5

My code is based on VPI - Vision Programming Interface: KLT Feature Tracker. I’m noticing that if I merely rotate an image counterclockwise by 1 degree, about 20% of the bounding boxes fail to track. When I rotate an image counterclockwise by 5 degrees, almost every bounding box fails to track. Is this unexpected behavior? When I use opencv’s tracker (calcOpticalFlowPyrLK), it successfully tracks all features through 5 degree image rotations. Note, predictions are identity in both cases. Relaxing the above parameters seems to have no effect on the rate of invalid bounding boxes. I’m on jetpack 4.4, currently using CUDA backend.

Sample first image:
image

Sample second image is first image rotated one degree counterclockwise, many bounding boxes are marked invalid.
image

3

Hi,

KLT default tracker is tuned for tracking an object within a video.
Please set the parameter based on your use case first.

https://docs.nvidia.com/vpi/group__VPI__KLTBoundingBoxTracker.html#structVPIKLTBoundingBoxTrackerParams
For example. maxTranslationChange indicates the maximum possible motion between two image.
Set it into 1.5 indicates the maximal valid tracking range is 1.5 *sizeof(bbox).
For an rotation use case, you may need a larger search range.

Thanks.