Feature Correspondences Using Visionworks

I’m currently trying to use Visionworks for feature tracking. I’ve run the Feature Tracking demo, but I can’t seem to figure out how to determine temporal feature correspondences. The predefined functions “getCurrFeatures()” and “getPrevFeatures()” return arrays of different sizes (which makes sense since some feature were not tracked and some features were added). Without documentation about how these arrays are structured it seems impossible to determine correspondences between features in these arrays.

I know this is possible since the demo draws arrows between the previous and current location of tracked features, but I couldn’t trace down where that was happening.

Hi,

You can find the visionworks document here:
https://developer.nvidia.com/embedded/visionworks

getCurrFeatures() return the kp_curr_list_ which is calculated by frame T-1 and frame T.
The vx_array is described as vx_keypoint_t.

getPrevFeatures() return the kp_curr_list_ of T-1 frame, which is calculated by frame T-2 and frame T-1.

You can find the source code of feature tracker here:
/usr/share/visionworks/sources/demos/feature_tracker/feature_tracker.cpp

Thanks.

Hi AastaLLL,

I’ve already extensively read over the documentation and source code. I am able to extract arrays of features from getCurrFeatures() and getPrevFeatures(). I am trying to determine feature correspondences between elements in these two arrays. Since there is no feature ID associated with each element in the array it doesn’t seems obvious how to do this. I also need to be able to track correspondences across multiple frames. (This is fundamental for any useful feature tracking application).

To follow up, I’m specifically looking for information about how the “nvxFastTrackNode” structures the output array given the “tracked_points” input. The documentation provides the following description of the function parameters:

Parameters
[in]	context	Specifies the context.
[in]	input	Specifies the input image. Only VX_DF_IMAGE_U8 format is supported. The image size must be not greater than 232 pixels.
[out]	output	Specifies the output list of corners. Only VX_TYPE_KEYPOINT, NVX_TYPE_KEYPOINTF and NVX_TYPE_POINT2F item types are supported. The array capacity must be explicitly provided, even for virtual arrays.
[in]	mask	Specifies the optional mask. Only VX_DF_IMAGE_U8 format is supported. The mask should have the same size as input image.
[in]	tracked_points	Specifies the optional tracked features list. If specified, it should have the same item type as output array.
[in]	type	Specifies the number of neighborhoods to test. Supported values : 9, 10, 11, 12.
[in]	threshold	Specifies the threshold on difference between intensity of the central pixel and pixels of a circle around this pixel. The threshold value should be less than 255.
[in]	cell_size	Specifies the size of cells for cell-based non-max suppression. The cell_size should be less than input image dimensions.
[out]	num_corners	Specifies the total number of detected corners in image (optional). It should be VX_TYPE_SIZE scalar.

However it is unclear how the “output list of corners” is structures. For example, are new feature added at the end of the array? Is the order of the tracked points preserved?

Hi,

There are some clues from the output render.

Please check the code to draw motion arrow in feature tracker sample:

vx_array old_points = tracker->getPrevFeatures();
vx_array new_points = tracker->getCurrFeatures();
renderer->putArrows(old_points, new_points, arrowStyle);

This function is implemented like this:

uint32_t num_items = std::min(old_points.num_items, new_points.num_items);

for (uint32_t start_x = 0u; start_x < num_items; start_x += bufCapacity_)
{
    uint32_t end_x = std::min(start_x + bufCapacity_, num_items);
    updateLinesArray(start_x, end_x, old_points, new_points, width, height, scaleRatio);
    renderArray(end_x - start_x, line_style);
}

The points between old and new is matching across the index, except the new points located in the end.
Thanks.

Hi Aasta,

Thanks for the reply. Sorry I’ve taken a few days to get back to you, I’ve wanted to make sure I tested this thoroughly. I’ve checked it out and I’m successfully able to use the ‘getCurrFeatures()’ and ‘getPrevFeatures()’ functions to match features for a given frame, and even draw arrows showing the motion. However, I’m still unable to track features across multiple frames.

At any given frame I can call ‘getCurrFeatures()’ and ‘getPrevFeatures()’ and find temporal matches. However, at the next frame the order of the features in the array seems random. For example, the feature that was the first element in ‘getCurrFeatures()’ at time t, is not necessarily the first element of ‘getCurrFeatures()’ at time t+1. Without a feature id field associated with the point it still seems like a challenge to actually track features over multiple frames.

Hi,

Sorry that if my previous statement is not clear enough.

The order in next frame doesn’t be guaranteed.
The matching algorithm create a new memory and doesn’t reserve the order information.

So the corresponding information is only available in the same time slot.

Thanks.

That’s unfortunate. Correct me if I’m wrong, but doesn’t that essentially make it impossible to use this API for any real world feature tracking application? If this is actually the case do you have any other recommendations for usable feature trackers on Jetson platforms. I imagine that there has to be some working version that exists (for example as part of the SfM library)?

Hi,

The main problem of VisionWorks feature point is that it doesn’t include an ID information.
This make tracking in temporal domain become difficult.

Maybe you can check if the function in OpenCV can meet your requirement.

Thanks.