Nvxcu to vision primitive API

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
,
1

Hi. I see that the feature_tracker_nvxcu provides an implementation of calculating optic flow as well as performing the harris_track.

There is another sample video_stabilizer that shows using these optic flow values to find a homography and warp perspective. However, these are vx (vision primitive APIs). Whereas the previous example uses nvxcu API. Is there a way for these two to interact?

The following will be the flow:
cuda_buffer (frame) → nvxcu_image → color coverting (nvxcu) → LK features and optical flow (nvxcu) → homography → warping → harris track (nvxcu).

3

Hi,

VisionWorks won’t be updated anymore, so it’s recommended to use our new VPI instead.
https://docs.nvidia.com/vpi/index.html

It also has a feature tracking and a warping (stabilization) sample:
https://docs.nvidia.com/vpi/sample_klt_tracker.html
https://docs.nvidia.com/vpi/sample_perspwarp.html

Thanks.

4

Hi thank you for your response. The link for warping mentions it is only available on pva. What does this mean? How can it run on cuda.

5

Another follow up question. If I have a cuda buffer allocated using cudamallocmanaged, how to map this to the vpi image in a zero copy method?

6

Hi,

1.
PVA is an extra hardware for Xavier.
The GPU-based implementation is not available yet.

2.
You can create the VPI image based on this function:
https://docs.nvidia.com/vpi/group__VPI__Image.html#ga569f21c44a9ef309ed6fdb617113e0c7
Just assign the GPU buffer pointer to it should be enough.

Thanks.

7

Thanks for your response. For item 2, not sure I understand you.

If I have created a buffer like this cudaMallocManaged(&image, Height*Width*sizeof(short int)); How can this be mapped to a VPI Image?

Thanks.

8

I tried the following. Please let me know if this is correct. I see the following error messages when I run my application though.

cudaMallocManaged(&image, HeightWidthsizeof(short int));
VPIImageData testData;
memset(&testData, 0, sizeof(testData));
testData.numPlanes = 1;
testData.type = VPI_IMAGE_TYPE_S16;
testData.planes[0].width = width;
testData.planes[0].height = height;
testData.planes[0].rowStride = width*sizeof(short int);
testData.planes[0].data = image;
vpiImageWrapCudaDeviceMem(&testData,0,&image);

The errors that I get are:

[WARN ] 2020-07-19 19:05:36 EGL error: 12289
[WARN ] 2020-07-19 19:05:36 EGL error: 12289
malloc(): memory corruption
Aborted (core dumped)

9

Hi,

image should be a VPIImage variable rather than a buffer pointer.

Ex.

    char* data;
    cudaMallocManaged(&data, width*height*sizeof(char));

    VPIImage image    = NULL;
    VPIImageData imgData;
    memset(&imgData, 0, sizeof(imgData));
    imgData.type                = VPI_IMAGE_TYPE_U8;
    imgData.numPlanes           = 1;
    imgData.planes[0].width     = width;
    imgData.planes[0].rowStride = width;
    imgData.planes[0].height    = height;
    imgData.planes[0].data      = data;

    VPIStatus status = vpiImageWrapHostMem(&imgData, 0, &image);

Thanks.

10

Thank you. I tried this, but get the same errors:

[WARN ] 2020-07-20 09:47:59 EGL error: 12289
[WARN ] 2020-07-20 09:47:59 EGL error: 12289
malloc(): memory corruption
Aborted (core dumped)
11

Hi in one of your previous responses you had mentioned the KLT tracker example as feature tracking. How do I pass the keypoints detected in harris corner detector into the KLT tracker? Thanks.

12

Hi,

We didn’t meet the EGL error when using the sample above.
If the error goes on, would you mind to attach your source so we can reproduce this issue on our side?

For KLT tracker question, you can get the harris output like this:

        // Now let's retrieve the output
        {
            // Lock output keypoints and scores to retrieve its data on cpu memory
            VPIArrayData outKeypointsData;
            VPIArrayData outScoresData;
            CHECK_STATUS(vpiArrayLock(keypoints, VPI_LOCK_READ, &outKeypointsData));
            CHECK_STATUS(vpiArrayLock(scores, VPI_LOCK_READ, &outScoresData));
 
            VPIKeypoint *outKeypoints = (VPIKeypoint *)outKeypointsData.data;
            uint32_t *outScores       = (uint32_t *)outScoresData.data;

Then feed it into KLT submit:
https://docs.nvidia.com/vpi/group__VPI__KLTBoundingBoxTracker.html#gaf5358eeb94a1933c5b79451c0b8110fc

Thanks.

13

Thanks a lot. This is great help. Have a good day.

14

Hi Sorry had a follow-up. For KLTBoundingBoxTracker, you need VPIArray as inputs for prediction and boxes. In your example above, keypoints and scores are already in VPIArray format. So why not pass them into KLTSubmit? Why do we need to retrieve data into cpu memory? Also is it okay that KLTSubmit expects VPIArray to be ’ The array type must be VPI_ARRAY_TYPE_KLT_TRACKED_BOUNDING_BOX’ and the prediction array to be ’ The array type must be VPI_ARRAY_TYPE_HOMOGRAPHY_TRANSFORM_2D.’ So basically I need to wrap the keypoints and scores array into this format correct?

15

Hi,

YES. KLTBoundingBoxTracker requires more information than keypoints and scores.

We implement a VPI sample that use Harris and KLTracker at the same time.
Please check it to get some information first:

Thanks.

1 Like
17