VPI DCF Tracker

Autonomous Machines Jetson & Embedded Systems Jetson Orin NX
1

Hi. I’m trying to write a simple sample application of DCF tracker which is based on VPI3. And I’m facing a problem with the “vpiSubmitCropScalerBatch” function. After I put a frame and data in it, I’m waiting for outPatches with some non-zero data. But I always get zeros.

As suggested the software is written on the ground of another sample application which is KLT tracker, which is supplied with VPI3.

This is part of code:

#define NUM_SEQUENCES 10
#define MAX_OBJECTS 10
...
VPIPayload cropPayload;
CHECK_STATUS(vpiCreateCropScaler(backend, NUM_SEQUENCES, MAX_OBJECTS, &cropPayload));
...
int patchWidth = createParams.featurePatchSize * createParams.hogCellSize; // it doesn't work with other values either
int patchHeight = patchWidth * MAX_OBJECTS;
...
/// fetchFrame return VPI_IMAGE_FORMAT_RGB8 image
frames[0] = fetchFrame(frame);
// debug. checking the correctness of input parameters
getNumObj(inObjects); // checking that VPI_ARRAY_TYPE_DCF_TRACKED_BOUNDING_BOX array contains objects
// Extract from the first frame the image patches of each object given its current bounding box.
CHECK_STATUS(vpiSubmitCropScalerBatch(stream, backend, cropPayload, frames, NUM_SEQUENCES,
                                          inObjects, patchWidth, patchHeight, objPatches));
// debug. checking the correctness of output patches
extractImages(objPatches); // <- always zeros

Systems:

  1. Ubuntu 22.04 x64 with installed libvpi3
  2. Ubuntu 22.04 on Jetson Orin NX, CUDA 12.2
5

Hi,

Could you share the details of extractImages() as well?
Based on the API below, the outPatches only support the CUDA format.
https://docs.nvidia.com/vpi/group__VPI__CropScaler.html

Thanks.

6

Hi. I have found a solution for this problem. Need call vpiStreamSync(stream) after calling vpiSubmitCropScalerBatch. But new problem has appeared. After calling vpiSubmitDCFTrackerLocalizeBatch for a new frame nothing has changed. Selected ROIs don’t move.

What do you mean by the CUDA format? I create images by vpiImageCreate and get data by vpiImageLockData/vpiImageUnlock.

void extractImages(VPIImage img)
{
    int32_t w, h;
    VPIImageFormat fmt;
    vpiImageGetSize(img, &w, &h);
    vpiImageGetFormat(img, &fmt);

    VPIImageData data;
    auto status = vpiImageLockData(img, VPI_LOCK_READ, VPI_IMAGE_BUFFER_HOST_PITCH_LINEAR, &data);

    if(status == VPI_SUCCESS){
        auto tp = data.buffer.pitch.planes[0].pixelType;
        if(tp == VPI_PIXEL_TYPE_4U8){
...
7

Hi,

Could you share more info about the new issue?

Ideally, the w and h should be updated.
As well as the img value.

Do the above parameters remain unchanged?
Thanks.

8

I was able to make the tracker respond only if these values from VPIDCFTrackedBoundingBox are set: filterLR, filterChannelWeightsLR. but it doesn’t work correctly with them (bounding boxes jump on the spot or fly away). it is unclear what they should be. I tried the values from 0.0001 to 1. Nothing work correctly

9

Hi,

Could you set filterLR=0.5 and filterChannelWeightsLR=0.5 and try if it works?

Thanks.

10

Hi. For these values, the result is not very good

11

Hi,

Thanks for sharing the testing results.

We are checking internally to see if we can provide a sample for the DCF tracker, including the parameters.
Will provide more info to you later.

Thanks.

13

Hi,

Thanks for your patience.

Do you have the detected objects’ info during the frames?
If yes, could you use it to refine the tracker to see if the results are improved?

https://docs.nvidia.com/vpi/algo_dcf_tracker.html#algo_dcf_tracker_howto

Thanks.

14

What do you mean by the detected objects? I use that scheme: read frame → load to a sequence → vpiSubmitCropScalerBatch → vpiSubmitDCFTrackerLocalizeBatch → read objects → vpiSubmitCropScalerBatch → vpiSubmitDCFTrackerUpdateBatch ->Swap inObjects and outObjects → repeat

15

Hi,

To get better tracking quality, users need to handle the keypoint detection and refinement.
For our tracker sample in Deepstream, this is handled by the deep learning detector.

Thanks.