DeepStream 6.3 NvDCF plugin strange behavior

[jedi.vova]

• Hardware Platform: GPU
• DeepStream Version: 6.3
• TensorRT Version: 8.5.3-1
• NVIDIA GPU Driver Version: 530.30.02
• Issue Type: bug
• sample app used: deepstream-test2
• Requirement details: NvTracker plugin

There is some strange behavior in outputReidTensor. As stated here:
the Re-ID features will be extracted at intervals only when PGIE generates bounding boxes and reidExtractionInterval is met.
So, if I put reidExtractionInterval=0, Re-ID will be inferred on all PGIE (YOLO detector in my case) generated bboxes. BUT, this works only for NvDeerSORT, and not for NvDCF:

legend: each line represents index for detected objects
trackIds BEFORE tracking del line - track id for all detected objects (if object is not tracked yetm there will be *)
trackIds from obj_meta_list line - track ids for TRACKED objects (so uuntracked objects are deleted by NvTracker plugin)
objectIndex in reidTensor line - for each object in obj_meta_list print index in ReidTensor.

NvDeerSORT, all detected bboxes are inferred by ReID engine, as expected. And indexes for all objects are from 0 to N_obj, as they were in frame_meta->obj_meta_list.

Num detected objects: 25, pReidTensor len: 25
trackIds BEFORE tracking del: * 5 0 13 25 * 1 31 27 12 * 17 30 8 29 19 10 6 28 18 14 20 16 21 24
trackIds from obj_meta_list: 5 0 13 25 1 31 27 12 17 30 8 29 19 10 6 28 18 14 20 16 21 24
objectIndex in reidTensor: 1 2 3 4 6 7 8 9 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Num detected objects: 26, pReidTensor len: 26
trackIds BEFORE tracking del: * * 5 31 29 * 16 25 12 1 0 8 27 30 19 6 28 20 17 13 3 14 18 21 10 24
trackIds from obj_meta_list: 5 31 29 16 25 12 1 0 8 27 30 19 6 28 20 17 13 3 14 18 21 10 24
objectIndex in reidTensor: 2 3 4 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

NvDCF, reid is used for re-assoc. Here we can see that indexes in reidTensor are shuffled and not all objects from detector were inferred by ReID engine.

Num detected objects: 25, pReidTensor len: 24
trackIds BEFORE tracking del: * 0 22 17 13 1 * 19 * 16 8 9 4 18 23 20 11 6 21 14 5 10 7 12 3
trackIds from obj_meta_list: 0 22 17 13 1 19 16 8 9 4 18 23 20 11 6 21 14 5 10 7 12 3
objectIndex in reidTensor: 2 0 18 15 3 20 17 10 11 5 19 22 21 13 7 8 16 6 12 9 14 4

So, why Visual tracking shuffles indexes and infers not on all detected objects? Is this by design feture?

Also, can we see source code for libnvds_nvmultiobjecttracker.so? It will be much painless to debug deepstream applications…

Thank you!

[fangyul]

Only active tracked objects are guaranteed to have ReID embeddings. If a detected object is not tracked due to low confidence or tentative state, it may not have ReID embeddings. It is ok that indexes in reidTensor are shuffled. Please retrieve the correct objectIndex for each tracked object.
If you want the newly detected objects to be tracked immediately, set probationAge: 0 in tracker config.

[jedi.vova]

So, as for NvDeerSORT ReID tensor has vectors for all objects, I suppose that inside libnvds_nvmultiobjecttracker.so it is taking all detected objects and infers on them, and after DataAssociation starts.

For NvDCF, I guess DataAssociation is done using filters (visual tracking), and AFTER inferring ReID engine for vectors…

Am I right?

[kesong]

Yes, you are right. As fangyu mentioned “Only active tracked objects are guaranteed to have ReID embeddings.” Please check the nvtracker document for Target Re-Association Gst-nvtracker — DeepStream 6.3 Release documentation