Use resnet50_market1501 model as secondary gie

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
Orin NX / AGX
• DeepStream Version
6.3
• JetPack Version (valid for Jetson only)
5.1.2

This is a follow up to this question: How can I get a secondary-gie classifier CNN model output as object feature for reid?

It appears that deepstream 6.3 comes with the resetnet50_market1501.etlt model, but the only reference I can see that uses it is in the tracker library. I’m trying to do something similar only I’m not able to use the libnvds_nvmultiobjecttracker library directly. Is there a way to use that model directly as an sgie and and grab the feature vector an attach it with the bounding box output? As far as I can tell, the source code for libnvds_multiobjecttracker.so is not available to use as a reference.

3

please refer to this ready-made sample deepstream-mdx-perception-app in tao4.0_ds6.3ga branch. As the diagram shown, sgie saves the feature vector to output meta with resnet50_market1501_aicity156.etlt.

4

That looks like exactly what I’d like to do. I’ll start going through it. What’s the difference between the resnet50_market1501_aicity156 model and resnet50_market1501 model that comes with deepstream?

5

Both are reidentificationnet model. resnet50_market1501 is an old version.

6

I am going through the example code. I was able to build the deepstream_mdx_perception app and run it with the It looks like for that , there is a tee after the the multiobject tracker with one branch going to the sgie and the other going to the analytics branch (see attached pipeline). Why is it set up that way?

The way I’d like to set it up is much simpler: (sources) → nvstreammux → queue → nvinfer (pgie) → queue → nvinfer (sgie) → nvmultistreamtiler → (sinks)
I’ve got that building and playing, so now just trying to figure out how to extract the tensors. I currently have a callback registered on the pgie_src_pad to get the bounding boxes. Can I just move that callback to the output of the of the sgie so I can grab the bounding boxes, classes, and tensors all at once? (It looks like this is the relevant piece from the example code.

playing
playing26897×899 1.43 MB

7

Sorry for the late reply, Is this still an DeepStream issue to support? Thanks!
it is for performance. with tracker, sgie classfication results will saved. secondary_gie_bin_tee will send buffer downstream without waiting sgie inference.

right, all inference results are saved in metadata. you may add a probe on src pad of sgie and refer to the mentioned code.

8

I’ve been running this pipeline for a week or so and have noticed that sometimes no feature vector is returned. My network is trained to return two classes and the sgie is set to operate on both of them. Most of the time feature vectors are there, but occasionally they’re not:

1743687326.994099 [debug] [Processed Frame] received processed frame. number of objects: 3
1743687326.994103 [debug]   class: person bbox x,y:(420,88) sz: (12,35) feature vector length: 0
1743687326.994105 [debug]   class: person bbox x,y:(898,226) sz: (17,47) feature vector length: 256
1743687326.994107 [debug]   class: vehicle bbox x,y:(734,41) sz: (81,52) feature vector length: 256
1743687327.147618 [debug] [Processed Frame] received processed frame. number of objects: 3
1743687327.147623 [debug]   class: person bbox x,y:(430,78) sz: (12,35) feature vector length: 0
1743687327.147626 [debug]   class: person bbox x,y:(913,225) sz: (17,46) feature vector length: 256
1743687327.147628 [debug]   class: vehicle bbox x,y:(748,35) sz: (82,52) feature vector length: 256
1743687327.347562 [debug] [Processed Frame] received processed frame. number of objects: 3
1743687327.347567 [debug]   class: person bbox x,y:(464,61) sz: (12,37) feature vector length: 0
1743687327.347571 [debug]   class: person bbox x,y:(952,217) sz: (15,45) feature vector length: 0
1743687327.347574 [debug]   class: vehicle bbox x,y:(784,20) sz: (83,54) feature vector length: 256
1743687327.547659 [debug] [Processed Frame] received processed frame. number of objects: 3
1743687327.547679 [debug]   class: person bbox x,y:(496,47) sz: (12,37) feature vector length: 0
1743687327.547683 [debug]   class: person bbox x,y:(989,207) sz: (16,47) feature vector length: 256
1743687327.547685 [debug]   class: vehicle bbox x,y:(820,9) sz: (84,51) feature vector length: 256
1743687327.747246 [debug] [Processed Frame] received processed frame. number of objects: 3
1743687327.747251 [debug]   class: person bbox x,y:(524,33) sz: (13,37) feature vector length: 0
1743687327.747255 [debug]   class: person bbox x,y:(1021,199) sz: (18,48) feature vector length: 256
1743687327.747258 [debug]   class: vehicle bbox x,y:(852,1) sz: (85,45) feature vector length: 256

The numbers in parentheses are the bounding box location and size so it appears that the objects that are missing the feature vectors are on the smaller side. I guess I would assume there’s a minimum bounding box size needed for the sgie to operate on. Is that correct? Is there a way to “expand” the bounding box so it meets that threshold? Best effort is better than nothing.

Thanks

9

Sorry for the late reply! can you reprodcue this issue with deepstream_mdx_perception? Here are some solutions:

  1. you can set detected-min-w and detected-min-h of nvinfer to fiter the small object.
  2. you can expand the rect of object. please refer to deepstream_infer_tensor_meta_test.cpp for how to get the object rect.