How to Map Detected Objects to the ROI They Were Detected In

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

• Hardware Platform (Jetson / GPU): Any
• DeepStream Version: 7.0 / 7.1 / 8.0 / 9.0
• Issue Type: Question

Hello,

in our application, we use multiple, overlapping region of interests. If an object is situated in an area of overlap, it will be detected in all the related ROIs (i.e., there will be multiple NvDsObjectMeta in the frame’s obj_meta_list). What is the best way to determine in which ROI an object was detected (see sketch below)?

I checked the documentation pages about various meta data, the release notes of the latest DeepStream versions and searched the forum, but the only way I found to determine in which ROI an object was detected is by comparing its bounding box coordinates with the ROI coordinates. However, for this strategy, the mapping from the detection to the ROI is not unambiguous (see sketch below).

sketch_ROI
sketch_ROI684×418 15.6 KB

Thank you for your time!

5

Can you tell me which ROIs you are talking about? The ROIs set by the nvdspreprocess plugin for objects detection or the ROIs set by the nvdsanalytics plugin for ROI objects present filtering?

6

Sorry, I am talking about the ROIs set by the nvdspreprocess-plugin.

7

Hi， for nvdspreprocess + nvinfer ROI metadata, please refer to the sample app : /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-preprocess-test, the NvDsPreProcessBatchMeta is the correct metadata to be used.

8

Thanks for your response. Following the preprocess-test-example I have been able to alter our DeepStream pipeline such that we extract the GstNvDsPreProcessBatchMeta and thus the NvDsRoiMeta for our various ROIs. However, it is not entirely clear to me what properties of this meta data I need to use to achieve the mapping of objects to their ROIs as illustrated in the picture above.

Could you please elaborate further on how to achieve such a mapping? Thank you.

9

The NvDsPreProcessBatchMeta contains all ROIs meta which are put in the NvDsRoiMeta vector “roi_vector”. Each NvDsRoiMeta element in the “roi_vector” is a ROI metadata. The NvDsRoiMeta element contains the “NvDsFrameMeta * frame_meta” which shows the frame information the ROI belongs to, and the " NvDsObjectMeta * object_meta" which contains all objects detected in this ROI.

10

Thanks for your response. Unfortunately, in our DeepStream 8.0 pipeline, the object_meta property is None for NvDsRoiMeta-objects, despite the frame_meta featuring an object count num_obj_meta different from 0, i.e.,

...

for roi_meta in preprocess_batchmeta.roi_vector:
    print("Object Meta:", roi_meta.object_meta, "-",
          "Type Object Meta:", type(roi_meta.object_meta), "-",
          "Objects in frame meta:", roi_meta.frame_meta.num_obj_meta)
    ...

...

prints

Object Meta: None - Type Object Meta: <class ‘NoneType’> - Objects in frame meta: 3

(there is only a single ROI defined).

What could be the problem here?

11

Are your nvdspreprocess working with PGIE? If so, only frame meta in ROI meta is correct. The object meta inside frame meta contains all corresponding objects found in the ROI.

12

Thanks for your response. Yes, the nvdspreprocess-elements are working with PGIE (process-on-frame=1).

However, in our tests, the object meta inside the frame meta appears to be always on frame-level and never on ROI-level, i.e., when running the following code inside the probe function on a setup with a single video source and two ROIs, the pointer to the obj_meta_list is the same for each ROI and hence the objects inside the list are the same. I.e.,

...
roi_cnt = 0
for roi_meta in preprocess_batchmeta.roi_vector:
    print("ROI ID", roi_cnt)
    print("C ptr(obj_list):", pyds.get_ptr(roi_meta.frame_meta.obj_meta_list))
    print("C ptr(frame_meta):", pyds.get_ptr(roi_meta.frame_meta))
    print("----")

    roi_cnt += 1
...

prints

...
----
ROI ID 0
C ptr(obj_list): 140097471166752
C ptr(frame_meta): 140095055410464
----
ROI ID 1
C ptr(obj_list): 140097471166752
C ptr(frame_meta): 140095055410464
----
ROI ID 0
C ptr(obj_list): 140097471160928
C ptr(frame_meta): 140095055588752
----
ROI ID 1
C ptr(obj_list): 140097471160928
C ptr(frame_meta): 140095055588752
----
...

i.e., the pointers and thus the frame_meta and the obj_meta_listare the same for each ROI.

Is there an error in above code?
What is the right way to get ROI-specific obj_meta_list?
Note that the two ROIs used in the output above do not see the same objects.

13

The objects detected in all ROIs in the same frame are attached to the same frame_meta, you may need to identify which object belongs to which ROI according to the coordinations of the objects and ROIs.

14

Thanks for your response. Unfortunately, relying on coordinates does not solve the problem of uniquely assigning objects to the ROI they were detected in, as illustrated in the image in the first post of this thread.

As an alternative approach, is the Gst-nvinfer code open source? And if so, may I ask you to point out where exactly one could modify the code of Gst-nvinfer to append additional information to the metadata that includes the ROI that has been used in each inference?

15

gst-nvinfer is open source, you can find the code /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvinfer and /opt/nvidia/deepstream/deepstream/sources/libs/nvdsinfer after DeepStream SDK is installed.

There is also “Gst-Nvinfer source code diagram” in DeepStream SDK FAQ - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums to help the users to understand the source code.

I will suggest you to customize your own object user meta to contain the related ROI information.