Could not find output coverage layer for parsing objects

Please read the sample deepstream_python_apps/apps/deepstream-test2 at master · NVIDIA-AI-IOT/deepstream_python_apps (github.com) carefully, it has provided every detail you need.

The parameters are explained in the document too. Gst-nvinfer — DeepStream documentation 6.4 documentation

Yes, my setup is similar to the LPR example. Unfortunately that example doesn’t have custom postprocessing in python so the details I’m looking for are not in that example at all. The deepstream-test2 example is really not relevant here.

I think you misunderstood my question. I’m not asking about gie-unique-id parameter at all, I’m asking about scaling the bounding boxes. Using the LPR app as an example: the car detector takes the entire image as its input, then the license plate detector takes the car crop as its input. Since the input to each model is resized, part of the postprocessing involves scaling the bounding boxes back to the original video size. In order to scale the bounding boxes of the license plate detector, you need to know the size of the car bounding box. This is what I’m asking about. How do you find that crop size?

If your models’ relationship is similar to the LPR sample, you may know that the first SGIE(LPD) output tensor is attached to the PGIE’s NvDsObjectMeta — Deepstream Deepstream Version: 6.4 documentation but not NvDsFrameMeta — Deepstream Deepstream Version: 6.4 documentation. So you can get the bbox info from the NvDsObjectMeta.

In addition, since you want to use “output-tensor-meta” customization for your SGIE. c/c++ is recommended since it is hard to convert c/c++ pointer to python language. And for the SGIE output object meta, the upper level bboxes coordinates and resolution is available in objects meta NvDsObjectMeta — Deepstream Deepstream Version: 6.4 documentation. The “parent” is the object meta of the upper level object. So please make sure the third parameter of pyds.nvds_add_obj_meta_to_frame(frame_meta, obj_meta, parent_obj) should not be none for the SGIE.

Please refer to the nvinfer SGIE tensor meta attachment code. The attach_metadata_detector() function in /opt/nvidia/deepstream/deepstrea/sources/gst-plugins/gst-nvinfer/gstnvinfer_meta_utils.cpp

Again, I’m doing this entirely in python.

FYI the search function in your documentation appears to be totally broken as I’ve tried several browsers and it hangs indefinitely. I also tried recursively printing dir for these objects, but I can’t find how I would get NvDsObjectMeta from what I have access to in the second probe function. It makes sense that it would be stored there, but my question remains how do I access that from the second probe function?

For example, copying from the demo apps (where info is the second input arg to the probe function) I can get the batch metadata as follows:

gst_buffer = info.get_buffer()
batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))

Is the object metadata accessible via some property or method of the batch metadata, or is it entirely separate? In either case, precisely how do I access it and is the ordering of this metadata identical to the order of the crops passed to the second model (or is there an id property I need to use instead to align them)? If you could provide a concise example, that would be greatly appreciated.

This is the code to get NvDsFrameMeta: deepstream_python_apps/apps/deepstream-ssd-parser/deepstream_ssd_parser.py at master · NVIDIA-AI-IOT/deepstream_python_apps (github.com)

I’ve added some code in deepstream-test2 to get NvDsObjectMeta — Deepstream Deepstream Version: 6.4 documentation.
deepstream-test2-new.zip (12.6 KB)

It is in the sgie1_src_pad_buffer_probe() function.

batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
    l_frame = batch_meta.frame_meta_list
    while l_frame is not None:
        try:
            # Note that l_frame.data needs a cast to pyds.NvDsFrameMeta
            # The casting is done by pyds.NvDsFrameMeta.cast()
            # The casting also keeps ownership of the underlying memory
            # in the C code, so the Python garbage collector will leave
            # it alone.
            frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
        except StopIteration:
            break
        frame_number=frame_meta.frame_num
        num_rects = frame_meta.num_obj_meta
        l_obj=frame_meta.obj_meta_list
        while l_obj is not None:
            try:
                # Casting l_obj.data to pyds.NvDsObjectMeta
                obj_meta=pyds.NvDsObjectMeta.cast(l_obj.data)
            except StopIteration:
                break
            print("obj.unique_component_id=",obj_meta.unique_component_id)
            if obj_meta.unique_component_id==2:
                print("obj parent=", obj_meta.parent)
                meta_subobj=pyds.NvDsObjectMeta.cast(obj_meta.parent)
                print("parent obj unique_component_id=", meta_subobj.unique_component_id)
            if obj_meta.unique_component_id==1:
                l_user = obj_meta.obj_user_meta_list
                while l_user is not None:
                    try:
                    # Note that l_user.data needs a cast to pyds.NvDsUserMeta
                    # The casting also keeps ownership of the underlying memory
                    # in the C code, so the Python garbage collector will leave
                    # it alone.
                        user_meta = pyds.NvDsUserMeta.cast(l_user.data)
                    except StopIteration:
                        break

                    if (
                         user_meta.base_meta.meta_type
                        != pyds.NvDsMetaType.NVDSINFER_TENSOR_OUTPUT_META
                    ):
                        continue
                    print("obj has tensor output\n")
                    try:
                        l_user = l_user.next
                    except StopIteration:
                        break
            l_obj=l_obj.next
        try:
            l_frame=l_frame.next
        except StopIteration:
            break

Thanks for posting code. Unfortunately that still doesn’t answer my question regarding accessing the crop size and ensuring that’s aligned with the outputs of the second model so that they can be scaled appropriately. There are a few issues with adapting this code snippet:

  1. If I set l_user to obj_meta.obj_user_meta_list as you’ve done, I get None. I was defining l_user as frame_meta.frame_user_meta_list to get the outputs and convert them to numpy arrays. Is this a critical difference, and if so, how do I get the outputs there instead of None?
  2. Object parent is always None and unique_component_id is always 0. I’m not sure if this actually matters, as the question remains how to get the crop size for a given detection from the second model so that it can be scaled.
  3. If I get the crop size from obj_meta.rect_params (with obj_meta defined the same as you have it), the lengths of l_user (defined as frame_meta.frame_user_meta_list, as stated above) and l_object do not match, so while I can access the crop sizes from the first model, I still have no way of assigning them to a given output of the second model to scale the bounding box predictions. The length of l_user is always 1, and the length of l_object varies with the number of detections from the initial detector.

@carlosgs there’s been no reply for a few days now. Is there anyone else that could provide assistance here? It’d be greatly appreciated

Can you please answer the question in the post Could not find output coverage layer for parsing objects - #21 by Fiona.Chen? Can you describe the relationship between the models you use?

How did you add the object meta in your customized postprocessing code?

Can you show us how did you add the object meta?

Like I said previously, it’s similar to the LPR example - i.e. the first model detects an object and the second model detects a small part (or parts) of that object. Each model can detect multiple instances per input. If that doesn’t answer your question, could you please state specifically what information you’re looking for?

Here is the code. It is based on deepstream_python_apps/apps/deepstream-ssd-parser/deepstream_ssd_parser.py at master · NVIDIA-AI-IOT/deepstream_python_apps · GitHub

Just as in the SSD Parser example, the det_object is a NvDsInferObjectDetectionInfo object, the batch meta is obtained in the probe function with pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer)) and frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data) with l_frame = batch_meta.frame_meta_list

def add_object_meta_to_frame(det_object, batch_meta, frame_meta):
    """ Inserts an object into the metadata """
    obj_meta = pyds.nvds_acquire_obj_meta_from_pool(batch_meta)
    # Set bbox properties. These are in video resolution.
    obj_meta.rect_params.left = det_object.left
    obj_meta.rect_params.top = det_object.top
    obj_meta.rect_params.width = det_object.width
    obj_meta.rect_params.height = det_object.height

    # Set object info including class, detection confidence, etc.
    obj_meta.confidence = det_object.detectionConfidence
    obj_meta.class_id = det_object.classId
    obj_meta.obj_label = 'object_class'

    # Insert the object into current frame meta
    # This object has no parent
    pyds.nvds_add_obj_meta_to_frame(frame_meta, obj_meta, None)

I modified the code to simulate your case. In the SGIE probe function sgie1_src_pad_buffer_probe(), the object meta detected by PGIE can be got by “if obj_meta.unique_component_id==1:” checking. Then you can get the bbox from this object meta, you can also got the SGIE output tensor by the code

            if obj_meta.unique_component_id==1:
                l_user = obj_meta.obj_user_meta_list
                while l_user is not None:
                    try:
                    # Note that l_user.data needs a cast to pyds.NvDsUserMeta
                    # The casting also keeps ownership of the underlying memory
                    # in the C code, so the Python garbage collector will leave
                    # it alone.
                        user_meta = pyds.NvDsUserMeta.cast(l_user.data)
                    except StopIteration:
                        break

                    if (
                         user_meta.base_meta.meta_type
                        != pyds.NvDsMetaType.NVDSINFER_TENSOR_OUTPUT_META
                    ):
                        continue
                    print("obj has tensor output\n")
                    try:
                        l_user = l_user.next
                    except StopIteration:
                        break

Then you can do your postprocessing and the bbox from PGIE is also here.

Please read the code carefully. The SGIE postprocessing should be implemented by yourself, please make sure the unique_component_id in (NVIDIA DeepStream SDK API Reference: _NvDsObjectMeta Struct Reference | NVIDIA Docs) should be set correctly for the downstream components to identify the object meta.
deepstream-test2-new.zip (18.0 KB)

Thanks, Fiona. It is working after setting the unique_component_id and fixing another issue