Secondary GIE with custom bounding boxes

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
GPU
• DeepStream Version
6
• JetPack Version (valid for Jetson only)
• TensorRT Version
8
• NVIDIA GPU Driver Version (valid for GPU only)
510.06
• Issue Type( questions, new requirements, bugs)
question
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Hi,

I am trying to build a pipeline with a primary and secondary gie using custom models.
Example:

gst-launch-1.0 uridecodebin uri=“file://test.mp4” ! nvvideoconvert ! “video/x-raw(memory:NVMM), format=NV12, width=640, height=640” ! m.sink_0 nvstreammux name=“m” batch-size=1 ! nvinfer config-file-path=“primary.txt” ! nvinfer config-file-path=“secondary.txt” ! fakesink

At the src pad of the primary NvInfer element, custom bounding boxes are attached to the buffer meta using the example described here using the nvds_acquire_obj_meta_from_pool and nvds_add_obj_meta_to_frame`described in NVIDIA DeepStream SDK API Reference 6.0 Release.

Example:

    var obj = nvds_acquire_obj_meta_from_pool(batchMeta);
    obj->class_id = 0;
    obj->confidence = 0;
    obj->detector_bbox_info.org_bbox_coords.left = 0;
    obj->detector_bbox_info.org_bbox_coords.top = 0;
    obj->detector_bbox_info.org_bbox_coords.width = 128;
    obj->detector_bbox_info.org_bbox_coords.height = 256;
    obj->object_id = 0xFFFFFFFFFFFFFFFF;
    obj->unique_component_id = 1;
    nvds_add_obj_meta_to_frame(frameMeta, obj, null);

This works fine and the metadata is readable downstream - however the second gie doesn’t do any inference on the ROI’s described by the above bounding boxes and the NvDsUserMetaList of each NvDsObjectMeta (which contains the previously mentioned bounding boxes) are not populated as otherwise described in Gst-nvinfer — DeepStream 6.0 Release documentation

Config files:

[property]
gpu-id=0
net-scale-factor=1
model-color-format=0
model-engine-file=primary.onnx_b1_gpu0_fp16.engine
batch-size=1
network-mode=2
interval=0
gie-unique-id=1
process-mode=1
network-type=100
output-tensor-meta=1

[property]
gpu-id=0
net-scale-factor=1
model-color-format=0
model-engine-file=secondary.onnx_b1_gpu0_fp32.engine
batch-size=1
network-mode=0
interval=0
gie-unique-id=2
process-mode=2
network-type=100
output-tensor-meta=1
input-object-min-width=0
input-object-min-height=0
input-object-max-width=0
input-object-max-height=0
operate-on-gie-id=1

Any help would be greatly appreciated!

/M

Can you share your code?

Hi @Fiona.Chen ,

The application is written in C# interfacing with DS through P/Invoke.

The pipeline is simple and looks like below. The config files from my previous post.

The callbacks are non-blocking buffer probes on each nvinfer src pad.

PrimarySrcCallback:


unsafe PadProbeReturn PrimarySrcCallback(Pad pad, PadProbeInfo info)
{
    using var buffer = info.Buffer;

    var batchMeta = NvDsBatchMeta.FromBuffer(buffer);
    var frameMetaList = batchMeta->FrameMetaList;
    var frameMeta = (NvDsFrameMeta*)(IntPtr)frameMetaList[0];
    var userMetaList = frameMeta->UserMetaList;
    var userMeta = (NvDsUserMeta*)(IntPtr)userMetaList[0];
    var inferTensorMeta = (NvDsInferTensorMeta*)userMeta->user_meta_data;
    var outputLayersInfo = inferTensorMeta->output_layers_info[3];
    var data = (*inferTensorMeta)[3];

    var detections = NMS(GetDetectionsAboveThreshold(data, 0.05f, 80));

    foreach (var (left, top, width, height, confidence, gender, age) in detections)
    {
        var obj = nvds_acquire_obj_meta_from_pool(batchMeta);
        obj->confidence = confidence;
        obj->misc_obj_info[0] = gender;
        obj->misc_obj_info[1] = age;
        obj->detector_bbox_info.org_bbox_coords.left = left;
        obj->detector_bbox_info.org_bbox_coords.top = top;
        obj->detector_bbox_info.org_bbox_coords.width = width;
        obj->detector_bbox_info.org_bbox_coords.height = height;
        obj->object_id = 0xFFFFFFFFFFFFFFFF;
        obj->unique_component_id = 1;
        nvds_add_obj_meta_to_frame(frameMeta, obj, null);
    }

    return PadProbeReturn.Ok;
}

SecondarySrcCallback:

unsafe PadProbeReturn SecondarySrcCallback(Pad pad, PadProbeInfo info)
{
    using var buffer = info.Buffer;

    var batchMeta = NvDsBatchMeta.FromBuffer(buffer);
    var frameMetaList = batchMeta->FrameMetaList;
    var frameMeta = (NvDsFrameMeta*)(IntPtr)frameMetaList[0];
    var objMetaList = frameMeta->ObjMetaList;
    var userMetaList = frameMeta->UserMetaList;
    var userMeta = (NvDsUserMeta*)(IntPtr)userMetaList[0];

    for (int i = 0; i < objMetaList.Count; i++)
    {
        var obj = (NvDsObjectMeta*)(IntPtr)objMetaList[i];

        var confidence = obj->confidence;
        var gender = obj->misc_obj_info[0];
        var age = obj->misc_obj_info[1];
        var left = obj->detector_bbox_info.org_bbox_coords.left;
        var top = obj->detector_bbox_info.org_bbox_coords.top;
        var width = obj->detector_bbox_info.org_bbox_coords.width;
        var height = obj->detector_bbox_info.org_bbox_coords.height;

        var secondaryRawOutput = obj->Obj_user_meta_list[0]; //This is null!?
    }

    return PadProbeReturn.Ok;
}

Below is a screenshot from a breakpoint his at the SecondarySrcCallback. As you can see the detection data from the Primary GIE is as expected - however the secondaryRawOutput (which should be populated by the Secondary GIE) is empty:

Perhaps you can check which fields has to be set in the NvDsObjectMeta in order for the secondary GIE to pick it up and do inference?

Thanks in advance,

/M

Hey @Fiona.Chen - any luck digging through the code?

If you could provide me with the requirements for the secondary GIE to pick up the objects from the primary GIE and do inference, it would be great.

I am wondering where in the meta data the gie-unique-id of the GIE is stored - ie. How does the secondary GIE know if an object/classification/segmentation from upstream is from the primary GIE? Is it the unique_component_id of the NvDsObjectMeta?

Thanks!