Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU)
GPU • DeepStream Version
6 • JetPack Version (valid for Jetson only) • TensorRT Version
8 • NVIDIA GPU Driver Version (valid for GPU only)
510.06 • Issue Type( questions, new requirements, bugs)
question • How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing) • Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)
Hi,
I am trying to build a pipeline with a primary and secondary gie using custom models.
Example:
At the src pad of the primary NvInfer element, custom bounding boxes are attached to the buffer meta using the example described here using the nvds_acquire_obj_meta_from_pool and nvds_add_obj_meta_to_frame`described in NVIDIA DeepStream SDK API Reference 6.0 Release.
This works fine and the metadata is readable downstream - however the second gie doesn’t do any inference on the ROI’s described by the above bounding boxes and the NvDsUserMetaList of each NvDsObjectMeta (which contains the previously mentioned bounding boxes) are not populated as otherwise described in Gst-nvinfer — DeepStream 6.3 Release documentation
The callbacks are non-blocking buffer probes on each nvinfer src pad.
PrimarySrcCallback:
unsafe PadProbeReturn PrimarySrcCallback(Pad pad, PadProbeInfo info)
{
using var buffer = info.Buffer;
var batchMeta = NvDsBatchMeta.FromBuffer(buffer);
var frameMetaList = batchMeta->FrameMetaList;
var frameMeta = (NvDsFrameMeta*)(IntPtr)frameMetaList[0];
var userMetaList = frameMeta->UserMetaList;
var userMeta = (NvDsUserMeta*)(IntPtr)userMetaList[0];
var inferTensorMeta = (NvDsInferTensorMeta*)userMeta->user_meta_data;
var outputLayersInfo = inferTensorMeta->output_layers_info[3];
var data = (*inferTensorMeta)[3];
var detections = NMS(GetDetectionsAboveThreshold(data, 0.05f, 80));
foreach (var (left, top, width, height, confidence, gender, age) in detections)
{
var obj = nvds_acquire_obj_meta_from_pool(batchMeta);
obj->confidence = confidence;
obj->misc_obj_info[0] = gender;
obj->misc_obj_info[1] = age;
obj->detector_bbox_info.org_bbox_coords.left = left;
obj->detector_bbox_info.org_bbox_coords.top = top;
obj->detector_bbox_info.org_bbox_coords.width = width;
obj->detector_bbox_info.org_bbox_coords.height = height;
obj->object_id = 0xFFFFFFFFFFFFFFFF;
obj->unique_component_id = 1;
nvds_add_obj_meta_to_frame(frameMeta, obj, null);
}
return PadProbeReturn.Ok;
}
SecondarySrcCallback:
unsafe PadProbeReturn SecondarySrcCallback(Pad pad, PadProbeInfo info)
{
using var buffer = info.Buffer;
var batchMeta = NvDsBatchMeta.FromBuffer(buffer);
var frameMetaList = batchMeta->FrameMetaList;
var frameMeta = (NvDsFrameMeta*)(IntPtr)frameMetaList[0];
var objMetaList = frameMeta->ObjMetaList;
var userMetaList = frameMeta->UserMetaList;
var userMeta = (NvDsUserMeta*)(IntPtr)userMetaList[0];
for (int i = 0; i < objMetaList.Count; i++)
{
var obj = (NvDsObjectMeta*)(IntPtr)objMetaList[i];
var confidence = obj->confidence;
var gender = obj->misc_obj_info[0];
var age = obj->misc_obj_info[1];
var left = obj->detector_bbox_info.org_bbox_coords.left;
var top = obj->detector_bbox_info.org_bbox_coords.top;
var width = obj->detector_bbox_info.org_bbox_coords.width;
var height = obj->detector_bbox_info.org_bbox_coords.height;
var secondaryRawOutput = obj->Obj_user_meta_list[0]; //This is null!?
}
return PadProbeReturn.Ok;
}
Below is a screenshot from a breakpoint his at the SecondarySrcCallback. As you can see the detection data from the Primary GIE is as expected - however the secondaryRawOutput (which should be populated by the Secondary GIE) is empty:
Hey @Fiona.Chen - any luck digging through the code?
If you could provide me with the requirements for the secondary GIE to pick up the objects from the primary GIE and do inference, it would be great.
I am wondering where in the meta data the gie-unique-id of the GIE is stored - ie. How does the secondary GIE know if an object/classification/segmentation from upstream is from the primary GIE? Is it the unique_component_id of the NvDsObjectMeta?
This is a major road block for us. Could you please consider the above?
It should be possible to “inject” objects into the SGIE by attaching them to meta data upstream. I’ve set the unique_component_id=1 to match the operate-on-gie-id=1 configured on the SGIE but still am unable to retrieve any inference results done by the SGIE.
Since DS is closed source, I am unable to debug this behavior myself.
The c/c++ sample /opt/nvidia/deepstream/deepstream-6.0/sources/apps/sample_apps/deepstream-infer-tensor-meta-test will show you how to use user tensor meta and how to generate correct bbox.
And please make sure you are familiiar with gstreamer before you start with deepstream.
Just as I wrote my last post we managed to get it working.
After digging through the source code of gstnvinfer.cpp at line 1542 should_infer_objectmethod.
/* Function to decide if object should be inferred on. */
static inline gboolean
should_infer_object (GstNvInfer * nvinfer, GstBuffer * inbuf,
NvDsObjectMeta * obj_meta, gulong frame_num,
GstNvInferObjectHistory * history)
{
if (nvinfer->operate_on_gie_id > -1 &&
obj_meta->unique_component_id != nvinfer->operate_on_gie_id)
return FALSE;
if (obj_meta->rect_params.width < nvinfer->min_input_object_width)
return FALSE;
if (obj_meta->rect_params.height < nvinfer->min_input_object_height)
return FALSE;
if (nvinfer->max_input_object_width > 0 &&
obj_meta->rect_params.width > nvinfer->max_input_object_width)
return FALSE;
if (nvinfer->max_input_object_height > 0 &&
obj_meta->rect_params.height > nvinfer->max_input_object_height)
return FALSE;
/* Infer on object if the operate_on_class_ids list is empty or if
* the flag at index class_id is TRUE. */
if (!nvinfer->operate_on_class_ids->empty () &&
((int) nvinfer->operate_on_class_ids->size () <= obj_meta->class_id ||
nvinfer->operate_on_class_ids->at (obj_meta->class_id) == FALSE)) {
return FALSE;
}
if (history && IS_CLASSIFIER_INSTANCE (nvinfer)) {
gboolean should_reinfer = FALSE;
/* Do not reinfer if the object area has not grown by the reinference area
* threshold and reinfer interval criteria is not met. */
if ((history->last_inferred_coords.width *
history->last_inferred_coords.height * (1 +
REINFER_AREA_THRESHOLD)) <
(obj_meta->rect_params.width * obj_meta->rect_params.height))
should_reinfer = TRUE;
if (frame_num - history->last_inferred_frame_num >
nvinfer->secondary_reinfer_interval)
should_reinfer = TRUE;
return should_reinfer;
}
if (history && IS_DETECTOR_INSTANCE (nvinfer)) {
gboolean should_reinfer = FALSE;
if (frame_num - history->last_inferred_frame_num >
nvinfer->secondary_reinfer_interval ||
nvinfer->secondary_reinfer_interval == DEFAULT_REINFER_INTERVAL)
should_reinfer = TRUE;
return should_reinfer;
}
return TRUE;
}
It seems that we have to set the obj_meta->rect_params.height and obj_meta->rect_params.width for the SGIE to pick up the objects. This seems like an oversight, as the rect_paramsis a part of the NvDsOSD while as of DS 5 the obj bounding boxes are stored in obj->detector_bbox_info.org_bbox_coords.left etc… The check should probably be performed on the detector_bbox_info and NOT on the OSD rect_params?
