Nvstreamdemux does not copy obj_meta parent structure to src pad

Please provide complete information as applicable to your setup.

• Hardware Platform: GPU
• DeepStream Version: 6.3-gc-triton-devel
• TensorRT Version: 8.5.3.1
• NVIDIA GPU Driver Version: 530.41.03
• Issue Type: Bug
• How to reproduce the issue? See below

Description

We have a pipeline with one primary and one secondary nvinfer element. The seconday operates on detections produced by the primary.

On nvstreamdemux sink_pad the obj_meta->parent is set to actual parent pointer value of any objects detected by the primary infer engine.

However, on nvstreamdemux src_pad the obj_meta->parent is nullptr even though the primary and the secondary engine have produced detections.

It seems that the (deep) copying of the obj_meta->parent structure between sink_pad and src_pad of nvstreamdemux fails.

Test case

Run test in NVIDIA deepstream container:

docker run -it -nvcr.io/nvidia/deepstream:6.3-gc-triton-devel

Repro

Build and run attached test program. There is an assertion that points out the bug in a padProbe on nvstreamdemux src pad.

g++ test.cpp -o test `pkg-config --cflags --libs gstreamer-1.0` -I/opt/nvidia/deepstream/deepstream/sources/includes -L/opt/nvidia/deepstream/deepstream/lib -lnvdsgst_meta 

There is an attached nvinfer config file for the secondary inference engine.

Test program

#include <gst/gst.h>

#include <gstnvdsmeta.h>

#include <cassert>
#include <iostream>

// clang-format off
// export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/usr/local/lib:/opt/nvidia/deepstream/deepstream/lib:/opt/nvidia/deepstream/deepstream/lib/cvcore_libs"
// g++ test.cpp -o test `pkg-config --cflags --libs gstreamer-1.0` -I/opt/nvidia/deepstream/deepstream/sources/includes -L/opt/nvidia/deepstream/deepstream/lib -lnvdsgst_meta
// clang-format on

GstPadProbeReturn padProbe(GstPad* pad, GstPadProbeInfo* info, void*) {
  std::string direction{};
  assert(gst_pad_get_direction(pad));
  if (gst_pad_get_direction(pad) == GST_PAD_SINK) {
    direction = "SINK";
  } else {
    direction = "SRC";
  }
  std::cout << "padProbe " << direction << " START" << std::endl;

  assert(info->type & GST_PAD_PROBE_TYPE_BUFFER);
  auto buffer = gst_pad_probe_info_get_buffer(info);
  assert(buffer);
  assert(gst_buffer_get_nvds_meta(buffer)->meta_type == NVDS_BATCH_GST_META);
  NvDsBatchMeta* batch_meta = gst_buffer_get_nvds_batch_meta(buffer);
  assert(batch_meta);

  for (auto frame_meta = batch_meta->frame_meta_list; frame_meta; frame_meta = frame_meta->next) {
    auto frame_meta_c = reinterpret_cast<NvDsFrameMeta*>(frame_meta->data);
    std::cout << "Frame meta timestamp: " << frame_meta_c->ntp_timestamp << std::endl;

    for (auto obj_meta = frame_meta_c->obj_meta_list; obj_meta; obj_meta = obj_meta->next) {
      auto obj_meta_c = reinterpret_cast<NvDsObjectMeta*>(obj_meta->data);
      std::cout << "Obj meta gie-unique-id: " << obj_meta_c->unique_component_id
                << ", Parent detection ptr: " << obj_meta_c->parent << std::endl;
      if (obj_meta_c->unique_component_id == 2) {
        // The demuxer has no obj_meta->parent set for secondary detection on outgoing src pad!!!
        assert(obj_meta_c->parent);
      }
    }
  }

  std::cout << "padProbe " << direction << " DONE" << std::endl;
  return GST_PAD_PROBE_PASS;
}

int main(int argc, char* argv[]) {
  GstElement* pipeline;
  GstBus* bus;
  GstMessage* msg;

  /* Initialize GStreamer */
  gst_init(&argc, &argv);
  /* Build the pipeline */
  // clang-format off
  pipeline = gst_parse_launch(
      "uridecodebin "
      "uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 !"
      "mux.sink_0 nvstreammux name=mux !"
      "nvinfer config-file-path=\"/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt\" !"
      "nvinfer config-file-path=\"config_infer_secondary_facenet.txt\" !"
      "nvstreamdemux name=demux demux.src_0 ! fakesink",
      NULL);
  // clang-format on

  // Add pad probes on nvstreamdemux sink and src pads
  GstElement* demux = gst_bin_get_by_name(GST_BIN(pipeline), "demux");
  GstPad* sink_pad = gst_element_get_static_pad(demux, "sink");
  gst_pad_add_probe(sink_pad,
                    GstPadProbeType(GST_PAD_PROBE_TYPE_BLOCK | GST_PAD_PROBE_TYPE_BUFFER),
                    padProbe,
                    NULL,
                    NULL);
  GstPad* src_pad = gst_element_get_static_pad(demux, "src_0");
  gst_pad_add_probe(src_pad,
                    GstPadProbeType(GST_PAD_PROBE_TYPE_BLOCK | GST_PAD_PROBE_TYPE_BUFFER),
                    padProbe,
                    NULL,
                    NULL);

  /* Start playing */
  gst_element_set_state(pipeline, GST_STATE_PLAYING);
  GST_DEBUG_BIN_TO_DOT_FILE_WITH_TS(GST_BIN(pipeline), GST_DEBUG_GRAPH_SHOW_ALL, "pipeline-test");

  /* Wait until error or EOS */
  bus = gst_element_get_bus(pipeline);
  msg = gst_bus_timed_pop_filtered(
      bus, GST_CLOCK_TIME_NONE, GstMessageType(GST_MESSAGE_ERROR | GST_MESSAGE_EOS));

  /* See next tutorial for proper error message handling/parsing */
  if (GST_MESSAGE_TYPE(msg) == GST_MESSAGE_ERROR) {
    g_error(
        "An error occurred! Re-run with the GST_DEBUG=*:WARN environment "
        "variable set for more details.");
  }

  /* Free resources */
  gst_message_unref(msg);
  gst_object_unref(bus);
  gst_element_set_state(pipeline, GST_STATE_NULL);
  gst_object_unref(pipeline);
  return 0;
}

Updated post

Run command:

USE_NEW_NVSTREAMMUX=yes LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/usr/local/lib:/opt/nvidia/deepstream/deepstream/lib:/opt/nvidia/deepstream/deepstream/lib/cvcore_libs" ./test

nvinfer config secondary network

[property]
gie-unique-id=2
process-mode=2
operate-on-gie-id=1
net-scale-factor=0.0039215697906911373
tlt-model-key=nvidia_tlt
tlt-encoded-model=facenet.etlt
labelfile-path=facenet.labels
int8-calib-file=int8_calibration.txt
model-engine-file=facenet.etlt_b1_gpu0_int8.engine
infer-dims=3;416;736
uff-input-order=0
uff-input-blob-name=input_1
batch-size=1
model-color-format=0
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=1
num-detected-classes=1
interval=0
output-blob-names=output_bbox/BiasAdd;output_cov/Sigmoid

[class-attrs-all]
pre-cluster-threshold=0.2
group-threshold=1
## Set eps=0.7 and minBoxes for cluster-mode=1(DBSCAN)
eps=0.2
#minBoxes=3

Why don’t you get the parent before nvstreamdemux?

The example above is just a minimum repro of the bug.

In our application we have a more extensive pipeline with multiple sources and sinks (before nvstreammux and after nvstreamdemux).
After the batch has been processed by nvinfer elements we want to demux the batch and operate on each stream individually,
Further down our pipeline we have elements which operate on each indivudal stream, and those elements are dependent on that obj_meta->parent is set for detections produced by any secondary infer engine operating on a prior one.

We have been able to make a work around by settings the parent_id in object_meta.misc_obj_info[0] field, but the bug in nvstreamdemux remains and should be solved. I will attach our work-around as well.

GstPadProbeReturn padProbe(GstPadProbeInfo* info) {
  if (info->type & GST_PAD_PROBE_TYPE_BUFFER) {
    // TODO: This is a workaround which fixes the bug reported in xxxx. nvstreamdemux seems to
    // not do a deep copy of the secondary detection's parent object structure, which results in
    // missing information on outgoing source pad. Instead set the parent id in the NvDsObjectMeta
    // misc_obj_info structure which is copied to the src pad of the nvstreamdemux. In
    // xxx (an element downstream) this field must be parsed to retreive the object parent id.
    // Remove this and pad probe of buffer above when bug is corrected in Deepstream!
    auto buffer = gst_pad_probe_info_get_buffer(info);
    NvDsBatchMeta* batch_meta = gst_buffer_get_nvds_batch_meta(buffer);
    glib::List<NvDsFrameMeta> frame_meta_list{batch_meta->frame_meta_list};
    for (auto& frame_meta : frame_meta_list) {
      glib::List<NvDsObjectMeta> obj_meta_list{frame_meta.obj_meta_list};
      for (auto& obj_meta : obj_meta_list) {
        // If this is a secondary detection, set its parent id in misc obj info, otherwise set it to
        // undefined.
        obj_meta.misc_obj_info[0] = obj_meta.parent ? obj_meta.parent->object_id : guint64(-1);
      }
    }
}

@Fiona.Chen
Do you have any news regarding this issue?

@Fiona.Chen It would be very good if we could get an update on this issue. The work around in the post above works well as long as the parent object is tracked, but if no tracker is used (or the tracker loses track of the object for any reason) a more complex work around is needed.

The nvstreamdemux calls nvds_copy_frame_meta to copy the meta for each frame from the original batch. That function is documented as making a deep copy, but the relationship between parent and child detections gets lost in the copy. The only correction needed to make this work is probably in nvds_copy_frame_meta. That function is however not open source, so we cannot make that correction ourselves.

For now we will need to make a work around in the application, but it would be very good for us to know that this issue will be solved so we can remove the work around when a new version of DeepStream is released.

The NvDsObjectMeta is a part of NvDsBatchMeta, it is based on batch. It is enough to get the “parent” before nvstreamdemux. Can you tell us what is the special reason for getting the information after nvstreamdemux?

@Fiona.Chen
We have the knowledge about the NvDsBatchMeta structure, and we see that Nvstreamdemux does not do deep-copy of the NvDsObjectMeta parent structure. I have attached an example in this topic which proves the bug.

We handle a lot of streams simultaneously as a batch by feeding them to Nvstreammux and run the batch through Nvinfer elements. After that we split the batch in Nvstreamdemux to handle each stream separately.

For each stream we use the NvDsBatchMeta structure to retrieve the Nvinfer-information to do post-processing of any detected objects by the inference engines.

In this post-process step we need to connect any child-detection to its parent-detection due to our application requirements. However, with the bug reported in this Topic it is not possible.

Luckily we have found a work-around, but we think this is an obvious bug which must be corrected at source (your side). Unfortunately the code is not available from your side so we cannot do a patch either.

1 Like

@Fiona.Chen
Any news on this one?

There is no update from you for a period, assuming this is not an issue anymore.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

The nvstreamdemux will separate every frame from the batch, the batch does not exist after nvstreamdemux. The best way is to get the batch meta before nvstreamdemux.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.