Re-ID vector embedding is incorrect when using new streammuxer

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU): RTX 4080
• DeepStream Version: 6.3, 7.0
• TensorRT Version: 8.5.3.1 for DS 6.3, 8.6.1.6 for DS 7.0
• NVIDIA GPU Driver Version (valid for GPU only): 535.171.04
• Issue Type( questions, new requirements, bugs): Bug
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

I’m currently using DeepStream Python.

I slightly modified the DeepStream Python bindings to access Re-ID vector embedding.

When I used original streammuxer, the Re-ID vector embedding was fine.

However, using the new streammuxer outputs a different Re-ID vector embedding than the original streammuxer.

For comparison, I dumped the images and found that very different images output similar Re-ID vector embeddings.

This may be a bug, which sample are you using?

Or can you share sample code that reproduces the problem?

Unfortunately, I cannot provide full source code.

However my implementation quite similar to below.

# uridecodebin -  nvstreammux - nvinfer (person detector) - nvtracker - nvvideoconvert - caps - nvosd - nveglglessink

def caps_src_pad_buffer_probe(unused_pad: Gst.Pad, info: Gst.PadProbeInfo, user_data: Any) -> Gst.PadProbeReturn:
    gst_buffer = info.get_buffer()
    if not gst_buffer:
        logger.error("Unable to get Gst.Buffer ")
        return

    batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))

    l_frame = batch_meta.frame_meta_list
    while l_frame is not None:
        try:
            frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
        except StopIteration:
            break

        arr = pyds.get_nvds_buf_surface(hash(gst_buffer), frame_meta.batch_id)
        rgba = np.array(arr, copy=True, order="C")
        rgb = cv2.cvtColor(rgba, cv2.COLOR_RGBA2RGB)

        l_obj = frame_meta.obj_meta_list
        while l_obj is not None:
            try:

                obj_meta = pyds.NvDsObjectMeta.cast(l_obj.data)
            except StopIteration:
                break

            if obj_meta.object_id != UNTRACKED_OBJECT_ID:
                user_meta, feature = None, None
                l_user_meta = obj_meta.obj_user_meta_list
                while l_user_meta is not None:
                    try:
                        user_meta = pyds.NvDsUserMeta.cast(l_user_meta.data)
                    except StopIteration:
                        break

                    if user_meta.base_meta.meta_type == pyds.NvDsMetaType.NVDS_TRACKER_OBJ_REID_META:
                        # custom python binding for NvDsObjReid
                        objReid = pyds.NvDsObjReid.cast(user_meta.user_meta_data)
                        feature = objReid.get_feature().copy()
                        # dump ReID vector embedding

                    try:
                        l_user_meta = l_user_meta.next
                    except StopIteration:
                        break

                if feature is not None:
                    rect_params = obj_meta.rect_params
                    box = np.asarray([rect_params.left, rect_params.top, rect_params.width, rect_params.height])
                    box[2:] += box[:2]

                    x1, y1, x2, y2 = box.astype(int)
                    image = rgb[y1 : y2 + 1, x1 : x2 + 1, :].copy()

                    # save above image
               
            try:
                l_obj = l_obj.next
            except StopIteration:
                break

        try:
            l_frame = l_frame.next
        except StopIteration:
            break

    return Gst.PadProbeReturn.OK

Did you add the bindings of pyds.NvDsObjReid by yourself? I used deepstream-app for testing.

Modify the source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt as below.

--- a/samples/configs/deepstream-app/source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt
+++ b/samples/configs/deepstream-app/source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt
@@ -14,6 +14,8 @@
 enable-perf-measurement=1
 perf-measurement-interval-sec=5
 #gie-kitti-output-dir=streamscl
+reid-track-output-dir=reid-new
+
 
 [tiled-display]
 enable=1
@@ -45,7 +47,7 @@ cudadec-memtype=0
 [sink0]
 enable=1
 #Type - 1=FakeSink 2=EglSink/nv3dsink (Jetson only) 3=File
-type=2
+type=1
 sync=1
 source-id=0
 gpu-id=0
@@ -154,8 +156,8 @@ ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.s
 # ll-config-file required to set different tracker types
 # ll-config-file=config_tracker_IOU.yml
 # ll-config-file=config_tracker_NvSORT.yml
-ll-config-file=config_tracker_NvDCF_perf.yml
-# ll-config-file=config_tracker_NvDCF_accuracy.yml
+# ll-config-file=config_tracker_NvDCF_perf.yml
+ll-config-file=config_tracker_NvDCF_accuracy.yml
 # ll-config-file=config_tracker_NvDeepSORT.yml
 gpu-id=0
 display-tracking-id=1

Then run the following command line to dump Reid.

USE_NEW_NVSTREAMMUX=yes ./deepstream-app -c /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt 

Except for the difference in object id, Re-ID vector embeddings are similar

You also refer to this link.
https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvtracker.html#re-id-feature-output

Yes, I added the code below to the " bindtrackermeta.cpp

        py::class_<NvDsReidTensorBatch>(m, "NvDsReidTensorBatch",
                                          pydsdoc::trackerdoc::NvDsReidTensorBatchDoc::descr)
                .def(py::init<>())
                .def_readwrite("featureSize",
                               &NvDsReidTensorBatch::featureSize)
                .def_readwrite("numFilled", &NvDsReidTensorBatch::numFilled)
                .def_readonly("ptr_host", &NvDsReidTensorBatch::ptr_host)
                .def_readonly("ptr_dev", &NvDsReidTensorBatch::ptr_dev)
                .def_readonly("priv_data", &NvDsReidTensorBatch::priv_data)

                .def("get_features", [](NvDsReidTensorBatch &self) -> py::array {
                        auto dtype = py::dtype(py::format_descriptor<float>::format());
                        return py::array(dtype,
                                         {self.numFilled, self.featureSize},
                                         {sizeof(float) * self.featureSize, sizeof(float)},
                                         self.ptr_host);

                    },
                     py::return_value_policy::reference,
                     pydsdoc::trackerdoc::NvDsReidTensorBatchDoc::get_features)

                .def("cast",
                     [](void *data) {
                         return (NvDsReidTensorBatch *) data;
                     },
                     py::return_value_policy::reference,
                     pydsdoc::trackerdoc::NvDsReidTensorBatchDoc::cast);

I think ReID tensor is fine, but the problem is reid index.

In “deepstream_app.c”, reidInd can be obtained by dereferencing user_meta_data as follows.

gint reidInd = *((int32_t *) (user_meta->user_meta_data));

In python, I got reidInd like

reidInd = ctypes.cast(pyds.get_ptr(user_meta.user_meta_data), ctypes.POINTER(ctypes.c_int32).contents.value

When I set USE_NEW_NVSTREAMMUX=yes, the ReID index is not unique in the batchmeta.

# [(Object ID, ReID index), ...] NvDsReidTensorBatch.numFilled
[(13, 7), (12, 6), (9, 4), (11, 5), (5, 2), (7, 3), (2, 1), (0, 0), (23, 7), (22, 6), (20, 4), (19, 3), (21, 5), (18, 2), (17, 1), (16, 0)] 8
[(24, 0)] 1
[(28, 3), (27, 2), (26, 1), (25, 0)] 4
[(30, 0)] 1
[(31, 0), (32, 0)] 1
[(24, 0), (30, 0)] 1
[(33, 0)] 1
[(34, 0)] 1
[(24, 0), (0, 0)] 1
[(30, 0), (28, 1), (27, 0)] 3
[(29, 0)] 1
[(35, 0)] 1
[(15, 9), (14, 8), (13, 7), (9, 4), (5, 2), (11, 5), (12, 6), (7, 3), (2, 1), (0, 0), (8, 14), (10, 15), (6, 13), (4, 12), (1, 10), (3, 11)] 16
[(33, 0), (36, 0), (16, 1), (24, 1)] 2
[(35, 0), (36, 0), (37, 1)] 2
[(8, 4), (10, 5), (6, 3), (4, 2), (1, 0), (3, 1), (23, 7), (22, 6), (17, 1), (19, 3), (20, 4), (18, 2), (16, 0), (21, 5)] 8
[(27, 0), (28, 1), (39, 2), (5, 2), (2, 1), (0, 0)] 3
[(30, 0), (29, 0)] 1
[(37, 0)] 1
[(35, 0)] 1
[(32, 0), (39, 0)] 1
[(19, 3), (23, 7), (22, 6), (17, 1), (20, 4), (21, 5), (18, 2), (16, 0), (15, 9), (14, 8), (13, 7), (9, 4), (12, 6), (11, 5), (5, 2), (7, 3), (2, 1), (0, 0)] 10
[(1, 0), (33, 0)] 1
[(25, 0), (30, 0)] 1
[(8, 4), (10, 5), (6, 3), (4, 2), (3, 1), (1, 0)] 6
[(0, 0), (24, 0)] 1
[(37, 1), (29, 0)] 2
[(36, 0), (39, 0)] 1
[(8, 4), (10, 5), (6, 3), (4, 2), (1, 0), (3, 1), (22, 6), (23, 7), (17, 1), (19, 3), (20, 4), (21, 5), (18, 2), (16, 0)] 8
[(35, 0), (36, 0)] 1
[(1, 0), (33, 0)] 1
[(5, 2), (0, 0), (2, 1), (28, 2), (27, 1), (25, 0)] 3
[(37, 0)] 1
[(35, 0)] 1
[(36, 0), (39, 6), (19, 10), (22, 13), (23, 14), (20, 11), (17, 8), (21, 12), (18, 9), (16, 7), (8, 4), (10, 5), (6, 3), (4, 2), (3, 1), (1, 0)] 15
[(15, 8), (14, 7), (12, 6), (9, 4), (11, 5), (7, 3), (5, 2), (0, 0), (2, 1)] 9
[(43, 0), (44, 0)] 1
[(47, 0)] 1
[(35, 0)] 1
[(36, 0)] 1
[(10, 4), (6, 2), (8, 3), (4, 1), (3, 0), (22, 5), (19, 2), (23, 6), (21, 4), (20, 3), (18, 1), (17, 0)] 7
[(15, 9), (12, 6), (14, 8), (9, 4), (13, 7), (11, 5), (5, 2), (7, 3), (0, 0), (2, 1), (48, 0)] 10
[(29, 0), (45, 1)] 2
[(37, 0)] 1
[(35, 0)] 1
[(36, 0), (19, 2), (23, 6), (22, 5), (20, 3), (18, 1), (21, 4), (17, 0)] 7
[(8, 3), (10, 4), (6, 2), (4, 1), (3, 0)] 5
[(15, 9), (14, 8), (12, 6), (7, 3), (9, 4), (11, 5), (5, 2), (2, 1), (13, 7), (0, 0), (33, 0)] 10
[(30, 0), (48, 0)] 1
[(29, 0), (45, 1)] 2
[(0, 0), (2, 1), (24, 1), (48, 0)] 2
[(50, 2), (37, 3), (41, 4), (29, 0), (45, 1)] 5
[(36, 0)] 1
[(8, 4), (10, 5), (6, 3), (4, 2), (3, 1), (1, 0)] 6
[(17, 0), (24, 0)] 1
[(23, 6), (19, 2), (18, 1), (22, 5), (20, 3), (21, 4), (17, 0)] 7
[(15, 9), (14, 8), (12, 6), (0, 0), (7, 3), (5, 2), (2, 1), (11, 5), (9, 4), (13, 7), (10, 4), (8, 3), (6, 2), (4, 1), (3, 0), (51, 5), (52, 10), (33, 11)] 12

1.This is completely different from the question you mentioned above

These are two different levels of Re-ID vectors
For NvDsReidTensorBatch

NvDsBatchMeta
    -> batch_user_meta_list (find type == NVDS_TRACKER_BATCH_REID_META)

For NVDS_TRACKER_OBJ_REID_META

NvDsBatchMeta
   --> NvFrameMeta
     --> NvObjectMeta
          -> obj_user_meta_list (find type == NVDS_TRACKER_OBJ_REID_META)

This is not reidInd, it is a meaningless value.

Sorry for confusion.

I tested both of DS 6.3 and DS 7.0, but the situation is quite similar.
In DS 6.3, according to deepstream_app.c, I can get ReID embedding for each object as follows.

static void
write_reid_track_output (AppCtx * appCtx, NvDsBatchMeta * batch_meta)
{
  if (!appCtx->config.reid_track_dir_path)
    return;

  gchar reid_file[1024] = { 0 };
  FILE *reid_params_dump_file = NULL;
  /** Find batch reid tensor in batch user meta. */
  NvDsReidTensorBatch *pReidTensor = NULL;
  for (NvDsUserMetaList *l_batch_user = batch_meta->batch_user_meta_list; l_batch_user != NULL;
      l_batch_user = l_batch_user->next) {
    NvDsUserMeta *user_meta = (NvDsUserMeta *) l_batch_user->data;
    if (user_meta && user_meta->base_meta.meta_type == NVDS_TRACKER_BATCH_REID_META) {
      pReidTensor = (NvDsReidTensorBatch *) (user_meta->user_meta_data);
    }
  }

  /** Save the reid embedding for each frame. */
  for (NvDsMetaList * l_frame = batch_meta->frame_meta_list; l_frame != NULL;
      l_frame = l_frame->next) {
    NvDsFrameMeta *frame_meta = (NvDsFrameMeta *) l_frame->data;

    /** Create dump file name. */
    guint stream_id = frame_meta->pad_index;
    g_snprintf (reid_file, sizeof (reid_file) - 1,
        "%s/%02u_%03u_%06lu.txt", appCtx->config.reid_track_dir_path,
        appCtx->index, stream_id, (gulong) frame_meta->frame_num);
    reid_params_dump_file = fopen (reid_file, "w");
    if (!reid_params_dump_file)
      continue;

    if (!pReidTensor)
      continue;

    /** Save the reid embedding for each object. */
    for (NvDsMetaList * l_obj = frame_meta->obj_meta_list; l_obj != NULL;
        l_obj = l_obj->next) {
      NvDsObjectMeta *obj = (NvDsObjectMeta *) l_obj->data;
      guint64 id = obj->object_id;

      for (NvDsUserMetaList * l_obj_user = obj->obj_user_meta_list; l_obj_user != NULL;
          l_obj_user = l_obj_user->next) {

        /** Find the object's reid embedding index in user meta. */
        NvDsUserMeta *user_meta = (NvDsUserMeta *) l_obj_user->data;
        if (user_meta && user_meta->base_meta.meta_type == NVDS_TRACKER_OBJ_REID_META
            && user_meta->user_meta_data) {

          gint reidInd = *((int32_t *) (user_meta->user_meta_data));
          if (reidInd >= 0 && reidInd < (gint)pReidTensor->numFilled) {
            fprintf (reid_params_dump_file, "%lu", id);
            for (guint ele_i = 0; ele_i < pReidTensor->featureSize; ele_i++) {
              fprintf (reid_params_dump_file, " %f",
                pReidTensor->ptr_host[reidInd * pReidTensor->featureSize + ele_i]);
            }
            fprintf (reid_params_dump_file, "\n");
          }
        }
      }
    }
    fclose (reid_params_dump_file);
  }
}

So, I implemented the above code using Python as follows.
First, I got NvDsReidTensorBatch from BatchMeta,

def caps_src_pad_buffer_probe(unused_pad: Gst.Pad, info: Gst.PadProbeInfo, user_data: Any) -> Gst.PadProbeReturn:
    gst_buffer = info.get_buffer()
    if not gst_buffer:
        logger.error("Unable to get Gst.Buffer ")
        return

    batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
    reidFeature: np.ndarray = None

    l_user_meta = batch_meta.batch_user_meta_list
    while l_user_meta is not None:
        try:
            user_meta = pyds.NvDsUserMeta.cast(l_user_meta.data)
        except StopIteration:
            break

        if user_meta.base_meta.meta_type == pyds.NvDsMetaType.NVDS_TRACKER_BATCH_REID_META:
            reidTensor = pyds.NvDsReidTensorBatch.cast(user_meta.user_meta_data)
            reidFeatures = reidTensor.get_features()

        try:
            l_user_meta = l_user_meta.next
        except StopIteration:
            break

Then, I got a ReID feature vector as follows.

    l_frame = batch_meta.frame_meta_list
    while l_frame is not None:
        try:
            frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
        except StopIteration:
            break

        l_obj = frame_meta.obj_meta_list
        while l_obj is not None:
            try:
                obj_meta = pyds.NvDsObjectMeta.cast(l_obj.data)
            except StopIteration:
                break

            l_user_meta = obj_meta.obj_user_meta_list
            while l_user_meta is not None:
                try:
                    user_meta = pyds.NvDsUserMeta.cast(l_user_meta.data)
                except StopIteration:
                    break

                if user_meta.base_meta.meta_type == pyds.NvDsMetaType.NVDS_TRACKER_OBJ_REID_META and user_meta.user_meta_data:
                    reidInd = ctypes.cast(
                        pyds.get_ptr(user_meta.user_meta_data), ctypes.POINTER(ctypes.c_int32)
                    ).contents.value
                    if reidInd >= 0 and reidInd < reidTensor.numFilled:
                        feature = reidFeatures[reidInd, :]

                try:
                    l_user_meta = l_user_meta.next
                except StopIteration:
                    break

            try:
                l_obj = l_obj.next
            except StopIteration:
                break

        try:
            l_frame = l_frame.next
        except StopIteration:
            break

I checked the reidInd values for each batch and found that reidInd was duplicated when USE_NEW_NVSTREAMMUX=yes

Is there something I misunderstood?

I think I know the cause of this problem. Let’s forget about python-bindings first and make C code work fine.

These codes worked in DS-6.3, but unfortunately their behavior changed in DS-7.0.
This is why I can’t reproduce it. See the write_reid_track_output implementation in DS-7.0.

Please use DS-7.0, you can read NvDsObjReid directly from NVDS_TRACKER_OBJ_REID_META without parsing NvDsReidTensorBatch. This way you don’t have to get reidInd.

In addition, if there are bugs, we may not fix them on DS-6.3

As I posted first, DS 7.0 also produce wierd vector embedding.

I also made python binding for DS7.0 as you mentioned.
And I tried to get reid vector emeddings for DS 7.0 as follows.

if user_meta.base_meta.meta_type == pyds.NvDsMetaType.NVDS_TRACKER_OBJ_REID_META:
    objReid = pyds.NvDsObjReid.cast(user_meta.user_meta_data)

    if objReid and objReid.ptr_host != 0 and objReid.featureSize > 0:
        feature = objReid.get_feature().copy()

But if I set USE_NEW_NVSTREAMMUX=yes, I still can’t get the correct ReID vector embedding.

Maybe I should compare the output vectors of the C code for the same input videos and check the distribution of each object_id.

First make sure that the C code is working properly. If there is a problem with the corresponding python code, I think it may be that the python bindings has encountered issues.