Updating/Modifying NvDsUserMeta

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

• Hardware Platform (Jetson / GPU): Jetson AGX Xavier
• DeepStream Version: 5.0.0
• TensorRT Version 7.1.3
• Issue Type( questions, new requirements, bugs): question

Hello,

In my DeepStream application, I add custom user metas to both frames and objects in a very similar fashion to nvinfer_src_pad_buffer_probe in deepstream-user-metadata-test. I’ve created a probe called src_pad_buffer_probe that gets added to my primary GIE bin.

I’ve also written some custom plugins that are placed after my primary GIE bin. I want to be able to edit the metadata created by my src_pad_buffer_probe in my plugin’s gst_dsexample_transform_ip function, but I am running into issues. In particular, I am doing something like this in gst_dsexample_transform_ip:

  nvds_acquire_meta_lock(batch_meta);
  for (l_frame = batch_meta->frame_meta_list; l_frame != NULL; l_frame = l_frame->next) {
    frame_meta = (NvDsFrameMeta *) l_frame->data;

    // iterate over detections in each frame
    for (l_obj = frame_meta->obj_meta_list; l_obj != NULL; l_obj = l_obj->next) {
      obj_meta = (NvDsObjectMeta *) l_obj->data;
      NvDsUserMetaList *l_user_meta = obj_meta->obj_user_meta_list;
      if (!g_list_length(l_user_meta)) {
          continue;
      }

      NvDsUserMeta *user_meta = (NvDsUserMeta *) l_user_meta->data;
      float *pos_meta_data = (float*) user_meta->user_meta_data;

      float global_coords[3];
      // do some stuff

      // update metadata
      pos_meta_data[1] = global_coords[0];
      pos_meta_data[2] = global_coords[1];
      pos_meta_data[3] = global_coords[2];
    }
  }
  nvds_release_meta_lock(batch_meta);

This code runs and gives me no errors, but when I look at the metadata for these same objects further down the pipeline, my changes weren’t propagated. What am I missing? Is it not possible to update user metas once they’ve been set?

3

How could you be sure the user meta is what you added. Please check the meta type before you use it. Just as the sample deepstream-user-metadata-test.

If the sample deepstream-user-metadata-test works in your platform, just refer to the sample.

If the sample works, the problem should be in your code. You can debug by yourself and refer to the sample.

4

Hm, I add the user metas with this:

static GstPadProbeReturn 
src_pad_buffer_probe(GstPad *pad, GstPadProbeInfo *info, gpointer u_data) {
    GstBuffer *buf = (GstBuffer *) info->data;
    NvDsMetaList *l_frame = NULL;
    NvDsUserMeta *user_meta = NULL;
    NvDsObjectMeta *obj_meta = NULL;

    NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta (buf);
    for (l_frame = batch_meta->frame_meta_list; l_frame != NULL; l_frame = l_frame->next) {
        NvDsFrameMeta *frame_meta = (NvDsFrameMeta *) (l_frame->data);

        // get user_meta from batch pool
        user_meta = nvds_acquire_user_meta_from_pool(batch_meta);

        // initalize and allocate memory for framewise user metadata
        user_meta->user_meta_data = set_frame_user_metadata_ptr();
        user_meta->base_meta.meta_type = FRAME_USER_META_TYPE;
        user_meta->base_meta.copy_func = (NvDsMetaCopyFunc) copy_frame_user_meta;
        user_meta->base_meta.release_func = (NvDsMetaReleaseFunc) release_frame_user_meta;

        // add user meta to frame
        nvds_add_user_meta_to_frame(frame_meta, user_meta);

        for (NvDsMetaList *l_obj = frame_meta->obj_meta_list; l_obj != NULL; l_obj = l_obj->next) {
            obj_meta = (NvDsObjectMeta *)l_obj->data;

            // get another user_meta from batch_pool
            user_meta = nvds_acquire_user_meta_from_pool(batch_meta);

            // initalize and allocate memory for framewise user metadata
            user_meta->user_meta_data = set_object_user_metadata_ptr();
            user_meta->base_meta.meta_type = OBJECT_USER_META_TYPE;
            user_meta->base_meta.copy_func = (NvDsMetaCopyFunc) copy_object_user_meta;
            user_meta->base_meta.release_func = (NvDsMetaReleaseFunc) release_object_user_meta;

            // add user meta to object
            nvds_add_user_meta_to_obj(obj_meta, user_meta);
        }

    }
    return GST_PAD_PROBE_OK;
}

where FRAME_USER_META_TYPE and OBJECT_USER_META_TYPE are defined in a header file with:

    #define FRAME_USER_META_TYPE (nvds_get_user_meta_type("NVIDIA.NVINFER.FRAME_USER_META"))
    #define OBJECT_USER_META_TYPE (nvds_get_user_meta_type("NVIDIA.NVINFER.OBJECT_USER_META"))

Then I add this probe to my primary GIE source pad with:

    GstPad *infer_src_pad = gst_element_get_static_pad(bin->primary_gie, "src");
    if (!infer_src_pad) {
        NVGSTDS_ERR_MSG_V("Failed to get source pad");
    } else {
        gst_pad_add_probe(infer_src_pad, GST_PAD_PROBE_TYPE_BUFFER, src_pad_buffer_probe, NULL, NULL);
    }
    gst_object_unref(infer_src_pad);

I’ve tried reproducing deepstream-user-metadata-test, but the issue is updating existing metadata, not setting it once, then only reading it later, so it hasn’t been very useful.

I’m not sure I understand your comment about meta type-I’m pretty sure what I am adding is a NvDsUserMeta. I think the code I’ve included in this reply does exactly that.

Thank you for your help so far!!

5

@Fiona.Chen I re-read your post and I think I’ve correctly added the type checks here:

  // iterate over frames
  nvds_acquire_meta_lock(batch_meta);
  for (l_frame = batch_meta->frame_meta_list; l_frame != NULL; l_frame = l_frame->next) {
    frame_meta = (NvDsFrameMeta *) l_frame->data;

    // iterate over detections in each frame
    for (l_obj = frame_meta->obj_meta_list; l_obj != NULL; l_obj = l_obj->next) {
      obj_meta = (NvDsObjectMeta *) l_obj->data;
      for (NvDsUserMetaList *l_user_meta = obj_meta->obj_user_meta_list; l_user_meta != NULL; l_user_meta = l_user_meta->next) {
        NvDsUserMeta *user_meta = (NvDsUserMeta *) l_user_meta->data;

        // make sure we're editing the right meta type
        if (user_meta->base_meta.meta_type != OBJECT_USER_META_TYPE) {
          continue;
        }

        float *pos_meta_data = (float*) user_meta->user_meta_data;

        float global_coords[3];
        // do some stuff

        // update metadata
        pos_meta_data[1] = global_coords[0];
        pos_meta_data[2] = global_coords[1];
        pos_meta_data[3] = global_coords[2];
        // then, estimate object velocities
      }
    }
  }
  nvds_release_meta_lock(batch_meta);

but I’m still having the same issue: changes I make here aren’t propagated down the pipeline

6

It is the same with deepstream-user-metadata-test, if the meta can be available in probe function, it can also be read and changed by plugin. We can not know what is wrong just with the pieces of codes you post here.