How to add custom data to NvDsObjectMeta

Leozih · September 27, 2024, 12:26pm

Hi guys,I am trying add my custom data to NvDsObjectMeta.It is an angle of each detected object in each frame.With this opinion,I have seen some posts in the forum，and I see the code below:

void
add_meta_to_buffer (GstBuffer * buf, gchar * meta)
{
  NvDsBatchMeta *batch_meta;
  NvDsUserMeta *user_meta;
  batch_meta = gst_buffer_get_nvds_batch_meta (buf);
  user_meta = nvds_acquire_user_meta_from_pool (batch_meta);
  if (user_meta) {
    NvDsEventMsgMeta *msg_meta =
        (NvDsEventMsgMeta *) g_malloc0 (sizeof (NvDsEventMsgMeta));
    msg_meta->otherAttrs = meta;
    msg_meta->objectId = g_strdup ("custom");
    user_meta->user_meta_data = (void *) msg_meta;
    user_meta->base_meta.batch_meta = batch_meta;
    user_meta->base_meta.meta_type = NVDS_EVENT_MSG_META;
    user_meta->base_meta.copy_func = (NvDsMetaCopyFunc) custom_meta_copy_func;
    user_meta->base_meta.release_func =
        (NvDsMetaReleaseFunc) custom_meta_free_func;
    nvds_add_user_meta_to_batch (batch_meta, user_meta);
  }
}

The description is :
NvDsEventMsgMeta is a type of DeepStream user meta and there is a NvDsUserMetaList in the batch, frame and object levels of DeepStream metadata, so you can add message meta per-batch, per-frame, or per-object. We use it on the batch level adding a JSON as message meta

As you can see,it is on the batch level to add data.So what if it is on the object level?My initial opinion is :

for (NvDsMetaList * l_frame = batch_meta->frame_meta_list; l_frame != NULL;
      l_frame = l_frame->next) {
      NvDsFrameMeta *frame_meta = l_frame->data;
      for (NvDsMetaList * l_obj = frame_meta->obj_meta_list; l_obj != NULL;
        l_obj = l_obj->next) {
        NvDsObjectMeta *obj = (NvDsObjectMeta *) l_obj->data;
        user_meta = nvds_acquire_user_meta_from_pool (obj);
        if (user_meta) {
        NvDsEventMsgMeta *msg_meta =(NvDsEventMsgMeta *) g_malloc0 (sizeof (NvDsEventMsgMeta));
        msg_meta->otherAttrs = meta;
        msg_meta->objectId = g_strdup ("custom");
        user_meta->user_meta_data = (void *) msg_meta;
        user_meta->base_meta.batch_meta = batch_meta;
        user_meta->base_meta.meta_type = NVDS_EVENT_MSG_META;
        user_meta->base_meta.copy_func = (NvDsMetaCopyFunc) custom_meta_copy_func;
        user_meta->base_meta.release_func =(NvDsMetaReleaseFunc) custom_meta_free_func;
        nvds_add_user_meta_to_batch (obj, user_meta);
  }

So as you can see the only difference is to replace batch_meta to obj,namely the obj_meta.The angle will be the meta parameter in the above codes.I wonder if my opinion is practical

junshengy · September 29, 2024, 2:51am

Yes, you are right.There are some code snippets in the sample application.
Such as the following code in /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvtracker/nvtracker_proc.cpp

if (m_Config.outputConvexHull && pTrackedObj->convexHull.numPointsAllocated != 0)
  {
      NvDsUserMeta *user_meta = nvds_acquire_user_meta_from_pool (pBatchMeta);
      NvDsObjConvexHull* pConvexHull = new NvDsObjConvexHull;
      pConvexHull->list = new int [pTrackedObj->convexHull.numPointsAllocated * 2];
      for (uint32_t i=0; i < pTrackedObj->convexHull.numPoints; i++)
      {
        pConvexHull->list[2*i] = (int) (pTrackedObj->convexHull.list[2*i] * scaleWidth);
        pConvexHull->list[2*i+1] = (int) (pTrackedObj->convexHull.list[2*i+1] * scaleHeight);
      }
      pConvexHull->numPoints = pTrackedObj->convexHull.numPoints;
      pConvexHull->numPointsAllocated = pTrackedObj->convexHull.numPointsAllocated;
      user_meta->user_meta_data = (void *)pConvexHull;
      user_meta->base_meta.meta_type = (NvDsMetaType) NVDS_OBJ_IMAGE_CONVEX_HULL;
      user_meta->base_meta.copy_func = (NvDsMetaCopyFunc) copy_nvtracker2_obj_convex_hull_meta;
      user_meta->base_meta.release_func = (NvDsMetaReleaseFunc) free_nvtracker2_obj_convex_hull_meta;
      nvds_add_user_meta_to_obj (pObjectMeta, user_meta);
  }

Leozih · September 29, 2024, 12:50pm

Thanks,can I delete some lines of the codes I provide?For example:

user_meta->base_meta.copy_func = (NvDsMetaCopyFunc) custom_meta_copy_func;
user_meta->base_meta.release_func =(NvDsMetaReleaseFunc) custom_meta_free_func;

Because I don’t define such custom function,if the two lines are deleted,any influence?
Beside, about this line:
user_meta->base_meta.batch_meta = batch_meta;
Is it necessary to replace the batch_meta to obj?

junshengy · September 30, 2024, 1:53am

No, this code must be exist. These callbacks will be called when meta data is copied or released between elements.

Please refer to the example above

Leozih · September 30, 2024, 2:11am

Thanks,could you please give me some code snippet about these callbacks?
Besides,according to your codes,this line:

user_meta->base_meta.batch_meta = batch_meta;

seems to be unnecessary?I don’t see similar code in your codes

junshengy · September 30, 2024, 7:59am

This depends on how you use it.

If you want to find the corresponding batchmeta when accessing objmeta, this is necessary.

Otherwise, it is not necessary.

Leozih · September 30, 2024, 8:07am

Thanks,could you please give me some code snippet about these callbacks?namely the :

user_meta->base_meta.copy_func = (NvDsMetaCopyFunc) custom_meta_copy_func;
user_meta->base_meta.release_func =(NvDsMetaReleaseFunc) custom_meta_free_func;

custom_meta_copy_func and custom_meta_free_func.I don’t know how they are defined.

junshengy · September 30, 2024, 8:20am

You can refer to /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-user-metadata-test/deepstream_user_metadata_app.c. Here is code snippet.

static gpointer copy_user_meta(gpointer data, gpointer user_data)
{
  NvDsUserMeta *user_meta = (NvDsUserMeta *)data;
  gchar *src_user_metadata = (gchar*)user_meta->user_meta_data;
  gchar *dst_user_metadata = (gchar*)g_malloc0(USER_ARRAY_SIZE);
  memcpy(dst_user_metadata, src_user_metadata, USER_ARRAY_SIZE);
  return (gpointer)dst_user_metadata;
}

/* release function set by user. "data" holds a pointer to NvDsUserMeta*/
static void release_user_meta(gpointer data, gpointer user_data)
{
  NvDsUserMeta *user_meta = (NvDsUserMeta *) data;
  if(user_meta->user_meta_data) {
    g_free(user_meta->user_meta_data);
    user_meta->user_meta_data = NULL;
  }
}

This is another sample.

github.com

NVIDIA-AI-IOT/deepstream_tao_apps/blob/master/apps/tao_others/deepstream-pose-classification/deepstream_pose_classification_app.cpp#L683


      
                  ds_joints->joints[i].confidence = keypoints_confidence[i];
                  ds_joints->joints[i].x = p3dLifted[i].x;
                  ds_joints->joints[i].y = p3dLifted[i].y;
                  ds_joints->joints[i].z = p3dLifted[i].z;
                  //g_print("%d point [%f, %f, %f]\n", i, p3dLifted[i].x, p3dLifted[i].y, p3dLifted[i].z);
                }
                user_meta->user_meta_data = ds_joints;
                user_meta->base_meta.meta_type = (NvDsMetaType) NVDS_OBJ_META;
                user_meta->base_meta.release_func = release_bodypose_meta;
                user_meta->base_meta.copy_func = copy_bodypose_meta;
                nvds_add_user_meta_to_obj (obj_meta, user_meta);
              }
            }
          }
          
          /** Function to get the absolute path of a file.*/
          gboolean
          get_absolute_file_path_yaml (
              const gchar * cfg_file_path, const gchar * file_path,
              char *abs_path_str)
          {

Leozih · October 9, 2024, 7:48am

Thanks very much,I will try that later

Leozih · October 10, 2024, 12:55pm

Sorry to bother after so long,I notice that the
NvDsEventMsgMeta is a type of DeepStream user meta and there is a NvDsUserMetaList in the batch, frame and object levels of DeepStream metadata, so you can add message meta per-batch, per-frame, or per-object. We use it on the batch level adding a JSON as message meta

It says that “add message”.
So,if I want to add a parameter like angle to the NvDsObjectMeta, are the codes I have mentioned still works.

junshengy · October 11, 2024, 8:03am

It is not recommended to modify the struct NvDsObjectMeta, because some libraries are not open source and this may cause binary incompatibility.

NvDsObjectMeta has nothing to do with NvDsUserMetaList/NvDsEventMsgMeta, I don’t quite understand what you mean

Leozih · October 11, 2024, 8:11am

just like I have mentioned at first，I imitate some codes trying to store angle parameter by using user_meta:

for (NvDsMetaList * l_frame = batch_meta->frame_meta_list; l_frame != NULL;
      l_frame = l_frame->next) {
      NvDsFrameMeta *frame_meta = l_frame->data;
      for (NvDsMetaList * l_obj = frame_meta->obj_meta_list; l_obj != NULL;
        l_obj = l_obj->next) {
        NvDsObjectMeta *obj = (NvDsObjectMeta *) l_obj->data;
        user_meta = nvds_acquire_user_meta_from_pool (obj);
        if (user_meta) {
        NvDsEventMsgMeta *msg_meta =(NvDsEventMsgMeta *) g_malloc0 (sizeof (NvDsEventMsgMeta));
        msg_meta->otherAttrs = meta;
        msg_meta->objectId = g_strdup ("custom");
        user_meta->user_meta_data = (void *) msg_meta;
        user_meta->base_meta.batch_meta = batch_meta;
        user_meta->base_meta.meta_type = NVDS_EVENT_MSG_META;
        user_meta->base_meta.copy_func = (NvDsMetaCopyFunc) custom_meta_copy_func;
        user_meta->base_meta.release_func =(NvDsMetaReleaseFunc) custom_meta_free_func;
        nvds_add_user_meta_to_batch (obj, user_meta);
  }

but the msg_meta which is stored seems to be “message” type, angel may be just a numble like 60 or 30.8456 and so on.I mean will this be incompatible to the msg_meta?

junshengy · October 11, 2024, 8:19am

No, you can think of NvDsEventMsgMeta as a handle to store “message”.

If the members of this structure do not meet your needs, you can use extMsg/extMsgSize to store custom data.

/opt/nvidia/deepstream/deepstream/sources/includes/nvdsmeta_schema.h

  /** Holds a pointer to event message meta data. This can be used to hold
   data that can't be accommodated in the existing fields, or an associated
   object (representing a vehicle, person, face, etc.). */
  gpointer extMsg;
  /** Holds the size of the custom object at @a extMsg. */
  guint extMsgSize;

Refer to deepstream_test5_app_main.c

Leozih · October 11, 2024, 8:24am

Thanks,so you mean that in theory the angle parameter is compatible to the NvDsEventMsgMeta?
So I can just replace the meta above to angle,then it may work.

junshengy · October 11, 2024, 9:16am

1.If you just only want to store angle to NvDsObjectMeta. Use misc_obj_info is more simpler.
Just do the following to avoid the lengthy code of object usemeta

objmeta->misc_obj_info[0] = angle;

2.Like NvDsEventMsgMeta

typedef struct _UserObjMeta {
   xxxx;
   xxx;
} UserObjMeta;

// meta type defined by customer
#define NVDS_USER_OBJ_META_EXAMPLE (nvds_get_user_meta_type("NVIDIA.NVINFER.USER_META"))

for (NvDsMetaList * l_frame = batch_meta->frame_meta_list; l_frame != NULL;
      l_frame = l_frame->next) {
      NvDsFrameMeta *frame_meta = l_frame->data;
      for (NvDsMetaList * l_obj = frame_meta->obj_meta_list; l_obj != NULL;
        l_obj = l_obj->next) {
        NvDsObjectMeta *obj = (NvDsObjectMeta *) l_obj->data;
        user_meta = nvds_acquire_user_meta_from_pool (obj);
        if (user_meta) {
        UserObjMeta *obj_meta =(UserObjMeta *) g_malloc0 (sizeof (UserObjMeta));
        user_meta->user_meta_data = (void *) obj_meta;
        user_meta->base_meta.batch_meta = batch_meta;
        user_meta->base_meta.meta_type = NVDS_USER_OBJ_META_EXAMPLE; 
        user_meta->base_meta.copy_func = (NvDsMetaCopyFunc) custom_meta_copy_func;
        user_meta->base_meta.release_func =(NvDsMetaReleaseFunc) custom_meta_free_func;
        nvds_add_user_meta_to_batch (obj, user_meta);
       }
  }

Leozih · October 11, 2024, 10:41am

Wow,thank you very much! misc_obj_info is really convenient and works fine!

system · October 25, 2024, 10:41am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Deepstream 5.0 python bindings for Gst-nvdsanalytics access meta data DeepStream SDK	34	4180	October 12, 2021
No changes being made in deepstream-test5 app DeepStream SDK	7	840	October 2, 2021
dsexample causing memory leak with cv::Mat attached to obj_meta DeepStream SDK	4	1764	April 27, 2020
Regarding doubts in deeepstream -test5 DeepStream SDK gstreamer	5	498	June 9, 2023
Why NvDsUserMetaList is NULL DeepStream SDK	16	318	December 19, 2023
Updating/Modifying NvDsUserMeta DeepStream SDK	5	1060	October 12, 2021
Sending Frames + MetaData (detections + classes + tracking IDs) to Kafka for each stream running DeepStream SDK kafka , deepstream	31	166	January 23, 2025
Add Frame to MSGBroker payload using EventMetaData DeepStream SDK nvbugs	37	1389	December 8, 2023
How to add extra function in the deepstream like edge detection? DeepStream SDK opencv , jetson-inference , deepstream	32	141	October 9, 2024
Saving meta data matching SmartRecord files DeepStream SDK	29	2137	October 12, 2021

How to add custom data to NvDsObjectMeta

Related topics