Deepstream 5.0 python bindings for Gst-nvdsanalytics access meta data

Hi

Is there any update for when the python bindings for the Gst-nvdsanalytics meta data will be available?

Best,
Niels

It should work on 5.0GA, refer https://docs.nvidia.com/metropolis/deepstream/python-api/NvDsAnalytics/pyanalytics.html

But how do I access the analytics meta data generated from the Gst-nvdsanalytics component? In the C++ example (deepstream-nvdsanalytics-test) they access it from the NvDsUserMeta attached to the NvDsObjectMeta:

for (l_obj = frame_meta->obj_meta_list; l_obj != NULL;
                l_obj = l_obj->next) {
            obj_meta = (NvDsObjectMeta *) (l_obj->data);

            // Access attached user meta for each object
            for (NvDsMetaList *l_user_meta = obj_meta->obj_user_meta_list; l_user_meta != NULL;
                    l_user_meta = l_user_meta->next) {
                NvDsUserMeta *user_meta = (NvDsUserMeta *) (l_user_meta->data);
                if(user_meta->base_meta.meta_type == NVDS_USER_OBJ_META_NVDSANALYTICS)
                {
                    NvDsAnalyticsObjInfo * user_meta_data =
                        (NvDsAnalyticsObjInfo *)user_meta->user_meta_data;
                    if (user_meta_data->dirStatus.length()){
                        out_string << " object " << obj_meta->object_id <<
                            " is moving in " <<  user_meta_data->dirStatus;
                    }
                }
            }
        }

I am able to iterate the user meta data in python and it contains something but I cant seem to figure out how to cast it to the NvDsAnalyticsObjInfo object such that I can access the data from python. I have tried the following in my python probe:

            user_meta_list = obj_meta.obj_user_meta_list
            while user_meta_list is not None:
                user_meta = pyds.NvDsUserMeta.cast(user_meta_list.data)
                user_meta_data = user_meta.user_meta_data
                try: 
                    user_meta_list=user_meta_list.next
                except StopIteration:
                    break 

+1

+1 Would also like to see an examle

Currently, the python bindings don’t have a “cast” function for NvDsAnalyticsObjInfo, but you can generate your bindings and cast function as follows

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include "nvds_analytics_meta.h"

namespace py = pybind11;
PYBIND11_MODULE(pyds_analytics, m) {
	py::class_<NvDsAnalyticsObjInfo>(m,"NvDsAnalyticsObjInfo")
            .def(py::init<>())
            .def_readwrite("roiStatus",&NvDsAnalyticsObjInfo::roiStatus)
            .def_readwrite("ocStatus",&NvDsAnalyticsObjInfo::ocStatus)
            .def_readwrite("lcStatus",&NvDsAnalyticsObjInfo::lcStatus)
            .def_readwrite("dirStatus",&NvDsAnalyticsObjInfo::dirStatus)
            .def_readwrite("unique_id",&NvDsAnalyticsObjInfo::unique_id)
	        .def(“cast”,[](void *data){
		 	            return (NvDsAnalyticsObjInfo*)data;
             });
    py::class_<NvDsAnalyticsFrameMeta>(m,"NvDsAnalyticsFrameMeta)
            .def(py::init<>())
            .def_readwrite("ocStatus",&NvDsAnalyticsFrameMeta::ocStatus)
            .def_readwrite("objInROIcnt",&NvDsAnalyticsFrameMeta::objInROIcnt)
            .def_readwrite("objLCCurrCnt",&NvDsAnalyticsFrameMeta::objLCCurrCnt)
            .def_readwrite("objLCCumCnt",&NvDsAnalyticsFrameMeta::objLCCumCnt)
            .def_readwrite("unique_id",&NvDsAnalyticsFrameMeta::unique_id)
            .def_readwrite("objCnt",&NvDsAnalyticsFrameMeta::objCnt)
            .def(“cast”,[](void *data){
		 	    return (NvDsAnalyticsFrameMeta *)data;
             });
}

Save this file as pyds_analytics.cpp and then to generate bindings please refer to the following repo https://github.com/mrtj/pyds_tracker_meta
Replace the pyds_tracker_meta.cpp with pyds_analytics.cpp and do the same in build.sh and setup.py files, to generate the bindings. Then you can cast the user_meta.user_meta_data to NvDsAnalyticsObjInfo as follows
import pyds_analytics
ObjInfo=pyds_analytics.NvDsAnalyticsObjInfo.cast(user_meta.user_meta_data)

Ahh perfect! Thanks for the detailed answer. I will make sure to test it as soon as possible.

@manojy Thanks for the workaround info, successfully built and install so file import also working, but import in deepstream app is giving this error

Traceback (most recent call last):
  File "temp.py", line 44, in <module>
    import pyds
ImportError: generic_type: type "NvDsAnalyticsObjInfo" is already registered!

Referring to https://pybind11.readthedocs.io/en/stable/advanced/classes.html added this py::module_local() now able to import in deepstream app but ObjInfo=pyds_analytics.NvDsAnalyticsObjInfo.cast(user_meta.user_meta_data) this causes Segmentation fault (core dumped) and exits

pyds_analytics.cpp

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include “nvds_analytics_meta.h”

namespace py = pybind11;
PYBIND11_MODULE(pyds_analytics, m) {
py::class_(m,“NvDsAnalyticsObjInfo”, py::module_local())
.def(py::init<>())
.def_readwrite(“roiStatus”,&NvDsAnalyticsObjInfo::roiStatus)
.def_readwrite(“ocStatus”,&NvDsAnalyticsObjInfo::ocStatus)
.def_readwrite(“lcStatus”,&NvDsAnalyticsObjInfo::lcStatus)
.def_readwrite(“dirStatus”,&NvDsAnalyticsObjInfo::dirStatus)
.def_readwrite(“unique_id”,&NvDsAnalyticsObjInfo::unique_id)
.def(“cast”,(void data){
return (NvDsAnalyticsObjInfo
)data;
});
py::class_(m,“NvDsAnalyticsFrameMeta”, py::module_local())
.def(py::init<>())
.def_readwrite(“ocStatus”,&NvDsAnalyticsFrameMeta::ocStatus)
.def_readwrite(“objInROIcnt”,&NvDsAnalyticsFrameMeta::objInROIcnt)
.def_readwrite(“objLCCurrCnt”,&NvDsAnalyticsFrameMeta::objLCCurrCnt)
.def_readwrite(“objLCCumCnt”,&NvDsAnalyticsFrameMeta::objLCCumCnt)
.def_readwrite(“unique_id”,&NvDsAnalyticsFrameMeta::unique_id)
.def_readwrite(“objCnt”,&NvDsAnalyticsFrameMeta::objCnt)
.def(“cast”,(void *data){
return (NvDsAnalyticsFrameMeta *)data;
});
}

App Function

def nvdsanalytics_src_pad_buffer_probe(pad,info,u_data):

gst_buffer = info.get_buffer()
if not gst_buffer:
    print("Unable to get GstBuffer ")
    return

batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
l_frame = batch_meta.frame_meta_list
while l_frame is not None:
    try:
        frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
    except StopIteration:
        break
    l_user = frame_meta.frame_user_meta_list

    while l_user is not None:
        try:
            user_meta = pyds.NvDsUserMeta.cast(l_user.data)
            ObjInfo=pyds_analytics.NvDsAnalyticsObjInfo.cast(user_meta.user_meta_data)

        except StopIteration:
            break
        try:
            l_user=l_user.next
        except sStopIteration:
            break
   
    try:
        l_frame=l_frame.next
    except StopIteration:
        break

return Gst.PadProbeReturn.OK

Can you check your code snippet format, forum support markdown syntax

As NvDsAnalyticsObjInfo struct has already binded in pyds, you can simply add the following casting function to get NvDsAnalyticsObjInfo.

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include "nvds_analytics_meta.h"

namespace py = pybind11;
PYBIND11_MODULE(pyds_analytics, m) {
     m.def("cast_to_NvDsAnalyticsObjInfo",[](void *data){
		 	            return (NvDsAnalyticsObjInfo*)data;
             });
}

@manojy Now really confused what to use and how to use, could you acknowledge what is wrong in above code snippets.

@bcao didn’t get you

Hello

the link to the pyds_tracker_meta bindings worked great

As for pyds_analytics_meta, I am having far more trouble.

Ive had to comment out the line
// .def_readwrite(“objCnt”,&NvDsAnalyticsFrameMeta::objCnt)
============= pyds_analytics_meta.cpp ===============

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include "nvds_analytics_meta.h"

namespace py = pybind11;

PYBIND11_MODULE(pyds_tracker_meta, m) {
	m.doc() = "pybind11 wrapper to access Nvidia DeepStream NvDsAnalyticsMeta* classes";
    // m.def("cast_to_NvDsAnalyticsObjInfo"[](void *data){
	// 	return (NvDsAnalyticsObjInfo*)data;
    // });
    py::class_<NvDsAnalyticsObjInfo>(m,"NvDsAnalyticsObjInfo", py::module_local())
    // py::class_(m,“NvDsAnalyticsObjInfo”, py::module_local())
            .def(py::init<>())
            .def_readwrite("roiStatus",&NvDsAnalyticsObjInfo::roiStatus)
            .def_readwrite("ocStatus",&NvDsAnalyticsObjInfo::ocStatus)
            .def_readwrite("lcStatus",&NvDsAnalyticsObjInfo::lcStatus)
            .def_readwrite("dirStatus",&NvDsAnalyticsObjInfo::dirStatus)
            .def_readwrite("unique_id",&NvDsAnalyticsObjInfo::unique_id)
	        .def("cast",[](void *data){
		 	            return (NvDsAnalyticsObjInfo*)data;
             });
    py::class_<NvDsAnalyticsFrameMeta>(m,"NvDsAnalyticsFrameMeta", py::module_local())
            .def(py::init<>())
            .def_readwrite("ocStatus",&NvDsAnalyticsFrameMeta::ocStatus)
            .def_readwrite("objInROIcnt",&NvDsAnalyticsFrameMeta::objInROIcnt)
            .def_readwrite("objLCCurrCnt",&NvDsAnalyticsFrameMeta::objLCCurrCnt)
            .def_readwrite("objLCCumCnt",&NvDsAnalyticsFrameMeta::objLCCumCnt)
            .def_readwrite("unique_id",&NvDsAnalyticsFrameMeta::unique_id)
            // .def_readwrite("objCnt",&NvDsAnalyticsFrameMeta::objCnt)     <------- needed to be uncommented
            .def("cast",[](void *data){
		 	    return (NvDsAnalyticsFrameMeta *)data;
             });
}

============= nvds_analytics_meta.cpp ===============

/**
 * Copyright (c) 2020, NVIDIA CORPORATION.  All rights reserved.
 *
 * NVIDIA Corporation and its licensors retain all intellectual property
 * and proprietary rights in and to this software, related documentation
 * and any modifications thereto.  Any use, reproduction, disclosure or
 * distribution of this software and related documentation without an express
 * license agreement from NVIDIA Corporation is strictly prohibited.
 *
 */
#ifndef _NVDS_ANALYTICS_META_H_
#define _NVDS_ANALYTICS_META_H_

    #include <gst/gst.h>

    #ifdef __cplusplus
    extern "C"
    {
    #endif

    #define NVDS_USER_FRAME_META_NVDSANALYTICS (nvds_get_user_meta_type((gchar*)"NVIDIA.DSANALYTICSFRAME.USER_META"))
    #define NVDS_USER_OBJ_META_NVDSANALYTICS (nvds_get_user_meta_type((gchar*)"NVIDIA.DSANALYTICSOBJ.USER_META"))

    /**
     */
    typedef struct
    {
      std::vector <std::string> roiStatus;
      std::vector <std::string> ocStatus;
      std::vector <std::string> lcStatus;
      std::string dirStatus;
      guint unique_id;
    } NvDsAnalyticsObjInfo;

    /**
     */
    typedef struct
    {
      std::unordered_map<std::string, bool> ocStatus;
      std::unordered_map<std::string, uint32_t> objInROIcnt;
      std::unordered_map<std::string, uint64_t> objLCCurrCnt;
      std::unordered_map<std::string, uint64_t> objLCCumCnt;
      guint unique_id;

    } NvDsAnalyticsFrameMeta;

    #ifdef __cplusplus
    }
    #endif

    #endif

when I check the header file for the class it is indeed missing object count? Could this mean i am using a more out of date deepstream (5.0 jetpack 4.4)

However I can build and install it when objCnt is removed, but I cannot import the class into my deepstream application, or even an empty project, it throws the following error

import pyds_analytics_meta
Traceback (most recent call last):
File “”, line 1, in
ImportError: dynamic module does not define module export function (PyInit_pyds_analytics_meta)

Any help would be greatly appreciated

Regards Andrew

I have fixed some of my issues

Firstly i cleared some space on hdd and managed to update via the software updater (I noticed there were deepstream updates, now nvds_analytics_meta.cpp has objCnt.

I also realised "PYBIND11_MODULE(pyds_tracker_meta, m) " should have been PYBIND11_MODULE(pyds_tracker, m)

Now it successfully imports and i can use:

user_meta_analytics = pyds_analytics.NvDsAnalyticsObjInfo.cast(user_meta.user_meta_data)

however the pipeline freezes and causes Segmentation fault (core dumped) Same as @priyanshthakore

@manojy please clarify this

note missing " , " after cast_to_NvDsAnalyticsObjInfo

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include "nvds_analytics_meta.h"

namespace py = pybind11;
PYBIND11_MODULE(pyds_analytics, m) {
     m.def("cast_to_NvDsAnalyticsObjInfo",[](void *data){
		 	            return (NvDsAnalyticsObjInfo*)data;
             });
}

then to access the NvDsAnalyticsObjInfo:

    l_user = frame_meta.frame_user_meta_list

    while l_user is not None:
        try:
            print('try cast')
            user_meta = pyds.NvDsUserMeta.cast(l_user.data)
            if user_meta.user_meta_data:
                try:
                    print('user meta', user_meta.user_meta_data)
                    user_meta_analytics = pyds_analytics.cast_to_NvDsAnalyticsObjInfo(user_meta.user_meta_data)
                    # user_meta_analytics = pyds_analytics.NvDsAnalyticsObjInfo.cast(user_meta.user_meta_data)
                    print('user_meta_analytics:', user_meta_analytics)
                except Exception as ex:
                    print('exception', ex)

            print('user_meta', user_meta)
        except StopIteration:
            break
        try:
            l_user = l_user.next
        except StopIteration:
            break
    print('=======================================================================================')

or

user_meta_list = batch_meta.batch_user_meta_list
    while user_meta_list is not None:
        user_meta = pyds.NvDsUserMeta.cast(user_meta_list.data)
        if user_meta.user_meta_data:
            try:
                print('user meta',user_meta.user_meta_data)
                user_meta_analytics = pyds_analytics.cast_to_NvDsAnalyticsObjInfo(user_meta.user_meta_data)
                # user_meta_analytics = pyds_analytics.NvDsAnalyticsObjInfo.cast(user_meta.user_meta_data)
                print('user_meta_analytics:', user_meta_analytics)
            except Exception as ex:
                print('exception',ex)

Yields the following exception

exception Unable to convert function return value to a Python type! The signature was
(arg0: capsule) -> NvDsAnalyticsObjInfo

I am seen the same as @priyanshthakore and @Andrewzgas9.

I did the following as suggested by @manojy and added the py::module_local() (ref @priyanshthakore) :

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include "nvds_analytics_meta.h"

namespace py = pybind11;
PYBIND11_MODULE(pyds_analytics, m) {
	py::class_<NvDsAnalyticsObjInfo>(m,"NvDsAnalyticsObjInfo",py::module_local())
            .def(py::init<>())
            .def_readwrite("roiStatus",&NvDsAnalyticsObjInfo::roiStatus)
            .def_readwrite("ocStatus",&NvDsAnalyticsObjInfo::ocStatus)
            .def_readwrite("lcStatus",&NvDsAnalyticsObjInfo::lcStatus)
            .def_readwrite("dirStatus",&NvDsAnalyticsObjInfo::dirStatus)
            .def_readwrite("unique_id",&NvDsAnalyticsObjInfo::unique_id)
	        .def("cast",[](void *data){
		 	            return (NvDsAnalyticsObjInfo*)data;
             });
    py::class_<NvDsAnalyticsFrameMeta>(m,"NvDsAnalyticsFrameMeta",py::module_local())
            .def(py::init<>())
            .def_readwrite("ocStatus",&NvDsAnalyticsFrameMeta::ocStatus)
            .def_readwrite("objInROIcnt",&NvDsAnalyticsFrameMeta::objInROIcnt)
            .def_readwrite("objLCCurrCnt",&NvDsAnalyticsFrameMeta::objLCCurrCnt)
            .def_readwrite("objLCCumCnt",&NvDsAnalyticsFrameMeta::objLCCumCnt)
            .def_readwrite("unique_id",&NvDsAnalyticsFrameMeta::unique_id)
            .def_readwrite("objCnt",&NvDsAnalyticsFrameMeta::objCnt)
            .def("cast",[](void *data){
		 	    return (NvDsAnalyticsFrameMeta *)data;
             });
}

And accessing the NvDsAnalyticsObjInfo in python through:

user_meta_list = obj_meta.obj_user_meta_list
print(user_meta_list)
while user_meta_list is not None:
     user_meta = pyds.NvDsUserMeta.cast(user_meta_list)
     print(user_meta)
      try:
           analytic_obj = pyds_analytics.NvDsAnalyticsObjInfo.cast(user_meta.user_meta_data)
           if analytic_obj is not None:
                print(analytic_obj)
           try: 
               user_meta_list=user_meta_list.next
           except StopIteration:
               break

I also did as suggested by @manojy and tried with:

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include "nvds_analytics_meta.h"

namespace py = pybind11;
PYBIND11_MODULE(pyds_analytics, m) {
     m.def("cast_to_NvDsAnalyticsObjInfo",[](void *data){
		 	            return (NvDsAnalyticsObjInfo *)data;
             });
     m.def("cast_to_NvDsAnalyticsFrameMeta",[](void *data){
				    return (NvDsAnalyticsFrameMeta *)data;
	});
}

And again accessing the NvDsAnalyticsObjInfo in python through:

user_meta_list = obj_meta.obj_user_meta_list
print(user_meta_list)
        while user_meta_list is not None:
            user_meta = pyds.NvDsUserMeta.cast(user_meta_list)
            print(user_meta)
            try:
                analytic_obj = pyds_analytics.cast_to_NvDsAnalyticsObjInfo(user_meta.user_meta_data)
                if analytic_obj is not None:
                      print(analytic_obj)
            try: 
                user_meta_list=user_meta_list.next
            except StopIteration:
                break

The result is the same for both approaches and my terminal output looks like this when it freezes and stops:

user meta (obj) list <pyds.GList object at 0x7f505f29d0>
<pyds.NvDsUserMeta object at 0x7f505f2a40>
<pyds_analytics.NvDsAnalyticsObjInfo object at 0x7f505f2a08>
Segmentation fault (core dumped)

The problem is occurring due to the py::return_value_policy::automatic_reference, which is default pybind11 reference, this is causing memory dealloc problem which is resulting in segmentation fault. So changing it to py::return_value_policy::reference will work. Please use the following solution.

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include "nvds_analytics_meta.h"

namespace py = pybind11;
PYBIND11_MODULE(pyds_analytics, m) {
	py::class_<NvDsAnalyticsObjInfo>(m,"NvDsAnalyticsObjInfo",py::module_local())
            .def(py::init<>())
            .def_readwrite("roiStatus",&NvDsAnalyticsObjInfo::roiStatus)
            .def_readwrite("ocStatus",&NvDsAnalyticsObjInfo::ocStatus)
            .def_readwrite("lcStatus",&NvDsAnalyticsObjInfo::lcStatus)
            .def_readwrite("dirStatus",&NvDsAnalyticsObjInfo::dirStatus)
            .def_readwrite("unique_id",&NvDsAnalyticsObjInfo::unique_id)
	        .def_static("cast",[](void *data){
		 	            return (NvDsAnalyticsObjInfo*)data;
             },py::return_value_policy::reference);
    py::class_<NvDsAnalyticsFrameMeta>(m,"NvDsAnalyticsFrameMeta",py::module_local())
            .def(py::init<>())
            .def_readwrite("ocStatus",&NvDsAnalyticsFrameMeta::ocStatus)
            .def_readwrite("objInROIcnt",&NvDsAnalyticsFrameMeta::objInROIcnt)
            .def_readwrite("objLCCurrCnt",&NvDsAnalyticsFrameMeta::objLCCurrCnt)
            .def_readwrite("objLCCumCnt",&NvDsAnalyticsFrameMeta::objLCCumCnt)
            .def_readwrite("unique_id",&NvDsAnalyticsFrameMeta::unique_id)
            .def_readwrite("objCnt",&NvDsAnalyticsFrameMeta::objCnt)
            .def_static("cast",[](void *data){
		 	    return (NvDsAnalyticsFrameMeta *)data;
             },py::return_value_policy::reference);
}

@manojy - I just tried the fix and I am still seeing a segmentation fault (core dumped). Have you tried the code yourself and if so do you have a complete working example?

Secondly, is it possible to get and see the actual python deepstream bindings (pyds)? I can only find the compiled .so (pyds.so) in the deepstream SDK.

I tried the fix and it works, no segmentation fault
however I am still attempting to access the data

user_meta_list = batch_meta.batch_user_meta_list
print(’======================================================================================’)

while user_meta_list is not None:
    user_meta = pyds.NvDsUserMeta.cast(user_meta_list.data)
    if user_meta.user_meta_data:
        try:
            print('user meta',user_meta.user_meta_data)
            # user_meta_analytics = pyds_analytics.cast_to_NvDsAnalyticsObjInfo(user_meta.user_meta_data)
            user_meta_analytics = pyds_analytics.NvDsAnalyticsObjInfo.cast(user_meta.user_meta_data)
            print('user_meta_analytics:', user_meta_analytics)
            ============== Breaks here ==============
            print('dirStatus',user_meta_analytics.dirStatus)
            print('lcStatus',user_meta_analytics.lcStatus)
            print('ocStatus',user_meta_analytics.ocStatus)
            print('roiStatus',user_meta_analytics.roiStatus)

        except Exception as ex:
            print('exception',ex)

    try:
        user_meta_list = user_meta_list.next
    except StopIteration:
        break
    print('======================================================================================')

True, I am also able to do the cast but it fails when I access the data fields just as you