How to use pyds with custom Object classes?

m.fruhner · February 17, 2021, 8:28am

Hello,
I have a DeepStream 5 application in C/C++ (deepstream-app) where I added some custom classes to the Schema (NvDsVehicleObject → MyObject). I then recompiled the libs and was able to use the new classes in my code and the msg conv and msg broker.

Now I try to do this in pyds, but as it seems the classes are hardcoded into the bindings? (Vehicle, Person, Face, …). To generate my custom msg meta I have to cast an object to my own like so:

obj = pyds.NvDsVehicleObject.cast(data)

as described in python test-app 4

Is creating custom objects possible at all in Python at the moment? The sources are not open, so I could recompile myself, right?

Can I use

Generic buffer: alloc_buffer(size)

with the correct size of a python object to allocate my custom objects from python code?

Thanks for your help!

m.fruhner · February 18, 2021, 10:41am

Can someone help me figure this out? I want to try to create a custom message meta as stated in the README of python test-app 4:

Generating custom metadata for different type of objects:
In addition to common fields provided in NvDsEventMsgMeta structure, user can
also create custom objects and attach to buffer as NVDS_META_EVENT_MSG metadata.
To do that NvDsEventMsgMeta provides "extMsg" and "extMsgSize" fields. User can
create custom structure, fill that structure and assign the pointer of that
structure as "extMsg" and set the "extMsgSize" accordingly.
If custom object contains fields that can't be simply mem copied then user should
also provide function to copy and free those objects.

Refer generate_event_msg_meta() to know how to use "extMsg" and "extMsgSize"
fields for custom objects and how to provide copy/free function and attach that
object to buffer as metadata.

Therefore I created a simple python class with one member:

class MyObject:     
    def __init__(self, num):
         self.num = num

and I want to assign an instance of this to the meta.extMsg:

# set the values of the meta object
meta.type = pyds.NvDsEventType.NVDS_EVENT_CUSTOM
meta.objType = 42
meta.objClassId = 42

# create the meta extension object and set its obj
obj = MyObject(42)
obj_size = sys.getsizeof(obj)

# allocate a c buffer and assign the pointer address
meta.extMsg = pyds.alloc_buffer(obj_size)
meta.extMsgSize = obj_size

But this will only allocate empty memory without a reference to “obj”. How can I write it to the pointer address?

Sadly there is no documentation at all on how to use custom objects. I searched the Github and “alloc_buffer” is only used for a timestamp and not for python objects.

m.fruhner · February 22, 2021, 11:17am

I found similar issuse in the forum now and the suggestion is to create the classes in CPP and then regenerat the bindings.
I already have the structs for the C Version of my project. How do I re-generate the pyds bindings which contain my classes now? I thought the pyds.so was a closed-source pre compiled lib?

mchi · February 22, 2021, 1:10pm

Great! could you take a look DeepStream SDK FAQ - #16 by bcao if it can help you?

m.fruhner · February 23, 2021, 10:08am

Hello,
this definitly helped a lot. But I still need to know, how to alloc an object, which is not deleted by the python garbage collector. Both examples only cast an existing object to a custom type. I want to set the extMsg to my custom type.

I would like to alloc a new objects on the Cpp Heap.

Would this Binding create a reference, which is maintained after the python code finished?:

py::class_<MyObject>(m,"MyObject",py::module_local())
 .def(py::init<>())             
 .def_readwrite("identifier",&MyObject::identifier)             
 .def_static("cast",[  ](void *data){                 
    return (MyObject*)data;             
},py::return_value_policy::reference);

recharles · June 29, 2021, 4:00pm

Hi! Did you find a way to generate a custom object class using python bindings?

m.fruhner · June 30, 2021, 6:42am

Hello, yes. I used the Links posted by mchi. I created a my_bindings.cpp with this content:

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
//#include "./includes/nvdsmeta_schema.h"
#include "nvdsmeta_schema.h"

namespace py = pybind11;

MyObject *alloc_my_obj()
{
        return new MyObject();
}

PYBIND11_MODULE(pyds_custom, m)
{
        py::enum_<NvDsObjectType>(m, "NvDsObjectType", py::module_local())
            .value("OBJECT_TYPE_MY_CUSTOM", NvDsObjectType::OBJECT_TYPE_MY_CUSTOM)
            .export_values();

        py::enum_<NvDsPayloadType>(m, "NvDsPayloadType", py::module_local())
            .value("MY_PAYLOAD_1", NvDsPayloadType::MY_PAYLOAD_1)
            .export_values();

        py::class_<MyObject>(m, "MyObject", py::module_local())
            .def(py::init<>())
            .def_readwrite("number", &MyObject::number)
            .def_static(
                "cast", [](void *data) {
                        return (MyObject*)data;
                },
                py::return_value_policy::reference);

        m.def("alloc_my_object", &alloc_my_object, py::return_value_policy::reference);
}

Then you build it with the build script from the repo and install it with setup.py (you have to edit this as well). Then you can import the types and use them.

recharles · June 30, 2021, 10:24am

Thank you for your answer, I have tried using your solution, which I modified to:

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include "/opt/nvidia/deepstream/deepstream-5.1/sources/includes/nvdsmeta_schema.h"
#include "nvdsmeta_schema.h"

namespace py = pybind11;

MyObject *alloc_my_object()
{
        return new MyObject();
}

PYBIND11_MODULE(pyds_custom_meta, m)
{       
        py::enum_<NvDsObjectType>(m, "NvDsObjectType", py::module_local())
            .value("OBJECT_TYPE_MY_CUSTOM", NvDsObjectType::NVDS_OBJECT_TYPE_CUSTOM)
            .export_values();

        py::enum_<NvDsPayloadType>(m, "NvDsPayloadType", py::module_local())
            .value("MY_PAYLOAD_1", NvDsPayloadType::NVDS_PAYLOAD_CUSTOM)
            .export_values();

        py::class_<MyObject>(m, "MyObject", py::module_local())
            .def(py::init<>())
            .def_readwrite("number", &MyObject::number)
            .def_static(
                "cast", [](void *data) {
                        return (MyObject*)data;
                },
                py::return_value_policy::reference);

        m.doc() = "This is a Python binding of C++ Maia Library";
        m.def("alloc_my_object", &alloc_my_object, py::return_value_policy::reference);
}

Where instead ot the custom type you use, I switched to the NVDS_OBJECT_TYPE_CUSTOM, since I was not sure how I could have defined it as you do. By following your instructions I was able to generate the bindings and correctly import them in python (deepstream-test-4.py). There, I wanted to use this generated custom object for the custom field “extMsg”, but probably I am making mistakes in doing that and get the error:

 incompatible function arguments. The following argument types are supported:
    1. (self: pyds.NvDsEventMsgMeta, arg0: capsule) -> None

In the deepstream_test_4.py script I modified the function “generate_event_msg_meta” to use the custom object (similar to the way you proposed in this thread). Specifically:

def generate_event_msg_meta(data):
    meta = pyds.NvDsEventMsgMeta.cast(data)
    meta.sensorId = 0
    meta.placeId = 0
    meta.moduleId = 0
    meta.sensorStr = "sensor-0"
    meta.ts = pyds.alloc_buffer(MAX_TIME_STAMP_LEN + 1)
    pyds.generate_ts_rfc3339(meta.ts, MAX_TIME_STAMP_LEN)

    print("meta.ts is: " +str(meta.ts))

    meta.type = pyds.NvDsEventType.NVDS_EVENT_CUSTOM
    meta.objType = pyds.NvDsObjectType.NVDS_OBJECT_TYPE_CUSTOM

    obj = pyds_custom_meta.alloc_my_object()
    obj = pyds_custom_meta.MyObject
    obj.number = 4

    print(obj)

    meta.extMsg = obj
    meta.extMsgSize = sys.getsizeof(obj)
    return obj

As said the code broke when the obj is assigned to the extMsg field of meta. From here, I am a bit lost and am not sure how should I procede.

Do you have some advice?

wassim_mohsni · July 6, 2021, 1:17pm

I’m facing the same issue! Did you resolve it?

recharles · July 7, 2021, 7:57am

Unfortunately not, I will give a deeper look in the coming days

F_sharp · August 5, 2021, 9:59am

you can follow [DeepStream SDK FAQ - #16 by bcao] && [GitHub - NVIDIA-AI-IOT/deepstream-occupancy-analytics: This is a sample application for counting people entering/leaving in a building using NVIDIA Deepstream SDK, Transfer Learning Toolkit (TLT), and pre-trained models. This application can be used to build real-time occupancy analytics applications for smart buildings, hospitals, retail, etc. The application is based on deepstream-test5 sample application.]

xtianhb.glb · August 31, 2021, 1:09am

Hi! @recharles @wassim_mohsni

Did you solved the TypeError: (): incompatible function arguments error for custom objects?

I use pyds for DeepStream standard code and pyds_test to bind my custom objects or classes.
I don’t know what else to try, I followed all links here.

Thanks.

import sys
sys.path.append('/opt/nvidia/deepstream/deepstream/lib')
import pyds
import pyds_test

meta = pyds.alloc_nvds_event_msg_meta()
meta.objType = pyds_test.NvDsObjectType.NVDS_OBJECT_TYPE_TEST

Invoked with: <pyds.NvDsEventMsgMeta object at 0x7f16a9f673b0>, <NvDsObjectType.NVDS_OBJECT_TYPE_TEST: 291>
root@host:/opt/nvidia/deepstream/deepstream/msgconv# python test.py 
Traceback (most recent call last):
  File "test.py", line 10, in <module>
    meta.objType = pyds_test.NvDsObjectType.NVDS_OBJECT_TYPE_TEST
TypeError: (): incompatible function arguments. The following argument types are supported:
    1. (self: pyds.NvDsEventMsgMeta, arg0: pyds.NvDsObjectType) -> None

wassim_mohsni · September 1, 2021, 9:19am

Unfortunately, I didn’t solve the error, I had to send my data in the NvDsVehicleObject. I hope issues like this get addressed in Deepstream 6.0 because this is honestly a bit too rigid.