Adding new variables to NvDsVehicleObject Struct Reference

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
GPU
• DeepStream Version
6.4
• JetPack Version (valid for Jetson only)
• TensorRT Version
TensorRT 8.6.1.6
• NVIDIA GPU Driver Version (valid for GPU only)
535.171.04
• Issue Type( questions, new requirements, bugs)
Add a new variable to NvDsVehicleObject.
nvidia_schema.h

  • A new variable ImagePath was updated to this header file and compile it with deepstream_python_apps for .whl file to install for python.
  • Futhermore, I have updated the new variable in eventmsg_payload.cpp to push the json format to kafka
  • There’s no error after I run the code with new variable added in, but the json was not updated to kafka. I have tried to change either of the name of the object is not updated as well.
  • Is there any method could update the code for the my application?
    • How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
    • Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)
NvDsVehicleObjectExt *dsObj = (NvDsVehicleObjectExt *)meta->extMsg;
      if (dsObj)
      {
        json_object_set_string_member(jobject, "type", dsObj->imagePath);
        json_object_set_string_member(jobject, "make", dsObj->make);
        json_object_set_string_member(jobject, "model", dsObj->model);
        json_object_set_string_member(jobject, "color", dsObj->color);
        json_object_set_string_member(jobject, "licenseState", dsObj->region);
        json_object_set_string_member(jobject, "license", dsObj->license);
        json_object_set_string_member(jobject, "imagePath", dsObj->imagePath);
        json_object_set_double_member(jobject, "confidence", meta->confidence);

        objectMask = dsObj->mask;
      }

This is the json output in kafka

{
    "messageid": "9ecced89-d1f9-424b-aa5b-eb999bdaba07",
    "mdsversion": "1.0",
    "@timestamp": "2024-05-09T08:01:07.635Z",
    "place": {
        "id": "1",
        "name": "XYZ",
        "type": "garage",
        "location": {"lat": 35.32, "lon": -40.55, "alt": 100.0},
        "aisle": {
            "id": "walsh",
            "name": "lane1",
            "level": "P2",
            "coordinate": {"x": 1.0, "y": 2.0, "z": 3.0},
        },
    },
    "sensor": {
        "id": "1",
        "type": "Camera",
        "description": '"JALAN AWAN BESAR"',
        "location": {"lat": 45.293701447, "lon": -75.8303914499, "alt": 48.1557479338},
        "coordinate": {"x": 5.2, "y": 10.1, "z": 11.2},
    },
    "analyticsModule": {
        "id": "XYZ",
        "description": '"Vehicle Detection and License Plate Recognition"',
        "source": "OpenALR",
        "version": "1.0",
    },
    "object": {
        "id": "8",
        "speed": 0.0,
        "direction": 0.0,
        "orientation": 0.0,
        "vehicle": {
            "type": "Vehicle",
            "make": "Bugatti",
            "model": "M",
            "color": "blue",
            "licenseState": "CA",
            "license": "VCQ8701",
            "confidence": 0.0,
        },
        "bbox": {
            "topleftx": 243,
            "toplefty": 358,
            "bottomrightx": 474,
            "bottomrighty": 635,
        },
        "location": {"lat": 0.0, "lon": 0.0, "alt": 0.0},
        "coordinate": {"x": 0.0, "y": 0.0, "z": 0.0},
        "pose": {},
    },
    "event": {"id": "e96f589b-2c50-47f5-a39c-a4ca81de4809", "type": "moving"},
    "videoPath": "/home/wesenkhoo/Documents/ITMAX/deepstream_ws/test/deepstream_lpr/new_ds/deepstream_lpr/images/VCQ8701.jpg",
}

Did you use the code snippet of generate_vehicle_meta in deepstream-test4.py ?

Have you added the corresponding python bindings in the pyds.

Generally speaking, you need to modify the following two codes

Forgot to mention that I have changed everything that contains NvDsVehicleObject in the library as well.

py::class_<NvDsVehicleObject>(m, "NvDsVehicleObject",
                                pydsdoc::metaschema::VehicleObjectDoc::descr)
    .def(py::init<>())
    .def_property("type", STRING_PROPERTY(NvDsVehicleObject, type))
    .def_property("make", STRING_PROPERTY(NvDsVehicleObject, make))
    .def_property("model",
                    STRING_PROPERTY(NvDsVehicleObject, model))
    .def_property("color",
                    STRING_PROPERTY(NvDsVehicleObject, color))

    .def(
        "cast",
        [](void *data)
        {
            return (NvDsVehicleObject *)data;
        },
        py::return_value_policy::reference,
        pydsdoc::metaschema::VehicleObjectDoc::cast)

    .def(
        "cast",
        [](size_t data)
        {
            return (NvDsVehicleObject *)data;
        },
        py::return_value_policy::reference,
        pydsdoc::metaschema::VehicleObjectDoc::cast)

    .def_property("region",
                    STRING_PROPERTY(NvDsVehicleObject, region))
    .def_property("imagePath",
                    STRING_PROPERTY(NvDsVehicleObject, imagePath))
    .def_property("license",
                    STRING_PROPERTY(NvDsVehicleObject, license));
NvDsVehicleObject *obj = (NvDsVehicleObject *)srcData->extMsg;
    if (obj->type)
        g_free(obj->type);
    if (obj->color)
        g_free(obj->color);
    if (obj->make)
        g_free(obj->make);
    if (obj->model)
        g_free(obj->model);
    if (obj->license)
        g_free(obj->license);
    if (obj->imagePath)
        g_free(obj->imagePath);
    if (obj->region)
        g_free(obj->region);

Since you didn’t provide the code, I don’t know where you went wrong. Use NvDsVehicleObject to replace NvDsVehicleObjectExt

Please refer to the patch below. After my test, it works normally. I use deepstream_test_4.py.

python3 deepstream_test_4.py -i /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264 -p /opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so --no-display --conn-str="localhost;9092;test"

for deepstream_python_apps

diff --git a/apps/deepstream-test4/deepstream_test_4.py b/apps/deepstream-test4/deepstream_test_4.py
index 22675a5..5119055 100755
--- a/apps/deepstream-test4/deepstream_test_4.py
+++ b/apps/deepstream-test4/deepstream_test_4.py
@@ -61,6 +61,7 @@ def generate_vehicle_meta(data):
     obj.model = "M"
     obj.license = "XX1234"
     obj.region = "CA"
+    obj.imagePath = "test.jpg"
     return obj
 
 
@@ -185,7 +186,7 @@ def osd_sink_pad_buffer_probe(pad, info, u_data):
             # Ideally NVDS_EVENT_MSG_META should be attached to buffer by the
             # component implementing detection / recognition logic.
             # Here it demonstrates how to use / attach that meta data.
-            if is_first_object and (frame_number % 30) == 0:
+            if is_first_object and obj_meta.class_id == PGIE_CLASS_ID_VEHICLE and (frame_number % 30) == 0:
                 # Frequency of messages to be send will be based on use case.
                 # Here message is being sent for first object every 30 frames.
 
diff --git a/bindings/src/bindschema.cpp b/bindings/src/bindschema.cpp
index 7085d3d..1f2858e 100644
--- a/bindings/src/bindschema.cpp
+++ b/bindings/src/bindschema.cpp
@@ -81,6 +81,8 @@ namespace pydeepstream {
                 obj->license = g_strdup (srcObj->license);
                 if (srcObj->region)
                 obj->region = g_strdup (srcObj->region);
+                if (srcObj->imagePath)
+                    obj->imagePath = g_strdup (srcObj->imagePath);
                 destData->extMsg = obj;
                 destData->extMsgSize = sizeof (NvDsVehicleObject);
             } else if (srcData->objType == NVDS_OBJECT_TYPE_PERSON) {
@@ -146,6 +148,8 @@ namespace pydeepstream {
                             g_free (obj->license);
                         if (obj->region)
                             g_free (obj->region);
+                        if (obj->imagePath)
+                            g_free (obj->imagePath);
                     } else if (srcData->objType == NVDS_OBJECT_TYPE_PERSON) {
                         NvDsPersonObject *obj = (NvDsPersonObject *) srcData->extMsg;
                         if (obj->gender)
@@ -353,7 +357,9 @@ namespace pydeepstream {
                 .def_property("region",
                               STRING_PROPERTY(NvDsVehicleObject, region))
                 .def_property("license",
-                              STRING_PROPERTY(NvDsVehicleObject, license));
+                              STRING_PROPERTY(NvDsVehicleObject, license))
+                .def_property("imagePath",
+                              STRING_PROPERTY(NvDsVehicleObject, imagePath));
 
         m.def("alloc_nvds_vehicle_object",
               []() {

for nvdsmeta_schema.h and eventmsg_payload.cpp

diff --git a/sources/includes/nvdsmeta_schema.h b/sources/includes/nvdsmeta_schema.h
index 47f86c3..0814b81 100644
--- a/sources/includes/nvdsmeta_schema.h
+++ b/sources/includes/nvdsmeta_schema.h
@@ -156,6 +156,7 @@ typedef struct NvDsVehicleObject {
   gchar *color;     /**< Holds a pointer to the color of the vehicle. */
   gchar *region;    /**< Holds a pointer to the region of the vehicle. */
   gchar *license;   /**< Holds a pointer to the license number of the vehicle.*/
+  gchar *imagePath;
 } NvDsVehicleObject;
 
 /**
diff --git a/sources/libs/nvmsgconv/deepstream_schema/eventmsg_payload.cpp b/sources/libs/nvmsgconv/deepstream_schema/eventmsg_payload.cpp
index d345289..0d97c1c 100644
--- a/sources/libs/nvmsgconv/deepstream_schema/eventmsg_payload.cpp
+++ b/sources/libs/nvmsgconv/deepstream_schema/eventmsg_payload.cpp
@@ -402,6 +402,7 @@ generate_object_object (void *privData, NvDsEventMsgMeta *meta)
           json_object_set_string_member (jobject, "color", dsObj->color);
           json_object_set_string_member (jobject, "licenseState", dsObj->region);
           json_object_set_string_member (jobject, "license", dsObj->license);
+          json_object_set_string_member (jobject, "imagePath", dsObj->imagePath);
           json_object_set_double_member (jobject, "confidence", meta->confidence);
         }
       } else {
@@ -412,6 +413,7 @@ generate_object_object (void *privData, NvDsEventMsgMeta *meta)
         json_object_set_string_member (jobject, "color", "");
         json_object_set_string_member (jobject, "licenseState", "");
         json_object_set_string_member (jobject, "license", "");
+        json_object_set_string_member (jobject, "imagePath", "");
         json_object_set_double_member (jobject, "confidence", 1.0);
       }
       json_object_set_object_member (objectObj, "vehicle", jobject);

Here is my result

"vehicle" : {
      "type" : "sedan",
      "make" : "Bugatti",
      "model" : "M",
      "color" : "blue",
      "licenseState" : "CA",
      "license" : "XX1234",
      "imagePath" : "test.jpg",
      "confidence" : 0.20320838689804077
    },

Mornin junshengy, I have done the exact same step as you shown before but I gave it a new try. Unfortunately, I couldn’t get the result as well. When I change the string name for instance, “licenseState” to a new variable the result isn’t updated to result as well. Therefore, I assumed that my code wasn’t update but the bindschema.cpp. Is there any ways to update deepstream eventpayload code?

If the installation is correct and confirm that the above patch has been applied, try the following steps.

1.reinstall nvmsgconv

cd /opt/nvidia/deepstream/deepstream/sources/libs/nvmsgconv

make CUDA_VER=12.2 clean  

make CUDA_VER=12.2 install
  1. reinstall pyds:
cd /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/build

make

pip uninstall pyds

pip install ./pyds-1.1.11-py3-none-linux_x86_64.whl # I'm using DeepStream7.0 and I think DS-6.4 is the same

3.run app with kafka.

cd /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/apps/deepstream-test4

python3 deepstream_test_4.py -i /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264 -p /opt/nvidia/deepstream/deepstream/lib/libnvds_kafka_proto.so --no-display --conn-str="localhost;9092;test"

If it still doesn’t work correctly, please check your code

msgconv.set_property('config', MSCONV_CONFIG_FILE)
AttributeError: 'NoneType' object has no attribute 'set_property'

Did you face any error about the msgconv after I rebuild the lib?

No, this problem has nothing to do with nvmsgconv.

It is caused by the related dependencies of pyds not being installed correctly.

(python3:23238): GStreamer-WARNING **: 11:00:57.224: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_msgconv.so': ///opt/nvidia/deepstream/deepstream-6.4/lib/libnvds_msgconv.so: undefined symbol: _ZN4YAML6detail9node_data12empty_scalarB5cxx11E

but it shows an error about it

Please follow the README in the /opt/nvidia/deepstream/deepstream/sources/libs/nvmsgconv

sudo apt-get install libglib2.0-dev libjson-glib-dev uuid-dev libyaml-cpp-dev protobuf-compiler
(python3:26188): GStreamer-CRITICAL **: 11:25:19.259: gst_element_factory_create_valist: assertion 'newfactory != NULL' failed
 Unable to create msgconv 

This happened after I rebuild the .so library

msgconv = Gst.ElementFactory.make("nvmsgconv", "nvmsg-converter")
    if not msgconv:
        sys.stderr.write(" Unable to create msgconv \n")

This error was shown while creating the msgconv

BTW, the packages was installed

Due to lack of dependencies before, nvmsgconv is in gstreamer’s blacklist.

gst-inspect-1.0 -b can view the blacklist of plugins

try

sudo /opt/nvidia/deepstream/deepstream/install.sh

It is recommended that you use docker to avoid various dependency issues

Blacklisted files:
  libcustom2d_preprocess.so
  libgstmsdk.so
  libnvdsgst_deepstream_bins.so
  libnvdsgst_inferserver.so
  libnvdsgst_msgconv.so
  libnvdsgst_udp.so

Total count: 6 blacklisted files

I could see there’s 6 blacklisted file. Also,I have run install.sh but I receive the same issue as well

Check what dependencies you are missing

 ldd /opt/nvidia/deepstream/deepstream/lib/libnvds_msgconv.so
ldd /opt/nvidia/deepstream/deepstream-6.4/lib/libnvds_msgconv.so 
	linux-vdso.so.1 (0x00007ffe49ff8000)
	libjson-glib-1.0.so.0 => /lib/x86_64-linux-gnu/libjson-glib-1.0.so.0 (0x00007f9414709000)
	libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007f94145cf000)
	libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f94145c6000)
	libyaml-cpp.so.0.7 => /lib/x86_64-linux-gnu/libyaml-cpp.so.0.7 (0x00007f9414587000)
	libprotobuf.so.23 => /lib/x86_64-linux-gnu/libprotobuf.so.23 (0x00007f9414200000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f9413e00000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f9414565000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9413a00000)
	libgio-2.0.so.0 => /lib/x86_64-linux-gnu/libgio-2.0.so.0 (0x00007f9413826000)
	libgobject-2.0.so.0 => /lib/x86_64-linux-gnu/libgobject-2.0.so.0 (0x00007f9414505000)
	libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f941418a000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f94140a3000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f94148ba000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f94144e7000)
	libgmodule-2.0.so.0 => /lib/x86_64-linux-gnu/libgmodule-2.0.so.0 (0x00007f94144e0000)
	libmount.so.1 => /lib/x86_64-linux-gnu/libmount.so.1 (0x00007f941405f000)
	libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f9414033000)
	libffi.so.8 => /lib/x86_64-linux-gnu/libffi.so.8 (0x00007f94144d1000)
	libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1 (0x00007f9413dc9000)
	libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007f9413d32000)

Check this plugin to find missing libraries.

ldd /opt/nvidia/deepstream/deepstream/lib/gst-plugins/libnvdsgst_msgconv.so
ldd /opt/nvidia/deepstream/deepstream/lib/gst-plugins/libnvdsgst_msgconv.so
        linux-vdso.so.1 (0x00007fff5eee7000)
        libnvds_meta.so => ///opt/nvidia/deepstream/deepstream-6.4/lib/libnvds_meta.so (0x000077f8fe9a2000)
        libnvdsgst_meta.so => ///opt/nvidia/deepstream/deepstream-6.4/lib/libnvdsgst_meta.so (0x000077f8fe99b000)
        libnvdsgst_3d_gst.so => ///opt/nvidia/deepstream/deepstream-6.4/lib/libnvdsgst_3d_gst.so (0x000077f8fe98d000)
        libnvds_msgconv.so => ///opt/nvidia/deepstream/deepstream-6.4/lib/libnvds_msgconv.so (0x000077f8fe831000)
        libgstreamer-1.0.so.0 => /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0 (0x000077f8fe6b8000)
        libgstbase-1.0.so.0 => /lib/x86_64-linux-gnu/libgstbase-1.0.so.0 (0x000077f8fe633000)
        libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x000077f8fe4f9000)
        libgobject-2.0.so.0 => /lib/x86_64-linux-gnu/libgobject-2.0.so.0 (0x000077f8fe499000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x000077f8fe200000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x000077f8fe479000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000077f8fde00000)
        libnvdsgst_helper.so => ///opt/nvidia/deepstream/deepstream-6.4/lib/libnvdsgst_helper.so (0x000077f8fe46d000)
        libgstapp-1.0.so.0 => /lib/x86_64-linux-gnu/libgstapp-1.0.so.0 (0x000077f8fe457000)
        libjson-glib-1.0.so.0 => /lib/x86_64-linux-gnu/libjson-glib-1.0.so.0 (0x000077f8fe1d3000)
        libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x000077f8fe44e000)
        libyaml-cpp.so.0.7 => /lib/x86_64-linux-gnu/libyaml-cpp.so.0.7 (0x000077f8fe194000)
        libprotobuf.so.23 => /lib/x86_64-linux-gnu/libprotobuf.so.23 (0x000077f8fda00000)
        libgmodule-2.0.so.0 => /lib/x86_64-linux-gnu/libgmodule-2.0.so.0 (0x000077f8fe445000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000077f8fe0ad000)
        libunwind.so.8 => /lib/x86_64-linux-gnu/libunwind.so.8 (0x000077f8fe092000)
        libdw.so.1 => /lib/x86_64-linux-gnu/libdw.so.1 (0x000077f8fdd54000)
        libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x000077f8fdcde000)
        libffi.so.8 => /lib/x86_64-linux-gnu/libffi.so.8 (0x000077f8fe436000)
        /lib64/ld-linux-x86-64.so.2 (0x000077f8fe9b9000)
        libgstrtp-1.0.so.0 => /lib/x86_64-linux-gnu/libgstrtp-1.0.so.0 (0x000077f8fe061000)
        libgio-2.0.so.0 => /lib/x86_64-linux-gnu/libgio-2.0.so.0 (0x000077f8fd826000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x000077f8fe045000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x000077f8fd7fb000)
        libelf.so.1 => /lib/x86_64-linux-gnu/libelf.so.1 (0x000077f8fd7dd000)
        libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x000077f8fe032000)
        libmount.so.1 => /lib/x86_64-linux-gnu/libmount.so.1 (0x000077f8fd799000)
        libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x000077f8fd76d000)
        libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1 (0x000077f8fd736000)
        libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x000077f8fd69f000)

What is the output of the following command line ?

rm -rf ~/.cache/gstreamer-1.0/registry.x86_64.bin 

gst-inspect-1.0 nvmsgconv 

gst-inspect-1.0 nvmsgconv
No such element or plugin ‘nvmsgconv’