Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU) • DeepStream Version: 7.1 • JetPack Version (valid for Jetson only) • TensorRT Version • NVIDIA GPU Driver Version (valid for GPU only) • Issue Type: questions • How to reproduce the issue: deepstream-app • Requirement details: N/A
Using the deepstream-app (or deepstream-test4 or test5 apps), I am trying to perform bounding box detection in PGIE (nvinfer) followed by NvDCF tracker with ReID model. The goal is to collect analytics and output a message into an Apache Kafka topi. The following info for each bounding box detection has to be outputted as a message:
frame ID
track ID
timestamp
bounding box coordinate (top-left X, Y and bottom-right X, Y)
embeddings vector
I have set up deepstream-app configuration which invokes bounding box detection model and performs tracking with NvDCF ReID model. I am able to output messages to a Kafka topic with info like track ID, bbox coordinates and camera metadata. However, I am unable to also output the embeddings vector. I am able to write embeddings to a directory for each frame.
The source code for nvmsgconv (eventmsg_payload.cpp) shows a check for presence meta->embedding.embedding_vector. If successful, the embeddig values are concatenated into a string and added to a payload message.
From the documentation for nvinfer, enabling outputting reID vectors (setting “outputReidTensor: 1” for tracking in ReID section), the embedding vectors are apparently attached to the metabuffers for downstream component consumption. I have searched multiple source code files and examples but was unable to get the desired behavior.
My question is: how to output ReID embeddings in a message into a Kafka topic? Display of the video is not necessary, only analytics.
right, nvsmgconv will convert vector to string. first please refer to the nvmsgconv doc. if msg2p-newapi is 0, nvsmonv will get information from NvDsEventMsgMeta, you need to pass embedding values to NvDsEventMsgMeta, please refer to osd_sink_pad_buffer_metadata_probe of test4; if 1, nvsmonv will get information from NvDsFrameMeta, which already includes embedding values when outputReidTensor is 1.
Going with the option msg2p-newapi=1, as you explain the embeddings will be already part of NvDsFrameMeta structure. Looking at the nvmsgconv source code, it appears the relevant place embedding information could be read is nvmsgconv/deepstream_schema/dsmeta_payload.cpp. I am thinking adding another function in generate_dsmeta_message() which would take frame_meta (already available there), parse it and output a JSON element with embedding information. Is this a good approach? Or is there an approach where I need to modify only the test app? One reason I dislike this approach is that this requires modification of libnvds_msgconv.so library..
There are two methods to add embedding information when msg2p-newapi=1.
As you said, modify dsmeta_payload.cpp to customize.
if don’t want to modify the so, please refer to this doc Payload with Custom Objects. you can attach a custom message blob(string) to payload using NVDS_CUSTOM_MSG_BLOB type by creating NvDsCustomMsgInfo object and specifying the fields message (custom message blob) and len (size of the custom message).