Kafka payload from deepstream with model confidence score for each bbox

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU). GPU
• DeepStream Version. : 7.0
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

I finetuned detectnet V2 using Nvidia tao on my cutom data, I exported the model in INT8 format and integrated to the deepstream piepleine, I am able to do the inference and everything is working fine. I am getting annotated infernece video from the sink.

I have one question, I want to send the detection payload(detection results after inference) using kafka to a topic. I am able to do that but I am not able get the bounding box confidence score of each detected object bbox. Does deepstream support this functionality by default(sending Model confidence for each bbox via kafka payload),
if not what can I do to achieve this functionality.

thanks.

3

which sample are you testing or referring to? please refer to this doc for how to add custom objects.

4

Does any sample in the deepstream supports sending the payload with detection confidence score or do we have to make changes to nvmsgconv code?

5

deepstream-test5, deepstream-test4 supports sending broker. If msg2p-newapi=1 and payload-type=0, test5 supports sending cofidence to broker.
nvmsgconv is respensbile for covnerting payloads to Json string. nvmsgconv low-level library is opensource. please refer to generate_object_object, which will add obj_meta->confidence to Json string.

6

Okay I tested as you described,

Now I can get the confidence score but frame id is lost, earlier I was getting frame id when using payload-type: 1, below is one sample payload

{
  "version" : "4.0",
  "id" : "0",
  "@timestamp" : "2025-02-27T08:51:09.252Z",
  "sensorId" : "reception_cam",
  "objects" : [
    "18446744073709551615|638.484|566.368|710.177|728.787|marker_1",
    "18446744073709551615|459.094|558.484|525.037|732.772|marker_1"
  ]
}

here id refers to frame id as per my understanding,

but when using payload-type: 0

Now i am getting the confidence score sas shown in payload below but the frame level info is lost, i.e the messages now are on object level not frame level:

{
  "messageid" : "f119d301-67d0-4118-9967-ffbf775f1d75",
  "mdsversion" : "1.0",
  "@timestamp" : "2025-02-27T09:13:58.070Z",
  "place" : {
    "id" : "0",
    "name" : "",
    "type" : "centific_IDC",
    "location" : {
      "lat" : 0.0,
      "lon" : 0.0,
      "alt" : 0.0
    },
    "place-sub-field" : {
      "id" : "",
      "name" : "",
      "level" : "",
      "coordinate" : {
        "x" : 0.0,
        "y" : 0.0,
        "z" : 0.0
      }
    }
  },
  "sensor" : {
    "id" : "reception_cam",
    "type" : "Camera",
    "description" : "",
    "location" : {
      "lat" : 0.0,
      "lon" : 0.0,
      "alt" : 0.0
    },
    "coordinate" : {
      "x" : 0.0,
      "y" : 0.0,
      "z" : 0.0
    }
  },
  "analyticsModule" : {
    "id" : "dwell_time",
    "description" : "",
    "source" : "",
    "version" : "1.0"
  },
  "object" : {
    "id" : "18446744073709551615",
    "speed" : 0.0,
    "direction" : 0.0,
    "orientation" : 0.0,
    "marker_1" : {
      "confidence" : 0.63341867923736572
    },
    "bbox" : {
      "topleftx" : 459,
      "toplefty" : 558,
      "bottomrightx" : 525,
      "bottomrighty" : 732
    },
    "location" : {    },
    "coordinate" : {    }
  },
  "event" : {
    "id" : "98f65235-82fb-468a-8696-a1c5c3a448d5",
    "type" : ""
  },
  "videoPath" : ""
}

Can we get the frame id along with this message. The issue I am facing here is that after reading the messges from the kafka I dont know for whcih frame whcih object belogs to.

7

if you want to add the custom data. please refer to the link in my first comment. If msg2p-newapi=1, you can attach a custom message blob(string) to payload using NVDS_CUSTOM_MSG_BLOB type by creating NvDsCustomMsgInfo object and specifying the fields message (custom message blob) and len (size of the custom message).
Or /opt/nvidia/deepstream/deepstream/sources/libs/nvmsgconv/ is opensource. you can modify the code to customize.