Gst-nvmsgconv: Payload with multiple object info

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

• Hardware Platform (Jetson / GPU) Jetson
• DeepStream Version 6.2
• JetPack Version (valid for Jetson only) 5.1
• Issue Type( questions, new requirements, bugs) Question

I have been working to build a custom payload for nvmsgconv by modifying dsmeta_payload.cpp where I have information about multiple objects in one payload/message. Per the overview, by default “Each payload has information about a single object”: Gst-nvmsgconv — DeepStream documentation 6.4 documentation (nvidia.com)

I would like to have my message/payload to have information about all detected objects, so I added an array of all the objects to my message (within a sensor object). This works, but my problem is I am getting multiple redundant messages per frame and I only want one message per frame. It seems that nvds_msg2p_generate_new is being called multiple times for the same frame. It’s not clear to me what the best/simplest approach is to generate only a single message, containing the info. for multiple objects. Any suggestions? A sample of two consecutive messages below:
{
“messageid” : “4595bb20-6db8-43cb-9aba-dda14abe947a”,
“mdsversion” : “1.0”,
“@timestamp” : “2024-06-10T22:44:34.421Z”,
“sensor” : {
“id” : “Dummy_hostname”,
“frame number” : 460968,
“lines” : [
{
“label” : “out-2”,
“was crossed” : 0,
“tally” : “0”
},
{
“label” : “in-1”,
“was crossed” : 0,
“tally” : “0”
}
],
“objects” : [
{
“id” : “5169”,
“class” : 0,
“x” : -44.268905639648438,
“y” : 553.148681640625,
“width” : 659.816162109375,
“height” : 307.33795166015625,
“type_id” : 5,
“isCrossed” : “false”,
“line” : “”,
“line_id” : -1
},
{
“id” : “6745”,
“class” : 0,
“x” : 1679.038818359375,
“y” : 515.64447021484375,
“width” : 242.6920166015625,
“height” : 160.63304138183594,
“type_id” : -1,
“isCrossed” : “false”,
“line” : “”,
“line_id” : -1
},
{
“id” : “6720”,
“class” : 0,
“x” : 880.9969482421875,
“y” : 395.63330078125,
“width” : 69.39678955078125,
“height” : 34.887428283691406,
“type_id” : -1,
“isCrossed” : “false”,
“line” : “”,
“line_id” : -1
},
{
“id” : “6707”,
“class” : 0,
“x” : 922.83306884765625,
“y” : 402.208740234375,
“width” : 101.13299560546875,
“height” : 38.906578063964844,
“type_id” : -1,
“isCrossed” : “false”,
“line” : “”,
“line_id” : -1
},
{
“id” : “4438”,
“class” : 0,
“x” : 1285.799560546875,
“y” : 467.8369140625,
“width” : 169.99237060546875,
“height” : 68.274452209472656,
“type_id” : -1,
“isCrossed” : “false”,
“line” : “”,
“line_id” : -1
}
]
}
}
{
“messageid” : “1977cfab-1977-48ea-ac71-8ffdab8b09d2”,
“mdsversion” : “1.0”,
“@timestamp” : “2024-06-10T22:44:34.420Z”,
“sensor” : {
“id” : “Dummy_hostname”,
“frame number” : 460968,
“lines” : [
{
“label” : “out-2”,
“was crossed” : 0,
“tally” : “0”
},
{
“label” : “in-1”,
“was crossed” : 0,
“tally” : “0”
}
],
“objects” : [
{
“id” : “5169”,
“class” : 0,
“x” : -44.268905639648438,
“y” : 553.148681640625,
“width” : 659.816162109375,
“height” : 307.33795166015625,
“type_id” : 5,
“isCrossed” : “false”,
“line” : “”,
“line_id” : -1
},
{
“id” : “6745”,
“class” : 0,
“x” : 1679.038818359375,
“y” : 515.64447021484375,
“width” : 242.6920166015625,
“height” : 160.63304138183594,
“type_id” : -1,
“isCrossed” : “false”,
“line” : “”,
“line_id” : -1
},
{
“id” : “6720”,
“class” : 0,
“x” : 880.9969482421875,
“y” : 395.63330078125,
“width” : 69.39678955078125,
“height” : 34.887428283691406,
“type_id” : -1,
“isCrossed” : “false”,
“line” : “”,
“line_id” : -1
},
{
“id” : “6707”,
“class” : 0,
“x” : 922.83306884765625,
“y” : 402.208740234375,
“width” : 101.13299560546875,
“height” : 38.906578063964844,
“type_id” : -1,
“isCrossed” : “false”,
“line” : “”,
“line_id” : -1
},
{
“id” : “4438”,
“class” : 0,
“x” : 1285.799560546875,
“y” : 467.8369140625,
“width” : 169.99237060546875,
“height” : 68.274452209472656,
“type_id” : -1,
“isCrossed” : “false”,
“line” : “”,
“line_id” : -1
}
]
}
}

3

nvmsgconv plugin is opensource. In gst_nvmsgconv_transform_ip_video_audio of \opt\nvidia\deepstream\deepstream\sources\gst-plugins\gst-nvmsgconv\gstnvmsgconv.cpp, msg2p_generate_new will be called many times because it is in the visiting loop of frame_meta_list.

4

Indeed it is – fair enough. But given that your documentation says “Each payload can have information for multiple objects in the frame,” I was wondering if there is a recommended way to do that. I can tweak the source all the way back into gstnvmsgconv.cpp plugin, but I’m not sure that’s the best way. I am inclined to just filter out redundant messages in dsmeta_payload.cpp but that seems inelegant.

5

msg2p_generate_new will be called many times because it is in the visiting loop of frame_meta_list. there maybe many frames in a batch, but in your log, the frame number is the same, how did you distinguish the different frames? did you find the root cause of “I am getting multiple redundant messages per frame”? could you share the code diff and configuration file? You can use forum private email for the private information. please click forum avatar-> personal messages->new message.

6

if setting “msg-conv-payload-type: 1”, each payload can have information for multiple objects in the frame. please refer to loglog.txt (1.2 KB). if you want to add other information, you can modify the opensource code \opt\nvidia\deepstream\deepstream-\sources\libs\nvmsgconv\deepstream_schema\dsmeta_payload.cpp.

7

Thanks for the info. I am only able to allocate a small fraction of my time to this problem each day, so I am making slow progress. I think I may have resolved it to my satisfaction now though.

To clarify, my issue wasn’t that I was unable to have multiple objects per frame message, it’s that I have redundant messages when I do this. I traced it to the gstnvmsgconv.c code, where there are nested loops, with the outer one iterating through batch frames and the inner one iterating through every object in that frame (see line ~609 onward). Then within that inner loop, msg2p_generate_new is being called each time.

At first I just tried deleting the inner for loop, but I must have made a mistake somewhere, as I was getting a segmentation fault. So in the end I just changed obj_l = obj_l->next to obj_l = NULL in the loop definition, so that it only goes through once. It seems to work as I wanted now.

8

Thanks for the sharing! closing this topic.

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.