Payload error from nvmsgconv

• Hardware Platform GPU
• DeepStream Version 6.0.1
• TensorRT Version 8
• NVIDIA GPU Driver Version (valid for GPU only) 470
• Issue Type bug

Hi there.
As I’m trying to write a mqtt protocal adaptor, I found a repo and it almost works.

The only problem is sometimes, the payload get some random characters at the tail like this

So I checked the codes, they are almost the same to the kafka_protocol_adaptor. I add a output at nvds_msgapi_send_async of proto.cpp. The broker using this custom mqtt proto seems to get wrong payload at the very first time.

Since the payload generated at the nvmsgconv is correct( I also output the payload there.) and the broker is not opensource, could you provide some advice on this?

thanks a lot

Hi, finally solve the problem. I figured out IT IS CAUSED by nvmsgconv, this may be a deepstream bug:

in the function nds_msg2p_generate from /opt/nvidia/deepstream/deepstream/sources/libs/nvmsgconv/nvmsgconv.cpp, it get the length of message by strlen, and in function nds_msg2p_generate, there is an annotation:

Remove '\0' character at the end of string and just copy the content

I don’t know the purpose of this operation, but that’s a problem, c/c++ use ‘\0’ to get the end of a string. I guess these random characters are because of this reason.

So I use len+1 instead of len when use g_memdup() to copy message to payload. And it worked.

I don’t know why remove ‘\0’ to cause this. Can you give me some hints.

thx a lot.

I tried amqp broker, but i can not reproduce your issue, got random characters at the message body tail. i do not think it have relation with specific protocal broker, anyway, i will try to use kafka broker to see if can repro your issue.

Yes, it’s not because of protocal, as i said, it’s because of nvmsgconv.

I didn’t try azure mqtt, bu I tried kafka before, the problem still exists. in the beginning, the messages received by consumer client are always correct. However, after maybe a dozen, the characters came up.

I also checked these “random” characters, actually, they are not. they maybe a word from code, may be a number from other message.

I’ll upload my codes later.

thanks for your help anyway.

I tried kafka broker, but i can not reproduce your issue. as said, it’s not broker issue. and the original msgconv and broker do not have this issue. it’s related with your code.

{
“messageid” : “e24cbffd-9d8c-4691-915b-f28d6940f5f9”,
“mdsversion” : “1.0”,
@timestamp” : “2022-04-25T04:16:12.897Z”,
“place” : {
“id” : “1”,
“name” : “XYZ”,
“type” : “garage”,
“location” : {
“lat” : 30.32,
“lon” : -40.549999999999997,
“alt” : 100.0
},
“aisle” : {
“id” : “walsh”,
“name” : “lane1”,
“level” : “P2”,
“coordinate” : {
“x” : 1.0,
“y” : 2.0,
“z” : 3.0
}
}
},
“sensor” : {
“id” : “CAMERA_ID”,
“type” : “Camera”,
“description” : ““Entrance of Garage Right Lane””,
“location” : {
“lat” : 45.293701446999997,
“lon” : -75.830391449900006,
“alt” : 48.155747933800001
},
“coordinate” : {
“x” : 5.2000000000000002,
“y” : 10.1,
“z” : 11.199999999999999
}
},
“analyticsModule” : {
“id” : “XYZ”,
“description” : ““Vehicle Detection and License Plate Recognition””,
“source” : “OpenALR”,
“version” : “1.0”
},
“object” : {
“id” : “18446744073709551615”,
“speed” : 0.0,
“direction” : 0.0,
“orientation” : 0.0,
“vehicle” : {
“type” : “sedan”,
“make” : “Bugatti”,
“model” : “M”,
“color” : “blue”,
“licenseState” : “CA”,
“license” : “XX1234”,
“confidence” : 0.89570862054824829
},
“bbox” : {
“topleftx” : 880,
“toplefty” : 481,
“bottomrightx” : 994,
“bottomrighty” : 548
},
“location” : {
“lat” : 0.0,
“lon” : 0.0,
“alt” : 0.0
},
“coordinate” : {
“x” : 0.0,
“y” : 0.0,
“z” : 0.0
}
},
“event” : {
“id” : “14882659-92c6-48b8-a0d4-29b5189780cb”,
“type” : “moving”
},
“videoPath” : “”
}
{
“messageid” : “e6adbc9e-a02b-4fad-8ec3-c5d0f14b4522”,
“mdsversion” : “1.0”,
@timestamp” : “2022-04-25T04:16:13.897Z”,
“place” : {
“id” : “1”,
“name” : “XYZ”,
“type” : “garage”,
“location” : {
“lat” : 30.32,
“lon” : -40.549999999999997,
“alt” : 100.0
},
“aisle” : {
“id” : “walsh”,
“name” : “lane1”,
“level” : “P2”,
“coordinate” : {
“x” : 1.0,
“y” : 2.0,
“z” : 3.0
}
}
},
“sensor” : {
“id” : “CAMERA_ID”,
“type” : “Camera”,
“description” : ““Entrance of Garage Right Lane””,
“location” : {
“lat” : 45.293701446999997,
“lon” : -75.830391449900006,
“alt” : 48.155747933800001
},
“coordinate” : {
“x” : 5.2000000000000002,
“y” : 10.1,
“z” : 11.199999999999999
}
},
“analyticsModule” : {
“id” : “XYZ”,
“description” : ““Vehicle Detection and License Plate Recognition””,
“source” : “OpenALR”,
“version” : “1.0”
},
“object” : {
“id” : “18446744073709551615”,
“speed” : 0.0,
“direction” : 0.0,
“orientation” : 0.0,
“vehicle” : {
“type” : “sedan”,
“make” : “Bugatti”,
“model” : “M”,
“color” : “blue”,
“licenseState” : “CA”,
“license” : “XX1234”,
“confidence” : 0.81706953048706055
},
“bbox” : {
“topleftx” : 609,
“toplefty” : 484,
“bottomrightx” : 709,
“bottomrighty” : 565
},
“location” : {
“lat” : 0.0,
“lon” : 0.0,
“alt” : 0.0
},
“coordinate” : {
“x” : 0.0,
“y” : 0.0,
“z” : 0.0
}
},
“event” : {
“id” : “81b7ed95-f263-43db-b389-4e3e4c5bfb96”,
“type” : “moving”
},
“videoPath” : “”
}
{
“messageid” : “34b19296-8ff4-4064-890e-2cc8fdd897f8”,
“mdsversion” : “1.0”,
@timestamp” : “2022-04-25T04:16:14.897Z”,
“place” : {
“id” : “1”,
“name” : “XYZ”,
“type” : “garage”,
“location” : {
“lat” : 30.32,
“lon” : -40.549999999999997,
“alt” : 100.0
},
“aisle” : {
“id” : “walsh”,
“name” : “lane1”,
“level” : “P2”,
“coordinate” : {
“x” : 1.0,
“y” : 2.0,
“z” : 3.0
}
}
},
“sensor” : {
“id” : “CAMERA_ID”,
“type” : “Camera”,
“description” : ““Entrance of Garage Right Lane””,
“location” : {
“lat” : 45.293701446999997,
“lon” : -75.830391449900006,
“alt” : 48.155747933800001
},
“coordinate” : {
“x” : 5.2000000000000002,
“y” : 10.1,
“z” : 11.199999999999999
}
},
“analyticsModule” : {
“id” : “XYZ”,
“description” : ““Vehicle Detection and License Plate Recognition””,
“source” : “OpenALR”,
“version” : “1.0”
},
“object” : {
“id” : “18446744073709551615”,
“speed” : 0.0,
“direction” : 0.0,
“orientation” : 0.0,
“vehicle” : {
“type” : “sedan”,
“make” : “Bugatti”,
“model” : “M”,
“color” : “blue”,
“licenseState” : “CA”,
“license” : “XX1234”,
“confidence” : 0.95825237035751343
},
“bbox” : {
“topleftx” : 1068,
“toplefty” : 485,
“bottomrightx” : 1248,
“bottomrighty” : 570
},
“location” : {
“lat” : 0.0,
“lon” : 0.0,
“alt” : 0.0
},
“coordinate” : {
“x” : 0.0,
“y” : 0.0,
“z” : 0.0
}
},
“event” : {
“id” : “2f3412d0-07df-45c8-a801-bff2056bd5c1”,
“type” : “moving”
},
“videoPath” : “”
}
{
“messageid” : “4489bc57-3bc8-4811-a756-bd11566418ab”,
“mdsversion” : “1.0”,
@timestamp” : “2022-04-25T04:16:15.897Z”,
“place” : {
“id” : “1”,
“name” : “XYZ”,
“type” : “garage”,
“location” : {
“lat” : 30.32,
“lon” : -40.549999999999997,
“alt” : 100.0
},
“aisle” : {
“id” : “walsh”,
“name” : “lane1”,
“level” : “P2”,
“coordinate” : {
“x” : 1.0,
“y” : 2.0,
“z” : 3.0
}
}
},
“sensor” : {
“id” : “CAMERA_ID”,
“type” : “Camera”,
“description” : ““Entrance of Garage Right Lane””,
“location” : {
“lat” : 45.293701446999997,
“lon” : -75.830391449900006,
“alt” : 48.155747933800001
},
“coordinate” : {
“x” : 5.2000000000000002,
“y” : 10.1,
“z” : 11.199999999999999
}
},
“analyticsModule” : {
“id” : “XYZ”,
“description” : ““Vehicle Detection and License Plate Recognition””,
“source” : “OpenALR”,
“version” : “1.0”
},
“object” : {
“id” : “18446744073709551615”,
“speed” : 0.0,
“direction” : 0.0,
“orientation” : 0.0,
“vehicle” : {
“type” : “sedan”,
“make” : “Bugatti”,
“model” : “M”,
“color” : “blue”,
“licenseState” : “CA”,
“license” : “XX1234”,
“confidence” : 0.88542348146438599
},
“bbox” : {
“topleftx” : 857,
“toplefty” : 480,
“bottomrightx” : 954,
“bottomrighty” : 540
},
“location” : {
“lat” : 0.0,
“lon” : 0.0,
“alt” : 0.0
},
“coordinate” : {
“x” : 0.0,
“y” : 0.0,
“z” : 0.0
}
},
“event” : {
“id” : “32303cde-7e2d-4e63-9540-1f70f6691e9b”,
“type” : “moving”
},
“videoPath” : “”
}
{
“messageid” : “6b27c138-5224-48f7-b543-b8742f8d08e4”,
“mdsversion” : “1.0”,
@timestamp” : “2022-04-25T04:16:16.897Z”,
“place” : {
“id” : “1”,
“name” : “XYZ”,
“type” : “garage”,
“location” : {
“lat” : 30.32,
“lon” : -40.549999999999997,
“alt” : 100.0
},
“aisle” : {
“id” : “walsh”,
“name” : “lane1”,
“level” : “P2”,
“coordinate” : {
“x” : 1.0,
“y” : 2.0,
“z” : 3.0
}
}
},
“sensor” : {
“id” : “CAMERA_ID”,
“type” : “Camera”,
“description” : ““Entrance of Garage Right Lane””,
“location” : {
“lat” : 45.293701446999997,
“lon” : -75.830391449900006,
“alt” : 48.155747933800001
},
“coordinate” : {
“x” : 5.2000000000000002,
“y” : 10.1,
“z” : 11.199999999999999
}
},
“analyticsModule” : {
“id” : “XYZ”,
“description” : ““Vehicle Detection and License Plate Recognition””,
“source” : “OpenALR”,
“version” : “1.0”
},
“object” : {
“id” : “18446744073709551615”,
“speed” : 0.0,
“direction” : 0.0,
“orientation” : 0.0,
“vehicle” : {
“type” : “sedan”,
“make” : “Bugatti”,
“model” : “M”,
“color” : “blue”,
“licenseState” : “CA”,
“license” : “XX1234”,
“confidence” : 0.90123236179351807
},
“bbox” : {
“topleftx” : 552,
“toplefty” : 478,
“bottomrightx” : 621,
“bottomrighty” : 536
},
“location” : {
“lat” : 0.0,
“lon” : 0.0,
“alt” : 0.0
},
“coordinate” : {
“x” : 0.0,
“y” : 0.0,
“z” : 0.0
}
},
“event” : {
“id” : “52f66969-8793-446b-ac46-3f7c6515b4a4”,
“type” : “moving”
},
“videoPath” : “”
}
{
“messageid” : “ffb373fe-68dd-4e55-a462-5a6129bc95b2”,
“mdsversion” : “1.0”,
@timestamp” : “2022-04-25T04:16:17.897Z”,
“place” : {
“id” : “1”,
“name” : “XYZ”,
“type” : “garage”,
“location” : {
“lat” : 30.32,
“lon” : -40.549999999999997,
“alt” : 100.0
},
“aisle” : {
“id” : “walsh”,
“name” : “lane1”,
“level” : “P2”,
“coordinate” : {
“x” : 1.0,
“y” : 2.0,
“z” : 3.0
}
}
},
“sensor” : {
“id” : “CAMERA_ID”,
“type” : “Camera”,
“description” : ““Entrance of Garage Right Lane””,
“location” : {
“lat” : 45.293701446999997,
“lon” : -75.830391449900006,
“alt” : 48.155747933800001
},
“coordinate” : {
“x” : 5.2000000000000002,
“y” : 10.1,
“z” : 11.199999999999999
}
},
“analyticsModule” : {
“id” : “XYZ”,
“description” : ““Vehicle Detection and License Plate Recognition””,
“source” : “OpenALR”,
“version” : “1.0”
},
“object” : {
“id” : “18446744073709551615”,
“speed” : 0.0,
“direction” : 0.0,
“orientation” : 0.0,
“vehicle” : {
“type” : “sedan”,
“make” : “Bugatti”,
“model” : “M”,
“color” : “blue”,
“licenseState” : “CA”,
“license” : “XX1234”,
“confidence” : 0.89527261257171631
},
“bbox” : {
“topleftx” : 1652,
“toplefty” : 485,
“bottomrightx” : 1838,
“bottomrighty” : 556
},
“location” : {
“lat” : 0.0,
“lon” : 0.0,
“alt” : 0.0
},
“coordinate” : {
“x” : 0.0,
“y” : 0.0,
“z” : 0.0
}
},
“event” : {
“id” : “797778e2-f622-4106-bbb3-ce295f6248e7”,
“type” : “moving”
},
“videoPath” : “”
}

I don’t think that’s the root of your issue.
I have a custom msgconv based on the DS one and both original msgconv and my one work correctly with that bit in place.

I might be wrong but they chop off the trailing “\0” because it marks end of sting within a memory buffer.
As soon as we return payload length with the payload (payload->payloadSize = len;), there is no need in that end-of-string marker. And it won’t make any change if you use len+1 in g_memdup because payloadSize is used by the broker code to determine how many bytes to send.

I use len+1 cause I tried output the message and payload at nvmsgconv.cpp, and

if (ctx->payloadType == NVDS_PAYLOAD_DEEPSTREAM) {
    message = generate_event_message (ctx->privData, events->metadata);
    if (message) {
      len = strlen (message);
      // Remove '\0' character at the end of string and just copy the content.
      payload->payload = g_memdup (message, len+1);
      payload->payloadSize = len;
      //cout<<"message is: "<<message<<endl;
      //cout<<"at nvdsmsgconv, payload is :"<<(char*)payload->payload<<endl;
      g_free (message);
    }

the messge is correct while the payload has some random characters at the tail. the only operation is gmemdup .That’s why i guess it is the issue with missing end character ‘\0’.

as @ amycao runs well with the codes, maybe there is something wrong with my codes.

i may upload my codes later, but as i know, the writer of the mqtt plugin also met the same problem. maybe that’s because of we use docker? I don’t know why, i’ll dig in if i have time.

Do you know what it does? It treats payload->payload as a string, i.e a sequence of characters terminated by “\0”. And when you give it a payload that has that “\0” chopped off it carries on until it finds that 0 somewhere.
I bet that if you replace it with something like

cout<<"message is: "<<message<<endl;
cout<<"at nvdsmsgconv, payload is: "
for (int i=0; i<len; len++)
    cout << ((char*)payload->payload)[i];
cout << endl;

you’ll get two identical outputs.
So the problem is somewhere else.

that’s exactly what i asked. if you use g_memdup(len) to remove the ‘\0’ at the message, the payload will not know where to stop until it finds 0 in the memory somewhere. I didn’t cut off the ‘\0’, that’s my question:why to cut off the 0.

I use len+1 instead of len so when use gmemdup to copy message to payload, it will copy the ‘\0’ as well.

Payload knows nothing. The broker receives payload->payload and payload->payloadSize and uses that payloadSize as a number of bites it needs to send (you can think about it as the for operator from my example if it’s easier).
It does not treat payload->payload as string and therefore does not need any string termination.
Please re-read my previous messages carefully. Hope it explains everything.

i understand g_memdup() copy the whole memory of message to payload->payload, it should be ok as your explanation.

however, when i print the payload, it has the same random characters as i received in consumer client. if the payload in nvmsgconv is correct as you said, the g_memdup() will allocates a new block of memory to save the right message.

So if you are right, the nvmsgconv only copy the message to payload, the problem should be at the decode stage?

but what confused me is that if you are right, why use len+1 worked :-P, i’ll check it later anyway.

thx a lot

@dmitry.skorokhod
hi there, thanks for your hints. I finally found the problem, it is because the writer of the mqtt proto adaptor, he didn’t specify the len of payload in his codes. the paho mqtt provides 4 ways to generate a single mqtt message:

message_ptr 	mqtt::make_message (string_ref topic, const void *payload, size_t len)
 	Constructs a message with the specified array as a payload, and all other values set to defaults. More...
 
message_ptr 	mqtt::make_message (string_ref topic, const void *payload, size_t len, int qos, bool retained)
 	Constructs a message with the specified array as a payload, and all other values set to defaults. More...
 
message_ptr 	mqtt::make_message (string_ref topic, binary_ref payload)
 	Constructs a message with the specified buffer as a payload, and all other values set to defaults. More...

he used the third one without the len of payload, that leads to this problem.

now i can gen and receive the message correctly! :D

thanks you very much!

Glad you managed to find the problem.
Are you going to make your code public?
I have no use case for a mqtt adaptor but might have one in future as my Home Assistant uses it.

the mqtt adaptor is open-source from github.

as for my code, it is public on github too, but i’d like to say it still nees a lot of work to do. if you want to have a look, you can find it here. the repo is incomplete and i’m working on it.

hope they’re helpful. :-P

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.