Deepstream (deepstream-test5-app) terminates when MQTT connection timed out

[helge]

• Hardware Platform (GPU)
• DeepStream Version 6.3
• NVIDIA GPU Driver Version 535
• Issue Type: bugs
• How to reproduce the issue ?

I am testing the new MQTT message broker feature of DeepStream 6.3.

I modified the source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt sample configuration to use a live stream from a RTSP camera. I added a sink3 to be used to connect the deepstream-test5-app to a Mosquitto MQTT broker running locally on default port 1883.

The sink3 part of the config file looks like this:

‘’’
[sink3]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=UDPSink 5=nvoverlaysink 6=MsgConvBroker
type=6
msg-conv-config=msgconv_config_empty.txt
disable-msgconv=0 # set to 0 to enable msgconv, which convert the NVEVENT metadata from primary/secondary GIE into mqtt message
#(0): PAYLOAD_DEEPSTREAM - Deepstream schema payload
#(1): PAYLOAD_DEEPSTREAM_MINIMAL - Deepstream schema payload minimal
#(256): PAYLOAD_RESERVED - Reserved type
#(257): PAYLOAD_CUSTOM - Custom schema payload
msg-conv-payload-type=0
msg-broker-proto-lib=/opt/nvidia/deepstream/deepstream/lib/libnvds_mqtt_proto.so
#Provide your msg-broker-conn-str here
msg-broker-conn-str=127.0.0.1;1883
topic=test
#Optional:
config=cfg_mqtt.txt
‘’’

The cfg_mqtt.txt contains:
‘’’
[message-broker]
username = user
password = password
#share-connection = 1
#loop-timeout = 2000
‘’’

The example is working as expected (sending JSON messages to the MQTT broker) - but only as long as there are objects detected within a certain period of time. When it takes too long (e.g. more than 60 seconds) until a detectable object appears in the video, the app terminates in that moment, where a new object was detected.

The error message in the console is:
‘’’
Error sending repeat publish: The client is not currently connected.ERROR from sink_sub_bin_sink3: GStreamer encountered a general supporting library error.
Debug info: gstnvmsgbroker.cpp(537): legacy_gst_nvmsgbroker_render (): /GstPipeline:pipeline/GstBin:sink_sub_bin3/GstNvMsgBroker:sink_sub_bin_sink3:
failed to send the message. err(1)
‘’’

So it looks like there is no automatic reconnection done by the client, when the previous connection to the message broker has timed out.

This error is not handled, so the app terminates completely.

I also played around with share-connection (uncommented and so set to 1) and a loop-timeout = 60000 but the problem cannot be solved using these parameters.

[fanzh]

can you set timeout on server side?

[helge]

Mosquitto seems to have also a timeout parameter but this is only used in conjunction with MQTT bridges (multiple MQTT brokers).
Clients get disconnected by default, when there is no message within 60 seconds. Even if I could increase the timeout behavior on broker side, this would not be a solution as it may take hours before the first detectable object comes into the scenery.

The timeout handling must happen on client side. The keep-alive convention is, to send a kind of PING message (PINGREQ & PINGRESP) within 60 seconds so that the broker holds the connection.

The gstnvmsgbroker currently does not seem to handle this correctly for MQTT. Another option would be to reconnect, if publishing a message fails due to “not connected” error, but this would unwanted overhead.

[fanzh]

thanks for the update. currently keeplive is hardcode. we will improve it.