Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU)
Jetson • DeepStream Version
6.3 • JetPack Version (valid for Jetson only)
5.1.3 • Issue Type( questions, new requirements, bugs)
Questions / bug • How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
Launching deepstream-app and configure a sink to use lib_mqtt and shutdown the mqtt broker it sends the message to
Hi,
I am using the MQTT adapter with nvmsgbroker and nvmsgconv to send result from an object detection model to an external MQTT broker. The issue I have is that sometimes, the connection to the MQTT broker is interrupted and the app fail instantly, stopping all the other processing the app is doing.
I get this error when a message is not sent :
[mosq_mqtt_log_callback] Client uniqueID sending PUBLISH (d0, q0, r0, m602, ‘pai’, … (7285 bytes))
Error sending repeat publish: The client is not currently connected.ERROR from sink_sub_bin_sink2: GStreamer encountered a general supporting library error.
Debug info: gstnvmsgbroker.cpp(537): legacy_gst_nvmsgbroker_render (): /GstPipeline:pipeline/GstBin:sink_sub_bin2/GstNvMsgBroker:sink_sub_bin_sink2:
failed to send the message. err(1)
Quitting
[NvMultiObjectTracker] De-initialized
Error disconnecti
Is it possible to prevent the app from failing ? And to configure auto-reconnection ? I have red the documentation of the nvmsgborker (Gst-nvmsgbroker — DeepStream documentation) and saw that auto reconnect is not supported on MQTT adaptor but is there any workaround ?
I have looked at this topic. I do not use client-id in my broker config file.
Following the topic, I just tested deepstream test5 and deepstream app with standard configuration on deepstream 7.1 (I’m using 6.3 in production) and got the same errors when doing “systemctl restart mosquitto” on the remote broker server (To simulate a connection problem).
Here are the logs :
Publish callback with reason code: Success.
Publish callback with reason code: Success. – I restart mosquitto here –
Error sending repeat publish: The client is not currently connected.ERROR from sink_sub_bin_sink2: GStreamer encountered a general supporting library error.
Debug info: gstnvmsgbroker.cpp(559): legacy_gst_nvmsgbroker_render (): /GstPipeline:pipeline/GstBin:sink_sub_bin2/GstNvMsgBroker:sink_sub_bin_sink2:
failed to send the message. err(1)
Quitting
nvstreammux: Successfully handled EOS for source_id=0
[mosq_mqtt_log_callback] Client null sending CONNECT
[mosq_mqtt_log_callback] Client null received CONNACK (0)
mqtt connection success; ready to send data
[mosq_mqtt_log_callback] Client null sending DISCONNECT
mqtt disconnected
[NvMultiObjectTracker] De-initialized
App run failed
Thanks for your answer.
My company is Phoenix AI (https://www.phoenix-ai.com/). We are Nvidia partners, specialized in edge AI using NVIDIA Jetson.
Do you have any plans to release the adapter source code at any time ?
Would you know any workaround allowing to perform reconnection ? It is a blocking issue on our side to deploy any solution with MQTT Integration.
Thank you very much for the patch.
Just tested to replace every cpp. In native gst-nvmsgbroker.cpp do not compile and IO got this error :
gstnvmsgbroker.cpp:534:40: error: ‘NVDS_CUSTOM_MSG_SPARSE4D’ was not declared in this scope; did you mean ‘NVDS_CUSTOM_MSG_BLOB’?
534 | if(batch_meta->misc_batch_info[0]==NVDS_CUSTOM_MSG_SPARSE4D) {
Removed temporarly the if condition (go through the else automatically) to make it compile.
It does not seems to reconnect to my MQTT broker with the new code. I am still analysing the situation.
Do you know if I need to use the nvmsgnbroker “new-api” or not ?
When I do, I don’t receive the message on my MQTT broker but it seems that the reconnection is working. With old API, I receive the message but the behaviour is the same as the old code.
you can compare the patch code with DS7.1. you only need to copy the modification in gst_nvmsgbroker_do_work function. you can add log to check if the disconnect and connect function are called after disconnection.
“new-api=0” means using the default lib. Please find the explanation in the plugin doc. please set new-api to 0.
Thank you, I am analyzing the changes.
Currently what I get when changing the code in gst_nvmsgbroker_do_work is this behaviour :
Error sending repeat publish: The client is not currently connected.ERROR from sink_sub_bin_sink2: GStreamer encountered a general supporting library error.
Debug info: gstnvmsgbroker.cpp(584): legacy_gst_nvmsgbroker_render (): /GstPipeline:pipeline/GstBin:sink_sub_bin2/GstNvMsgBroker:sink_sub_bin_sink2:
failed to send the message. err(1)
Quitting
[mosq_mqtt_log_callback] Client null sending CONNECT
[mosq_mqtt_log_callback] Client null received CONNACK (0)
mqtt connection success; ready to send data
(here, wait for the loop timeout to expire)
[mosq_mqtt_log_callback] Client null sending DISCONNECT
mqtt disconnected
[NvMultiObjectTracker] De-initialized
The MQTT reconnect succesfully which is nice, wait for the loop-timeout to expire and then automatically disconnect. Even if the broker is available again in the meantime ( I tested with very long loop timeout)
Yes there is still an issue. even if the the adaptor seems to succesfully reconnect, it does not send any new messages to the broker and disconnect everytime after the loop-timeout is expired. I cannot get it to send new messages. The pipeline just stops.
Thanks for the sharing! Since the plugin, low-level lib are adapter are all opensource, you can add log to check why send new messages failed after reconnecting. you can also use mosquitto api to test reconnection first, then port to adapter.
I finally found a solution. I rewrote most of the mqtt_adaptor.cpp so it can handle reconnect without stopping the main pipeline as well as authorize the pipeline to start without connection to the broker.
The main issue in old code was the mutual call to mosquitto_loop_start and recurrent call to mosquitto_loop. You’re solution was correcting most of it but I still got issues. I now launch mosquitto_loop_start in a thread completly independant from the deepstream pipeline and let it handle the broker connection.