DeepStream nvmsgbroker MQTT disconnect crashes entire pipeline after long runtime

[votrongtin882003]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 7.1
• JetPack Version (valid for Jetson only)
• TensorRT Version (comes with DeepStream 7.1 container)
• NVIDIA GPU Driver Version (valid for GPU only) 570.172.08
• Issue Type( questions, new requirements, bugs) questions
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Hi DeepStream team,

I’m using nvmsgbroker with MQTT in a DeepStream pipeline.

Everything runs fine at first, but after about 15–30 minutes, the MQTT connection drops. When that happens, nvmsgbroker fails to publish and the whole DeepStream pipeline shuts down.

Error sending repeat publish: The client is not currently connected.Error: gst-library-error-quark: GStreamer encountered a general supporting library error. (1): gstnvmsgbroker.cpp(559): legacy_gst_nvmsgbroker_render (): /GstPipeline:pipeline0/GstNvMsgBroker:nvmsg-broker:
failed to send the message. err(1)
 
[SHUTDOWN] Stopping pipeline...
Error disconnecting: (The client is not currently connected.).
Stopping the server..!!
Stopped the server..!!
[SHUTDOWN] Pipeline stopped

I’ve seen that MQTT reconnection isn’t supported yet.

I have searched and read many forum posts related to this same problem. Most suggested workarounds are not reliable for long-running deployment.

The most recent related post I found is from Alex:

Deepstream app fails when MQTT message could not reach broker

Yes indeed, found the source code this morning. Thank you very much for the patch. Just tested to replace every cpp. In native gst-nvmsgbroker.cpp do not compile and IO got this error : gstnvmsgbroker.cpp:534:40: error: ‘NVDS_CUSTOM_MSG_SPARSE4D’ was not declared in this scope; did you mean ‘NVDS_CUSTOM_MSG_BLOB’? 534 | if(batch_meta->misc_batch_info[0]==NVDS_CUSTOM_MSG_SPARSE4D) { Removed temporarly the if condition (go through the else automatically) to make it compile. It does not s…

I understand he implemented a solution, but the patch is not shared publicly, so I’m still struggling with the same issue.

I also tried enabling detailed MQTT logs following this post:

MQTT Broker crashes

Thanks for the sharing! keep-alive is configurable, it means “number of seconds after which the broker should send a PING message to the client if no other messages have been exchanged in that time.”, I will try to reproduce this issue. about low-level log, please enter /opt/nvidia/deepstream/deepstream/sources/tools/nvds_logger directory, then execute ./setup_nvds_logger.sh, then start the application. the log will be written to /tmp/nvds/ds.log. please refer to the doc. could you use D…

However, after enabling that logging, my pipeline started hitting a segmentation fault instantly after start, I’m struggling how to turn off that log.
So for now, I would also like to ask: how can I temporarily disable that MQTT debug logging to avoid the segfault?

Coming back to the main issue — I really need:

• MQTT automatic reconnection

• No DeepStream pipeline shutdown when MQTT disconnects

Could you please share a patch that implements this behavior? I’m using DS 7.1

I’m not experienced in modifying internal DeepStream / GStreamer plugin code, so having a working patch would help a lot.

Thanks for your time and support.

UPDATE: latest log before crash

[fanzh]

setup_nvds_logger.sh is opensource. you can rm /var/log/nvds/ds.log to disable log. DS8.0 already supports mqtt auto-reconnection. you can use DS8.0 instead. DS 8.0 deepstream:8.0-triton-multiarch is recommendated.

[votrongtin882003]

Thanks for the suggestion.

Unfortunately, I cannot move to DeepStream 8.0 due to hardware constraints (Jetson Orin Nano Super), so I must stay on DeepStream 7.1.

Since DS 8.0 already includes MQTT auto-reconnection, may I kindly ask if you could share the relevant patch that implements MQTT reconnection in nvmsgbroker? Even a minimal diff against DS 7.1 would help me backport the fix.

Thank you very much for your support.

[fanzh]

For simplification, you can copy /opt/nvidia/deepstream/deepstream/sources/libs/mqtt_protocol_adaptor and /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvmsgbroker to DS7.1, then rebuild and replace the related library according to the readme. nvmsgbroker plugin will call low-level library mqtt_protocol_adaptor. the specified mqtt protocol impletmentation is in mqtt_protocol_adaptor.

[votrongtin882003]

That’s great news. Thank you very much for sharing this solution.

I’m really happy to know there is still a workable path for my DS 7.1 setup. I will try copying
/opt/nvidia/deepstream/deepstream/sources/libs/mqtt_protocol_adaptor and
/opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvmsgbroker
from DS 8.0 into my DS 7.1 environment, rebuild the libraries following the README, and replace the related binaries.

I’ll test this approach and report back with results soon.

Thanks again for your support and guidance!

[fanzh]

Sorry for the late reply, Is this still an DeepStream issue to support? Thanks!

[votrongtin882003]

Hi @fanzh,

Thanks for following up.

The project is currently suspended for a short period, so I haven’t been able to test the DS8 → DS7.1 workaround yet. The DeepStream issue is still relevant, but I cannot verify the solution right now.

In this case, would you prefer that:

1. I leave this thread open and update it once the project resumes, or
2. You close this thread for now, and I will open a new post later referencing this one when I have test results?

Please let me know which option you prefer.
Thanks again for your support.

Best regards,
Tin

[fanzh]

@votrongtin882003

The project is currently suspended for a short period,

OK, Please leave this thread open and update it once the project resumes.