• Hardware Platform (Jetson / GPU): Jetson
• DeepStream Version: 7.1
• JetPack Version (valid for Jetson only) 6.1
• TensorRT Version 10.3.0
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs) Bug
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
Here are some details about my setup:
-
The Deepstream pipeline is running on a NVIDIA Jetson Orin NX (16GB) device inside the
nvcr.io/nvidia/deepstream-l4t:7.1-triton-multiarch
Docker container. -
I’ve configured the
nvmsgbroker
to send messages out of the Deepstream container using the Python plugins
msgbroker = Gst.ElementFactory.make("nvmsgbroker", "nvmsg-broker")
msgbroker.set_property('sync', 0)
msgbroker.set_property('new-api', 1)
- The Mosquitto service is managed outside this Docker environment
• Bug Description
These are what my logs look like:
- Normal Operation:
lvt-deepstream | 2025-02-05T15:53:44.841202+0000 : INFO : Stream #=stream1, Frame #=473, ts=1738770824478001000, pts=48.611070048, Person #=0, Vehicle #=1
lvt-deepstream | [mosq_mqtt_log_callback] Client 123 sending CONNECT
lvt-deepstream | [mosq_mqtt_log_callback] Client 123 received CONNACK (0)
lvt-deepstream | mqtt connection success; ready to send data
lvt-deepstream | 2025-02-05T15:53:44.844301+0000 : INFO : Stream #=stream2, Frame #=472, ts=1738770824478311000, pts=48.608900472, Person #=0, Vehicle #=5
lvt-deepstream | 2025-02-05T15:53:44.845670+0000 : INFO : Stream #=stream0, Frame #=446, ts=1738770824478774000, pts=48.69980177, Person #=0, Vehicle #=0
lvt-deepstream | Error disconnecting: (The client is not currently connected.).
lvt-deepstream | [mosq_mqtt_log_callback] Client 123 sending CONNECT
lvt-deepstream | [mosq_mqtt_log_callback] Client 123 received CONNACK (0)
lvt-deepstream | mqtt connection success; ready to send data
lvt-deepstream | Encode Latency = 43.239990
lvt-deepstream | [mosq_mqtt_log_callback] Client 123 sending PUBLISH (d0, q0, r0, m1, '/tracks', ... (3101 bytes))
lvt-deepstream | Publish callback with reason code: Success.
lvt-deepstream | 2025-02-05T15:53:44.946311+0000 : INFO : Stream #=stream2, Frame #=473, ts=1738770824589330000, pts=48.708684788, Person #=0, Vehicle #=5
lvt-deepstream | Encode Latency = 41.376953
lvt-deepstream | [mosq_mqtt_log_callback] Client 123 sending PUBLISH (d0, q0, r0, m2, '/tracks', ... (3101 bytes))
lvt-deepstream | Publish callback with reason code: Success.
lvt-deepstream | 2025-02-05T15:53:45.048182+0000 : INFO : Stream #=stream1, Frame #=475, ts=1738770824666035000, pts=48.810935175, Person #=0, Vehicle #=0
lvt-deepstream | 2025-02-05T15:53:45.051957+0000 : INFO : Stream #=stream2, Frame #=474, ts=1738770824666196000, pts=48.808284209, Person #=0, Vehicle #=5
lvt-deepstream | 2025-02-05T15:53:45.053890+0000 : INFO : Stream #=stream0, Frame #=448, ts=1738770824666447000, pts=48.899472051, Person #=0, Vehicle #=0
lvt-deepstream | Encode Latency = 43.037842
lvt-deepstream | [mosq_mqtt_log_callback] Client 123 sending PUBLISH (d0, q0, r0, m3, '/tracks', ... (2659 bytes))
lvt-deepstream | Publish callback with reason code: Success.
lvt-deepstream |
lvt-deepstream | **PERF: {'stream0': 8.6, 'stream1': 9.39, 'stream2': 9.59}
- On Mosquitto service restart
lvt-deepstream | 2025-02-05T15:53:46.869535+0000 : INFO : Stream #=stream1, Frame #=494, ts=1738770826556691000, pts=50.710565027, Person #=0, Vehicle #=0
lvt-deepstream | 2025-02-05T15:53:46.874242+0000 : INFO : Stream #=stream2, Frame #=493, ts=1738770826556833000, pts=50.772934986, Person #=0, Vehicle #=5
lvt-deepstream | 2025-02-05T15:53:46.875840+0000 : INFO : Stream #=stream0, Frame #=467, ts=1738770826556947000, pts=50.798956361, Person #=0, Vehicle #=0
lvt-deepstream | [mosq_mqtt_log_callback] Client 123 sending CONNECT
lvt-deepstream | [mosq_mqtt_log_callback] Client 123 received CONNACK (0)
lvt-deepstream | mqtt connection success; ready to send data
lvt-deepstream | Error disconnecting: (The client is not currently connected.).
lvt-deepstream | [mosq_mqtt_log_callback] Client 123 sending CONNECT
lvt-deepstream | [mosq_mqtt_log_callback] Client 123 received CONNACK (0)
lvt-deepstream | mqtt connection success; ready to send data
lvt-deepstream | Encode Latency = 48.074951
lvt-deepstream | [mosq_mqtt_log_callback] Client 123 sending PUBLISH (d0, q0, r0, m1, '/tracks', ... (2660 bytes))
lvt-deepstream | Publish callback with reason code: Success.
lvt-deepstream | 2025-02-05T15:53:46.981949+0000 : INFO : Stream #=stream1, Frame #=495, ts=1738770826611877000, pts=50.810994635, Person #=0, Vehicle #=0
lvt-deepstream | 2025-02-05T15:53:46.986502+0000 : INFO : Stream #=stream2, Frame #=494, ts=1738770826612008000, pts=50.854150408, Person #=0, Vehicle #=5
lvt-deepstream | Encode Latency = 38.187988
lvt-deepstream | [mosq_mqtt_log_callback] Client 123 sending PUBLISH (d0, q0, r0, m2, '/tracks', ... (2658 bytes))
lvt-deepstream | Publish callback with reason code: Success.
- When Deepstream finally silently crashes
lvt-deepstream | 2025-02-05T15:54:17.279242+0000 : INFO : Stream #=stream1, Frame #=797, ts=1738770856865785000, pts=81.011141431, Person #=0, Vehicle #=0
lvt-deepstream | 2025-02-05T15:54:17.283939+0000 : INFO : Stream #=stream2, Frame #=796, ts=1738770856865882000, pts=81.007415151, Person #=0, Vehicle #=5
lvt-deepstream | 2025-02-05T15:54:17.285902+0000 : INFO : Stream #=stream0, Frame #=770, ts=1738770856865956000, pts=81.098151222, Person #=0, Vehicle #=0
lvt-deepstream | [mosq_mqtt_log_callback] Client 123 sending CONNECT
lvt-deepstream | [mosq_mqtt_log_callback] Client 123 received CONNACK (0)
lvt-deepstream | mqtt connection success; ready to send data
lvt-deepstream | Encode Latency = 55.544922
lvt-deepstream | Error sending repeat publish: The client is not currently connected.2025-02-05T15:54:17.366136+0000 : INFO : Stream #=stream1, Frame #=798, ts=1738770856955782000, pts=81.111038711, Person #=0, Vehicle #=0
lvt-deepstream | 2025-02-05T15:54:17.369212+0000 : INFO : Stream #=stream2, Frame #=797, ts=1738770856955914000, pts=81.107422948, Person #=0, Vehicle #=5
lvt-deepstream | 2025-02-05T15:54:17.370572+0000 : INFO : Stream #=stream0, Frame #=771, ts=1738770856983033000, pts=81.198376394, Person #=0, Vehicle #=0
lvt-deepstream | Encode Latency = 30.634033
lvt-deepstream | 2025-02-05T15:54:17.460197+0000 : INFO : Stream #=stream1, Frame #=799, ts=1738770857065805000, pts=81.210818836, Person #=0, Vehicle #=0
lvt-deepstream | 2025-02-05T15:54:17.464705+0000 : INFO : Stream #=stream2, Frame #=798, ts=1738770857065930000, pts=81.207740903, Person #=0, Vehicle #=5
lvt-deepstream | 2025-02-05T15:54:17.466634+0000 : INFO : Stream #=stream0, Frame #=772, ts=1738770857066037000, pts=81.299408801, Person #=0, Vehicle #=0
lvt-deepstream | 2025-02-05T15:54:17.503547+0000 : INFO : Stream #=stream1, Frame #=800, ts=1738770857172569000, pts=81.310401128, Person #=0, Vehicle #=0
lvt-deepstream | 2025-02-05T15:54:17.506592+0000 : INFO : Stream #=stream2, Frame #=799, ts=1738770857172704000, pts=81.307897252, Person #=0, Vehicle #=4
lvt-deepstream | Encode Latency = 7.348145
lvt-deepstream | Encode Latency = 20.053955
lvt-deepstream | 2025-02-05T15:54:17.586879+0000 : INFO : Stream #=stream1, Frame #=802, ts=1738770857356802000, pts=81.510347212, Person #=0, Vehicle #=0
lvt-deepstream | 2025-02-05T15:54:17.589685+0000 : INFO : Stream #=stream2, Frame #=801, ts=1738770857357033000, pts=81.50757504, Person #=0, Vehicle #=4
lvt-deepstream | 2025-02-05T15:54:17.591177+0000 : INFO : Stream #=stream0, Frame #=775, ts=1738770857357384000, pts=81.598646284, Person #=0, Vehicle #=0
lvt-deepstream | Encode Latency = 36.311035
lvt-deepstream | [mosq_mqtt_log_callback] Client 123 sending CONNECT
lvt-deepstream | [mosq_mqtt_log_callback] Client 123 received CONNACK (0)
lvt-deepstream | mqtt connection success; ready to send data
lvt-deepstream | [mosq_mqtt_log_callback] Client 123 sending CONNECT
lvt-deepstream | [mosq_mqtt_log_callback] Client 123 received CONNACK (0)
lvt-deepstream | mqtt connection success; ready to send data
lvt-deepstream | [mosq_mqtt_log_callback] Client 123 sending CONNECT
lvt-deepstream | [mosq_mqtt_log_callback] Client 123 received CONNACK (0)
lvt-deepstream | mqtt connection success; ready to send data
As you can see from the log messages (and timestamps) above the Deepstream pipeline crashes about 30 seconds after the Mosquitto service restarts/stops.
I need your support to make the nvmsgbroker
element robust to these breaks in connection.
I further examined the code in /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvmsgbroker
and found that there is no mechanism for reconnecting back to the Mosquitto service.
Here I’m specifically referring to the empty nvds_msgapi_connect_callback
method in /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvmsgbroker/gstnvmsgbroker.cpp
static void
nvds_msgapi_connect_callback (NvDsMsgApiHandle h_ptr,
NvDsMsgApiEventType ds_evt)
{
}