MQTT Broker crashes

oleg.s · April 17, 2024, 2:50pm

Please provide complete information as applicable to your setup.

• Hardware Platform (GPU)
• DeepStream Version 6.4
• NVIDIA GPU Driver Version (valid for GPU only) NVIDIA GeForce GTX 1650 / Driver Version: 525.147.05 / CUDA Version: 12.0
• Issue Type( questions, new requirements, bugs) question
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Hello,

on short video my mqtt broker works, however, on live stream and after a couple of minutes it crashed with:

wrs-deepstream-app  | Error: gst-library-error-quark: GStreamer encountered a general supporting library error. (1): gstnvmsgbroker.cpp(553): legacy_gst_nvmsgbroker_render (): /GstPipeline:pipeline0/GstNvMsgBroker:nvmsg-broker:
wrs-deepstream-app  | failed to send the message. err(1)

my configs:
cfg_msgconv.txt (1010 Bytes)
cfg_mqtt.txt (1.1 KB)

I modified the output for msgbroker and using following commands to build the image:

COPY /deepstream/bindings/modified/v0/bindschema.cpp \
     /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/src/bindschema.cpp
COPY /deepstream/bindings/modified/v0/nvdsmeta_schema.h \
     /opt/nvidia/deepstream/deepstream/sources/includes/nvdsmeta_schema.h
COPY /deepstream/bindings/modified/v0/eventmsg_payload.cpp \
     /opt/nvidia/deepstream/deepstream/sources/libs/nvmsgconv/deepstream_schema/eventmsg_payload.cpp    
COPY /deepstream/bindings/modified/v0/Makefile \
     /opt/nvidia/deepstream/deepstream/sources/libs/nvmsgconv/Makefile

RUN cd /opt/nvidia/deepstream/deepstream/sources/deepstream_python_apps/bindings/ && \
    git submodule update --init && \
    mkdir build && cd build && \
    cmake .. && \
    make -j$(nproc) && \
    pip3 install ./pyds-1.1.10-py3-none*.whl --force-reinstall

RUN cd /opt/nvidia/deepstream/deepstream-6.4/sources/libs/nvmsgconv/ && \
    make && \
    mv -f /opt/nvidia/deepstream/deepstream-6.4/sources/libs/nvmsgconv/libnvds_msgconv.so /opt/nvidia/deepstream/deepstream/lib/libnvds_msgconv.so

And it does work and the message do look as expected. But then it suddenly crashes…

Can you help please?

fanzh · April 18, 2024, 5:15am

could you share the whole log? could you try keep-alive = 30 in cfg_mqtt.txt?

oleg.s · April 24, 2024, 2:01pm

I have set keep-alive = 30 and executed with GST_DEBUG=4… Unfortunately no messages are sent unter GST_DEBUG=5…

This was the error:

Error sending repeat publish: The client is not currently connected.0:03:37.993447228   110 0x5a68985845e0 WARN             nvmsgbroker gstnvmsgbroker.cpp:553:legacy_gst_nvmsgbroker_render:<nvmsg-broker> error: failed to send the message. err(1)
0:03:37.993497437   110 0x5a68985845e0 INFO        GST_ERROR_SYSTEM gstelement.c:2271:gst_element_message_full_with_details:<nvmsg-broker> posting message: GStreamer encountered a general supporting library error.
0:03:37.993550300   110 0x5a68985845e0 INFO        GST_ERROR_SYSTEM gstelement.c:2298:gst_element_message_full_with_details:<nvmsg-broker> posted error message: GStreamer encountered a general supporting library error.
0:03:37.993588010   110 0x5a68985845e0 INFO                    task gsttask.c:368:gst_task_func:<queue0:src> Task going to paused
Error: gst-library-error-quark: GStreamer encountered a general supporting library error. (1): gstnvmsgbroker.cpp(553): legacy_gst_nvmsgbroker_render (): /GstPipeline:pipeline0/GstNvMsgBroker:nvmsg-broker:
failed to send the message. err(1)
INFO:__main__:Exiting app

Please find attached the graph for my pipeline:
graph00.pdf (28.8 KB)

Configs:

[message-broker]
username = ***
password = ***
client-id = publish-deepstream-0
#enable-tls = 1
#tls-cafile =
#tls-capath =
#tls-certfile =
#tls-keyfile =
#share-connection = 1
#loop-timeout = 2000
keep-alive = 30
#log_topic: null

I need to have msgconv txt but all the parameters are not used…

[analytics0]
enable=0
id=test-id
description="Vehicle Detection and License Plate Recognition"
# source=OpenALR
version=1.0


[place0]
enable=0

[sensor0]
enable=0

Difficult to measure when it is happening but it looks like when there have been no messages for some time…

[mosq_mqtt_log_callback] Client publish-deepstream-0 sending PUBLISH (d0, q0, r0, m10047, 'wrs/gates/detection', ... (476 bytes))
Publish callback with reason code: Success.
INFO:__main__:Frame Number: 15300
INFO:__main__:Frame Number: 15300
INFO:deepstream.utils.fps:
Perf: {'stream0': 25.0, 'stream1': 25.0}

INFO:__main__:Frame Number: 15400
INFO:__main__:Frame Number: 15400
INFO:deepstream.utils.fps:
Perf: {'stream0': 25.0, 'stream1': 25.0}

INFO:__main__:Frame Number: 15500
INFO:__main__:Frame Number: 15500
INFO:__main__:Frame Number: 15600
INFO:__main__:Frame Number: 15600
INFO:deepstream.utils.fps:
Perf: {'stream0': 25.0, 'stream1': 25.0}

INFO:__main__:Frame Number: 15700
INFO:__main__:Frame Number: 15700
INFO:deepstream.utils.fps:
Perf: {'stream0': 25.0, 'stream1': 25.0}

INFO:__main__:Frame Number: 15800
INFO:__main__:Frame Number: 15800
INFO:deepstream.utils.fps:
Perf: {'stream0': 25.0, 'stream1': 25.0}

INFO:__main__:Frame Number: 15900
INFO:__main__:Frame Number: 15900
INFO:deepstream.utils.fps:
Perf: {'stream0': 24.8, 'stream1': 24.8}

INFO:__main__:Frame Number: 16000
INFO:__main__:Frame Number: 16000
INFO:__main__:Frame Number: 16100
INFO:__main__:Frame Number: 16100
INFO:deepstream.utils.fps:
Perf: {'stream0': 25.59, 'stream1': 25.59}

INFO:__main__:Frame Number: 16200
INFO:__main__:Frame Number: 16200
INFO:deepstream.utils.fps:
Perf: {'stream0': 25.0, 'stream1': 25.0}

INFO:__main__:Frame Number: 16300
INFO:__main__:Frame Number: 16300
INFO:deepstream.utils.fps:
Perf: {'stream0': 24.79, 'stream1': 24.79}

INFO:__main__:Frame Number: 16400
INFO:__main__:Frame Number: 16400
INFO:deepstream.utils.fps:
Perf: {'stream0': 24.79, 'stream1': 24.79}

INFO:__main__:Frame Number: 16500
INFO:__main__:Frame Number: 16500
INFO:__main__:Frame Number: 16600
INFO:__main__:Frame Number: 16600
INFO:deepstream.utils.fps:
Perf: {'stream0': 24.99, 'stream1': 24.99}

[mosq_mqtt_log_callback] Client publish-deepstream-0 sending PUBLISH (d0, q0, r0, m10048, 'wrs/gates/detection', ... (479 bytes))
Error sending repeat publish: The client is not currently connected.Error: gst-library-error-quark: GStreamer encountered a general supporting library error. (1): gstnvmsgbroker.cpp(553): legacy_gst_nvmsgbroker_render (): /GstPipeline:pipeline0/GstNvMsgBroker:nvmsg-broker:
failed to send the message. err(1)
INFO:__main__:Exiting app

also happened after 4800 frames (25) from live stream… first message was sent successfully but the second did not due to missing connection… how can this happen?

there were no messages before frame 4700

INFO:__main__:Frame Number: 4700
INFO:__main__:Frame Number: 4700
INFO:deepstream.utils.fps:
Perf: {'stream0': 25.0, 'stream1': 25.0}

INFO:__main__:Frame Number: 4800
INFO:__main__:Frame Number: 4800
INFO:deepstream.utils.fps:
Perf: {'stream0': 25.0, 'stream1': 25.0}

INFO:__main__:Frame Number: 4900
INFO:__main__:Frame Number: 4900
[mosq_mqtt_log_callback] Client publish-deepstream-0 sending PUBLISH (d0, q0, r0, m1, 'wrs/gates/detection', ... (473 bytes))
[mosq_mqtt_log_callback] Client publish-deepstream-0 received CONNACK (0)
mqtt connection success; ready to send data
Publish callback with reason code: Success.
[mosq_mqtt_log_callback] Client publish-deepstream-0 sending PUBLISH (d0, q0, r0, m2, 'wrs/gates/detection', ... (473 bytes))
Error sending repeat publish: The client is not currently connected.Error: gst-library-error-quark: GStreamer encountered a general supporting library error. (1): gstnvmsgbroker.cpp(553): legacy_gst_nvmsgbroker_render (): /GstPipeline:pipeline0/GstNvMsgBroker:nvmsg-broker:
failed to send the message. err(1)
INFO:__main__:Exiting app

fanzh · April 26, 2024, 9:22am

could you use the network issue to narrow down this issue? please check if the server can’t receive the messages or DeepStream did not send the messages.
please refer to this topic , could you share more low-level logs? Thanks!

oleg.s · April 26, 2024, 9:51am

Hi,

just some clarification… I am using MQTT Broker by starting in the container:

  mosquitto:
    container_name: wrs-mosquitto
    image: eclipse-mosquitto:2.0
    ports: 
      - "1883:1883"
      - "9001:9001"
    volumes:
      - ./mosquitto/:/mosquitto/

And the Broker is accessible and Deepstream can send messages to it. I tested it a lot of times on the sample videos.

However, when I switch to live stream and sometimes there are no objects to detect for a minute or so. And then, if after that period of time a new message needs to be sent to the broker, it cannot be sent as described above. The service can run for 30 minutes and send messages to MQTT Broker if there are continuously objects to detect. However, after a period of time with no objects to detect, the connection bŕeaks down.

When I restart just the Deepstream application again (MQTT Broker NO restart) it works perfectly fine. So the MQTT Broker is continuously accessable.

Both Images are executed on my local machine.

My assumption is, that the keep-alive is not working and if there have been not data exchange - the connection breaks down.

I would be happy to share more logs, but I did not really understand how can activate it for MQTT - could you please provide more detailed instruction how to do so?

many thanks in advance!

fanzh · April 26, 2024, 10:06am

Thanks for the sharing! keep-alive is configurable, it means “number of seconds after which the broker should send a PING message to the client if no other messages have been exchanged in that time.”, I will try to reproduce this issue.
about low-level log, please enter /opt/nvidia/deepstream/deepstream/sources/tools/nvds_logger directory, then execute ./setup_nvds_logger.sh, then start the application. the log will be written to /tmp/nvds/ds.log. please refer to the doc.
could you use DeepStream test4 or test5 to reproduce this issue?

fanzh · April 29, 2024, 8:54am

I can reproduce this issue. here is a workaround.

modify the following code in opt\nvidia\deepstream\deepstream\sources\gst-plugins\gst-nvmsgbroker\gstnvmsgbroker.cpp

    self->sleepTime = 20;   //modify from 0 to 20
    /*while (self->isRunning && self->pendingCbCount <= 0) {
      g_cond_wait (&self->flowCond, &self->flowLock);
    }*/   //comment out this code.

rebuild libnvdsgst_msgbroker.so according to /opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvmsgbroker/README.
use the following command-line to replace libnvdsgst_msgbroker.so.

mv /opt/nvidia/deepstream/deepstream/lib/gst-plugins/libnvdsgst_msgbroker.so /opt/nvidia/deepstream/deepstream/lib/gst-plugins/libnvdsgst_msgbroker.so_bk
mv libnvdsgst_msgbroker.so /opt/nvidia/deepstream/deepstream/lib/gst-plugins/

oleg.s · April 29, 2024, 4:26pm

Hey,

many thanks, it seems to work now! :)

So I modified this part of code, correct?:

static gpointer
gst_nvmsgbroker_do_work (gpointer data)
{
  GstNvMsgBroker *self = (GstNvMsgBroker *) data;
  self->sleepTime=20;
   while (self->isRunning) {
    g_mutex_lock (&self->flowLock);
    /* while (self->isRunning && self->pendingCbCount <= 0) {
      g_cond_wait (&self->flowCond, &self->flowLock);
    } */
    g_mutex_unlock (&self->flowLock);

    if (!self->isRunning) {
      return NULL;
    }

    self->nvds_msgapi_do_work (self->connHandle);
    g_usleep (self->sleepTime * 1000);
  }
  return self;
}

I added the the line “self->sleepTime=20;” as it was not there before.

Just to make sure during the build process…

I run first “make” command and after that “make install”?

make && \
make install && \
mv /opt/nvidia/deepstream/deepstream/lib/gst-plugins/libnvdsgst_msgbroker.so /opt/nvidia/deepstream/deepstream/lib/gst-plugins/libnvdsgst_msgbroker.so_bk && \
mv libnvdsgst_msgbroker.so /opt/nvidia/deepstream/deepstream/lib/gst-plugins/

I will be adding this to make docker build process…

Most importantly, as we will be using this in production, may I ask what have caused the issue?

fanzh · April 30, 2024, 2:17am

nvmsgbroker plugin is openosurce. sleepTime is a property of this plugin. it means “Sleep time between consecutive do_work calls in millisecond”. the default value is 0 and this value is configurable. you can reset it by g_object_set. or you can modify the code directly. “self->sleepTime = 0;” is defined in gst_nvmsgbroker_init, you can modify it to 20.

MAKFILE is opensource. I suggest using the command-line instead of “make install”. “make install” ony copy the so. it will not backup the original so.

if nvds_msgapi_do_work is not called in a period of time, the server will disconnect because it did not receive any message. nvsmsgbroker MQTT low-level is not opensource. the fix is just a workaround. if not use MQTT, please use the original so.

oleg.s · April 30, 2024, 1:14pm

Ok, many thanks!

So do we need the keep-alive parameter in the mqtt config?

fanzh · April 30, 2024, 3:26pm

it has the default value(60) if not set. and it is configurable.

oleg.s · April 30, 2024, 3:40pm

Ok, but which role does it play now after the modification?

fanzh · May 1, 2024, 12:38am

please refer to my comment on April 26.

system · May 20, 2024, 1:54am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Custom payload for MQTT Broker DeepStream SDK	24	813	March 1, 2024
Break in MQTT connection silently crashes Deepstream pipeline DeepStream SDK deepstream	7	119	February 17, 2025
Deepstream 6.3 MQTT clientID DeepStream SDK gstreamer , nvbugs	5	363	May 24, 2024
Gst-nvmsgbroker break the pipeline DeepStream SDK	9	693	April 6, 2023
GStreamer-CRITICAL **: 16:10:55.658: gst_meta_api_type_has_tag: assertion 'tag != 0' failed DeepStream SDK ubuntu , gstreamer , python , deepstream	18	73	February 21, 2025
A problem about the plugin of Gst-nvmsgbroker Jetson TX2	4	778	October 31, 2019
Dynamically deleting a stream causes a deadlock DeepStream SDK	16	2075	July 11, 2022
Deepstream 6.1: deepstream-app not working after install DeepStream SDK deepstream61	9	2041	September 5, 2022
Deepstream Detections to MQTT working sample DeepStream SDK gstreamer , demos-and-tutorials , deepstream61	5	2379	September 16, 2022
Customize Gst-nvmsgbroker adaptor for MQTT using paho.mqtt.c DeepStream SDK	4	1570	October 12, 2021

MQTT Broker crashes

Related topics