Frame rate drop, some on rtsp cameras

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

• Hardware Platform (GPU)
• DeepStream Version
deepstream-app version 6.3.0
DeepStreamSDK 6.3.0
CUDA Driver Version: 12.3
CUDA Runtime Version: 12.1
TensorRT Version: 8.5
cuDNN Version: 8.9
libNVWarp360 Version: 2.0.1d3

I have a pipeline that receives about 40 rtsp streams as input. After that, we send them to the detector, which returns detections, and we use these detections to generate crops. We send the received crops to the classifier. The problem is that over time, for some streams, the number of frames drops to 0. At the same time, if you completely stop the pipeline and start it again, all streams will start producing the required number of frames. I would like to know how I can restore the streams that have fallen without restarting the entire pipeline.
These are the settings streammux:

        self.streammux.set_property("width", cfg['streammux']['width'])
        self.streammux.set_property("height", cfg['streammux']['height'])
        self.streammux.set_property("batch-size", self.number_sources)
        self.streammux.set_property("batched-push-timeout", cfg['streammux']['batched-push-timeout'])
        self.streammux.set_property("attach-sys-ts", cfg['streammux']['attach-sys-ts'])
        self.streammux.set_property('live-source', 1)
        self.streammux.set_property("nvbuf-memory-type", 3)

These are the settings source:

            source_element = child_proxy.get_by_name("source")
            if source_element.find_property('drop-on-latency'):
                Object.set_property("drop-on-latency", False)
            if source_element.find_property('protocols'):
                Object.set_property("protocols", "tcp")
            if source_element.find_property('latency'):
                Object.set_property("latency", 1000)
            if source_element.find_property('cudadec-memtype'):
                Object.set_property('cudadec-memtype', 0)
            if source_element.find_property('udp-buffer-size'):
                Object.set_property('udp-buffer-size', 12000000)
            if source_element.find_property('select-rtp-protocol'):
                Object.set_property('select-rtp-protocol', 0)
            if source_element.find_property('type'):
                Object.set_property('type', 4)
            if source_element.find_property('rtsp-reconnect-interval-sec'):
                Object.set_property('rtsp-reconnect-interval-sec', 5)
            if source_element.find_property('rtsp-reconnect-attempts'):
                Object.set_property('rtsp-reconnect-attempts', 1)

These are the settings sink:

        self.sink = Gst.ElementFactory.make("fakesink", "fakesink")
        self.sink.set_property("qos", 0)
        self.sink.set_property('sync', 0)
3
  1. Are you using nvurisrcbin ? Only nvurisrcbin support rtsp-reconnect-interval-sec and rtsp-reconnect-attempts properties. You can set rtsp-reconnect-attempts to -1, so that If the reconnection fails, it will keep trying

Use fakesink sync=0, the pipeline will run at the fastest speed, usually used for performance testing

3.Try to use TCP only, Object.set_property('select-rtp-protocol', 4)
4.Try the latest version DS-7.1

4

I tried all the suggested options, but the problem remained. I do not exclude that the problem may be in the streams, but then the question arises why, when restarting the entire pipeline, the broken streams start working again. And how do I restart the problematic threads without restarting the entire pipeline?

5

You can refer to the operation in /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-app/deepstream_app.c, capture GST_MESSAGE_ERROR in bus_callback and try to find the source of the disconnection, and then reset or perform other operations on the source of the disconnection.

This is code snippets.

NvDsSrcParentBin *bin = &appCtx->pipeline.multi_src_bin;
      GstElement *msg_src_elem = (GstElement *) GST_MESSAGE_SRC (message);
      gboolean bin_found = FALSE;
      /* Find the source bin which generated the error. */
      while (msg_src_elem && !bin_found) {
        for (i = 0; i < bin->num_bins && !bin_found; i++) {
          if (bin->sub_bins[i].src_elem == msg_src_elem ||
              bin->sub_bins[i].bin == msg_src_elem) {
            bin_found = TRUE;
            break;
          }
        }
        msg_src_elem = GST_ELEMENT_PARENT (msg_src_elem);
      }
6

You also can use deepstream-app to do similar tests. I think deepstream-app will not cause the pipeline to fail to work due to a disconnection stream.

7

I’ve already done something similar, but it doesn’t work.

def bus_call(self, bus, message, loop):
        global global_stream_paths
        if self.loop is None:
            self.loop = loop
        t = message.type
        try:
            rtsp_url = message.src.get_property('location') 
        except:
            pass
        if t == Gst.MessageType.EOS:
            rtsp_url = message.src.get_property('location')
            rtsp_url = message.src
            for src in global_stream_paths:
                if src == rtsp_url:
                    print(f"Restarting stream {rtsp_url}")
                    message.src.set_state(Gst.State.NULL)
                    message.src.set_state(Gst.State.PLAYING) 
        elif t==Gst.MessageType.WARNING:
            err, debug = message.parse_warning()
            sys.stderr.write("Warning: %s: %s\n" % (err, debug))
        elif t == Gst.MessageType.ERROR:
            rtsp_url = message.src.get_property('location')
            err, debug = message.parse_error()
            sys.stderr.write("Error: %s: %s\n" % (err, debug))
            for src in global_stream_paths:
                if src == rtsp_url:
                    print(f"Restarting stream {rtsp_url}")
                    message.src.set_state(Gst.State.NULL)
                    message.src.set_state(Gst.State.PLAYING) 
        return True
8

Have you tried deepstream-app? If this is not a problem with deepstream, then I may not be able to debug

9

Do you want to exclude a problem with threads from possible causes? What example would you run as a check?

10

I want to exclude problems in your python application. When a source is disconnected, deepstream-app will not cause the pipeline to get stuck.
If the problem is caused by the application, you need to debug it yourself. I suggest you test it on the latest DS-7.1.