GLib.timeout_add_seconds and encode plugin stops working when some errors occur

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) rtx4060
• DeepStream Version 6.3
• JetPack Version (valid for Jetson only)
• TensorRT Version deepstream 6.3 official docker image
• NVIDIA GPU Driver Version (valid for GPU only) 535.86.05
• Issue Type( questions, new requirements, bugs) bugs
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
It happens after some camera rtsp streams disconnected and reconnect when using nvurisrcbin
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

My app is similar to python demo deepstream-rtsp-in-rtsp-out.
I’m using nvurisrcbin with smart record enabled and reconnect interval set to 20s.
Everything’s ok when camera is well connected, even if sometime it reconnects.
But I found one production use case with 10 rtsp streams. If theres error occurs, the code “GLib.timeout_add_seconds(5, perf_data.perf_print_callback)” will stops printing fps every 5 sec, and plugin “nvv4l2h264enc” seemed stop working because rtsp out display also not working , and the same time GPU utilization drops about 5% , probably the encode plugin cost.

-----------------------first log------------
2024-08-08 18:06:47,822 <frozen …deepstream.roadway.deepstream_roadway_monitor>(768) WARNING: Warning: gst-stream-error-quark: No data from source rtsp://admin:aq12345678@192.0.0.200:554/Streaming/Channels/101 since last 20 sec. Trying reconnection (1): gstdsnvurisrcbin.cpp(1266): watch_source_status (): /GstPipeline:pipeline0/GstBin:source-bin-06/GstDsNvUriSrcBin:uri-decode-bin

2024-08-08 18:06:47,823 <frozen …deepstream.roadway.deepstream_roadway_monitor>(768) WARNING: Warning: gst-resource-error-quark: Could not write to resource. (10): gstrtspsrc.c(6407): gst_rtspsrc_try_send (): /GstPipeline:pipeline0/GstBin:source-bin-06/GstDsNvUriSrcBin:uri-decode-bin/GstRTSPSrc:src:
Could not send message. (Received end-of-file)

2024-08-08 18:06:47,823 <frozen …deepstream.roadway.deepstream_roadway_monitor>(768) WARNING: Warning: gst-resource-error-quark: Could not write to resource. (10): gstrtspsrc.c(8096): gst_rtspsrc_close (): /GstPipeline:pipeline0/GstBin:source-bin-06/GstDsNvUriSrcBin:uri-decode-bin/GstRTSPSrc:src:
Could not send message. (Received end-of-file)

2024-08-08 18:06:48,360 <frozen …deepstream.roadway.deepstream_roadway_monitor>(606) INFO: Decodebin child added:h264parse131

2024-08-08 18:06:48,361 <frozen …deepstream.roadway.deepstream_roadway_monitor>(606) INFO: Decodebin child added:capsfilter131

2024-08-08 18:06:48,471 <frozen …deepstream.roadway.deepstream_roadway_monitor>(606) INFO: Decodebin child added:nvv4l2decoder131

2024-08-08 18:06:49,339 <frozen …deepstream.common.FPS>(66) INFO:
**PERF: {‘stream0’: 21.8, ‘stream1’: 24.8, ‘stream2’: 25.0, ‘stream3’: 25.2, ‘stream4’: 24.8, ‘stream5’: 24.8, ‘stream6’: 5.6, ‘stream7’: 0.0, ‘stream8’: 25.0, ‘stream9’: 24.2}

then no fps logs any more before restart.

-----------------------second log------------
2024-08-13 14:50:20,342 <frozen …deepstream.roadway.deepstream_roadway_monitor>(768) WARNING: Warning: gst-stream-error-quark: Could not decode stream. (7): gstrtph264depay.c(1287): gst_rtp_h264_depay_process (): /GstPipeline:pipeline0/GstBin:source-bin-00/GstDsNvUriSrcBin:uri-decode-bin/GstRtpH264Depay:depay:
Undefined packet type

2024-08-13 14:50:20,343 <frozen …deepstream.roadway.deepstream_roadway_monitor>(768) WARNING: Warning: gst-stream-error-quark: Could not decode stream. (7): gstrtph264depay.c(1287): gst_rtp_h264_depay_process (): /GstPipeline:pipeline0/GstBin:source-bin-00/GstDsNvUriSrcBin:uri-decode-bin/GstRtpH264Depay:depay:
Undefined packet type

2024-08-13 14:50:20,918 <frozen …deepstream.roadway.deepstream_roadway_monitor>(768) WARNING: Warning: gst-resource-error-quark: Could not read from resource. (9): gstrtspsrc.c(5560): gst_rtspsrc_loop_interleaved (): /GstPipeline:pipeline0/GstBin:source-bin-00/GstDsNvUriSrcBin:uri-decode-bin/GstRTSPSrc:src:
Could not receive message. (System error)

2024-08-13 14:50:20,918 <frozen …deepstream.roadway.deepstream_roadway_monitor>(768) WARNING: Warning: gst-stream-error-quark: Internal data stream error. (1): gstrtspsrc.c(6057): gst_rtspsrc_loop (): /GstPipeline:pipeline0/GstBin:source-bin-00/GstDsNvUriSrcBin:uri-decode-bin/GstRTSPSrc:src:
streaming stopped, reason error (-5)
2024-08-13 14:50:33,340 <frozen …deepstream.common.FPS>(66) INFO:
**PERF: {‘stream0’: 0.0, ‘stream1’: 25.0, ‘stream2’: 25.2, ‘stream3’: 25.6, ‘stream4’: 25.0, ‘stream5’: 25.0, ‘stream6’: 24.4, ‘stream7’: 0.0, ‘stream8’: 25.2, ‘stream9’: 24.6}
2024-08-13 14:50:38,340 <frozen …deepstream.common.FPS>(66) INFO:
**PERF: {‘stream0’: 0.0, ‘stream1’: 25.0, ‘stream2’: 25.0, ‘stream3’: 25.0, ‘stream4’: 22.8, ‘stream5’: 25.0, ‘stream6’: 25.6, ‘stream7’: 0.0, ‘stream8’: 25.0, ‘stream9’: 25.6}

then no fps logs any more before restart.

3

nvurisrcbin will only reconnect if it cannot read data.

The above error means that the rtspsrc element has an error, and the program should exit at this time.

You can monitor errors and restart the program in the following function

def bus_call(bus, message, loop):
    t = message.type
    if t == Gst.MessageType.EOS:
        sys.stdout.write("End-of-stream\n")
        loop.quit()
    elif t==Gst.MessageType.WARNING:
        err, debug = message.parse_warning()
        sys.stderr.write("Warning: %s: %s\n" % (err, debug))
    elif t == Gst.MessageType.ERROR:
        err, debug = message.parse_error()
        sys.stderr.write("Error: %s: %s\n" % (err, debug))
        loop.quit()
    elif t == Gst.MessageType.ELEMENT:
        sys.stdout.write("Element: %s\n" % message.src.get_name())
    return True
4 
Could not receive message. (System error)
Could not send message. (Received end-of-file)

These error logs happened every 5 minutes before my issue, may be the network with cameras is not good. I was wondering why the GLib.timeout_add_seconds and encode plugin suddenly stops working, but inference is still working with logs continue printing in the function nvanalytics_src_pad_buffer_probe. Thank you!

5

If I restart program when these warnings occur, the program will restart all the time. the reconnection feature of nvurisrcbin will be meaningless. How to solve this problem without restarting program?

6

Application errors cannot be caught in element, so nvurisrcbin can only handle reconnection when network data cannot be read.

You can refer to the bus_callback in deepstream-app.c for handling reconnection

g_printerr ("ERROR from %s: %s\n",
          GST_OBJECT_NAME (message->src), error->message);
      if (debuginfo) {
        g_printerr ("Debug info: %s\n", debuginfo);
      }

      NvDsSrcParentBin *bin = &appCtx->pipeline.multi_src_bin;
      GstElement *msg_src_elem = (GstElement *) GST_MESSAGE_SRC (message);
      gboolean bin_found = FALSE;
      /* Find the source bin which generated the error. */
      while (msg_src_elem && !bin_found) {
        for (i = 0; i < bin->num_bins && !bin_found; i++) {
          if (bin->sub_bins[i].src_elem == msg_src_elem ||
              bin->sub_bins[i].bin == msg_src_elem) {
            bin_found = TRUE;
            break;
          }
        }
        msg_src_elem = GST_ELEMENT_PARENT (msg_src_elem);
      }
7

using rtp protocol 0 upd+multicast instead of tcp can solve this problem, no more warnings .

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.