Deepstream pipeline with RTSP stream stalled time to time and stoped permanently

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

Please provide complete information as applicable to your setup.

• Hardware Platform - DGPU
• DeepStream Version - 6.0.1
• TensorRT Version - 8.0.1
• NVIDIA GPU Driver Version - 470.161.03
• CUDA Version - 11.4
• Issue Type - bug
• How to reproduce the issue ?
rtspsrc ->h264depay->h264parse->tee->queue->decodebin->queue->tee->nvvidconv->capsfilter->queue->nvstreammux->nvinfer->nvtracker->queue->nvtiler->nvvideoconvert->nvosd->nveglglessink
• Requirement details
After running the pipeline for 10mins detection video get stuck for several minutes and continue running again. Then after random intervals it also get stuck with 0.0 stream fps. Some warning logs are printed from nvv4l2. Sometimes detection video has a time lag with realtime. Also detection video plays faster than the real camera FPS after getting stuck. gstvideodecoder.c:2780:gst_video_decoder_prepare_finish_frame: decreasing timestamp warning log is printed all the time. After running 1hr frame processing become permanently stopped with FPS 0.0. But the EOS is not recieved.**
deepstream-stalling.log (68.6 KB)
deepstream-stalling.log (4.9 KB)

4

Frame processing permanently stopped after following log, stream FPS goes to 0.0

INFO   GST_EVENT gstevent.c:814:gst_event_new_caps:\u001B[00m creating caps event video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)014d002affe10018674d002a963540f0044fcb37010101400003a9800057e40101000468ee31b2, level=(string)4.2, profile=(string)main"}
INFO   h264parse gsth264parse.c:1793:gst_h264_parse_update_src_caps:<h264parse_elem0>\u001B[00m resolution changed 1920x1080"}
INFO   baseparse gstbaseparse.c:4004:gst_base_parse_set_latency:<h264parse_elem0>\u001B[00m min/max latency 0:00:00.083333333, 0:00:00.083333333"}
INFO   GST_EVENT gstevent.c:814:gst_event_new_caps:\u001B[00m creating caps event video/x-h264, stream-format=(string)avc, alignment=(string)au, codec_data=(buffer)014d002affe10018674d002a963540f0044fcb37010101400003a9800057e40101000468ee31b2, level=(string)4.2, profile=(string)main, width=(int)1920, height=(int)1080, framerate=(fraction)12/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true"}
INFO   baseparse gstbaseparse.c:4004:gst_base_parse_set_latency:<h264parse_elem0>\u001B[00m min/max latency 0:00:00.083333333, 0:00:00.083333333"}
INFO   baseparse gstbaseparse.c:4004:gst_base_parse_set_latency:<h264parse0>\u001B[00m min/max latency 0:00:00.083333333, 0:00:00.083333333"}
INFO   GST_EVENT gstevent.c:814:gst_event_new_caps:\u001B[00m creating caps event video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)4.2, profile=(string)main, width=(int)1920, height=(int)1080, framerate=(fraction)12/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true"}
INFO   basetransform gstbasetransform.c:1308:gst_base_transform_setcaps:<capsfilter0>\u001B[00m reuse caps"}
INFO   GST_EVENT gstevent.c:814:gst_event_new_caps:\u001B[00m creating caps event video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, level=(string)4.2, profile=(string)main, width=(int)1920, height=(int)1080, framerate=(fraction)12/1, interlace-mode=(string)progressive, chroma-format=(string)4:2:0, bit-depth-luma=(uint)8, bit-depth-chroma=(uint)8, parsed=(boolean)true"}
[debug] PipelineMetadata recieved."}
[trace] ObjectMetaMsg, jobId: fe1b6462-004f-47ad-8842-d6de35c0af8d timestamp:1673432296"}
WARN   v4l2videodec gstv4l2videodec.c:1271:gst_v4l2_video_dec_loop:<nvv4l2decoder0>\u001B[00m Decoder is producing too many buffers"}
WARN   v4l2videodec gstv4l2videodec.c:1271:gst_v4l2_video_dec_loop:<nvv4l2decoder0>\u001B[00m Decoder is producing too many buffers"}
INFO   task gsttask.c:316:gst_task_func:<nvv4l2decoder0:src>\u001B[00m Task going to paused"}
INFO   task gsttask.c:318:gst_task_func:<nvv4l2decoder0:src>\u001B[00m Task resume from paused"}
5

There is no clue in the log. Can you monitor your rtsp source status? What you have seen on your screen?

6

I notices vlc player plays the rtsp stream without being stuck. On the other hand nveglglessink is stuck in a single frame. I let the setup run few hours. still stuck in the same frame.

If there’s a method to monitor the PERF FPS and if it stays 0.0 for several seconds I can trigger the internal stream reconnection mechanism.

Also something abnormal I notices. My application is capable of adding and removing sources dynamically. I added the same rtsp stream as the second stream. Suddenly both sources started running. Source which was stuck earlier with higher FPS (40FPS) than the camera FPS (25FPS) and new source in normal FPS. After sometime both streams ran in normal 25FPS.

7

Can you debug with rtsprc? rtspsrc (gstreamer.freedesktop.org) Have you set “sync” property of nveglglessink to false? The GPU performance can be monitored with “nvidia-smi mon”.

8

Following parameters are set to the nveglglessink

  • “sync”, false
  • “async”, false
  • “qos”, 0
    If I use a fakesink what parameters should I set.
    Also what are the recommended parameter value for rtspsrc element?
9

Also I notices my rtsp stream runs in 12FPS and changes to 25FPS sometimes automatically. Can this be an issue to the pipeline with nvv2l4decorder?

10

The same as nveglglessink

Depends on your RTSP source. Such as “latency”, “protocols”,… Please refer to rtspsrc (gstreamer.freedesktop.org)

11

The decoder does not care about FPS. Are there multiple sources in your pipeline?

1 Like
12

Yes, my pipeline has multiple sources. Add and delete sourcebins in runtime.

I set the following parameters and streams are running without disconnection for more than 1 day now. I think the problem is solved for now.

gst-rtspsrc

    g_object_set(G_OBJECT(bin_data->src_elem), "latency", 1000, nullptr);
    g_object_set(G_OBJECT(bin_data->src_elem), "drop-on-latency", true, nullptr);
    g_object_set(G_OBJECT(bin_data->src_elem), "udp-buffer-size", 2000000, nullptr);
    g_object_set(G_OBJECT(bin_data->src_elem), "udp-reconnect", true, nullptr);

nveglsink and fakesink

    g_object_set(G_OBJECT(binData->sink), "sync", false, nullptr);
    g_object_set(G_OBJECT(binData->sink), "async", false, nullptr);
    g_object_set(G_OBJECT(binData->sink), "qos", 0, nullptr);
    g_object_set(G_OBJECT(binData->sink), "enable-last-sample", false, nullptr);

Also the network settings from Frequently Asked Questions — DeepStream 6.1.1 Release documentation,

    sudo sysctl -w net.core.rmem_max=52428800
    sudo sysctl -p
1 Like
13

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.