Hi @mchi / NVIDIA,
Any update on this?
Seems we have a simialr issue. We are checking this … ping me again days later if I don’t get back.
Thanks!
Hi,
Just to let you know, we tried to isolate this problem by removing all plugins in the pipeline except the Mux-Demux part (by hard coding some parts of it and disabling some parts in the config file). With this, we had 3 RTSP->Mux->Demux->3 RTSP streams. We did not observe the issue. We can not say this proves anything but may provide some possible clues.
I have similar issues when recording to files using smart record feature. The recorded files seem to show some frames playing in reverse and it jumps around a lot.
If I set streammux live-source to 0 (even though I am using live sources - rtsp) it is much better and I do not see any of the frames playing in reverse. However its still not perfect and occasionally you see a pause in playback of the file and even some frames running in slow or fast motion.
Hello,
Has this been fixed in the latest DS 5.0 GA Release?
Just with the config you shared to run deepstream-app, met segmentation fault issue.
(deepstream-app:30180): GStreamer-CRITICAL **: 10:10:44.043: gst_element_get_static_pad: assertion ‘GST_IS_ELEMENT (element)’ failed
Segmentation fault (core dumped)
Seems you have other code changes, can you share the patch or whole source code which get some changes to us to have a try?
You may use this repo: https://drive.google.com/file/d/1dJvVo0FnELeZFEQFgl6LEDDP7ujDxUGH/view?usp=sharing
Hi,
We used DeepStream-5.0 General Vailability Release and noticed that with the default deepstream-app and 4 RTSP input Streams and 4 RTSP output streams of 1080p 30FPS, the Software encoded stream is lagging by nearly 2 seconds on the hardware mentioned above. GPU consumption just 37% and CPU consumption (for doing the Primary Inference, Tracking, OSD) is 17%. Streammux has been enabled with live-source=1, RTSP output bitrate is 4000000.
So, clearly, even when there is enough CPU power, the delay in the pipeline or encoding is causing huge difference in hardware encoded streams and software encoded streams (as outlined many a times earlier).
We hope this will be fixed soon.
I notice the delay too, on an nx with 4 rtsp inputs and 1 muxed rtsp output. Perhaps its network latency or attributable to the protocol. The rtsp inputs all show around 30 fps. I have only tried with the yolov3 tiny weights which don’t really seem that accurate. I’m only looking for people not cars. The lag is easy to reproduce.
This has been more than a month that this issue was reported. I hope NVIDIA has take the inputs and is working on resolving this. Multiple RTSP output is one one of the typical use cases in deployment of IVA solutions in the field.
Sorry for a long delay.
We are looking into this, thanks for your patience.
I managed to run on Jetson Xavier, with 4 rtsp input, 4 rtsp output, do not repro the issue.
with same modification for deepstream_app.c on T4, but met segmentation fault issue, is there anything missing on dgpu side?
for (int k = 0; k < MAX_SINK_BINS;k++) {
if(pipeline->instance_bins[i].sink_bin.sub_bins[k].sink){
NVGSTDS_ELEM_ADD_PROBE(latency_probe_id, pipeline->instance_bins[i].sink_bin.sub_bins[k].sink, “sink”,
latency_measurement_buf_prob, GST_PAD_PROBE_TYPE_BUFFER, appCtx);
break;
}
}
root@ed8d498d1abf:/opt/nvidia/deepstream/deepstream-5.0# ./sources/apps/sample_apps/deepstream-app/deepstream-app -c samples/configs/deepstream-app/dgpu.txt
*** DeepStream: Launched RTSP Streaming at rtsp://localhost:5000/ds-test ***
*** DeepStream: Launched RTSP Streaming at rtsp://localhost:5001/ds-test ***
(deepstream-app:3310): GStreamer-CRITICAL **: 11:11:20.318: gst_element_get_static_pad: assertion ‘GST_IS_ELEMENT (element)’ failed
Segmentation fault (core dumped)
Hi,
If yo are referring to the segmentation fault that occurs during pipeline creation, then it is fixed in the latest DeepStreamSDK 5.0 GA release of deepstream-app
HI
For a quick repro, I try to repro your issue on NANO, with 2 rtsp input stream, 2 rtsp output.
first issue: lag or latency, in video 2rtspinput-2rtspoutput-sync-1.mov, do you mean feel like stuck? with sink sync both set to 1
second issue: buffer cache, or frame re-processing, in video 2rtspinput-2rtspoutput-sync-0.mov, how could you verify this? with sink sync both set to 0
NOTE: for both videos, the bottom one is for rtsp source streaming. the upper two is for rtsp output. vlc running on another machine, since on NANO desktop to run vlc always quit with error. ffprobe show rtsp source running in 30fps
ffprobe version N-93005-gd92f06e Copyright (c) 2007-2019 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
configuration: --enable-nonfree --prefix=/home/tse/work/ffmpeg/install --enable-libnpp --extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64
libavutil 56. 26.100 / 56. 26.100
libavcodec 58. 44.100 / 58. 44.100
libavformat 58. 26.100 / 58. 26.100
libavdevice 58. 6.101 / 58. 6.101
libavfilter 7. 48.100 / 7. 48.100
libswscale 5. 4.100 / 5. 4.100
libswresample 3. 4.100 / 3. 4.100
Input #0, rtsp, from ‘rtsp://10.19.225.167:9002/stream’:
Metadata:
title : Unnamed
comment : N/A
Duration: N/A, start: 8.554708, bitrate: N/A
Stream #0:0: Video: h264 (High), yuv420p(progressive), 1920x1080, 30 fps, 30 tbr, 90k tbn, 60 tbc
Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp
2rtspinput-2rtspoutput-sync-1.mov
2rtspinput-2rtspoutput-sync-0.mov
Hi!
Thanks for looking into this.
-
With the 2rtspinput-2rtspoutput-sync-1.mov, (a) the output video is seen to be stuck at times and we have observed this as well (b) There is huge delay (of the order of 3-4 seconds (check the exit of the white mini van around 0:03 secs in the original stream in the bottom against 0:07 secs at the two processed frames. This behavior can also be easily seen when the recording ends)!
-
Similar behavior is seen in the second movie (2rtspinput-2rtspoutput-sync-0.mov) as well. However, buffer caching isn’t seen explicitly in either of these movies. It is easy to spot: The vehicle goes back a few feet and moves again in the forward direction again as if it reversed a few feet and resumed movement in forward direction). This is as if the pipeline cached few frames and re-played it through the pipeline. This is different from the two issues noted above.
See inline.
Hi,
Reproducing it is fairly simple: Just by increasing the load on CPU and GPU by increasing the number of input and output RTSP streams (as outlined in many of the messages and using the config files shown earlier). The same input RTSP stream can be fed into multiple sources.
Hi, I just started using deepstream-app today to do rtsp streaming. I’m using deepstream_python_apps/apps/deepstream-test1-rtsp-out, I changed the gstreamer pipeline to our own pipeline, which simply reads a usb sensor /dev/video0 then udpsink with rtsp server. When i use VLC to see the stream, it has probably 1 sec delay however i set ‘sync=1 or 0’. i also tried to reduce the caching in vlc but didn’t change anything… Just wondering if this is a common phenomenon of rtsp streaming or it’s just my own setup not right? Thanks!
4 rtsp input, 4 rtsp output, on NANO, GPU utility almost full, CPU around 30%
please see the video if have frame cache issue or you may tell how to verify frame cache issue
the bottom one is rtsp source.