How can I improve instability of FPS when I use RTSP stream

masaki_yamagishi · June 10, 2024, 3:31pm

Sorry for your waiting.

In conclusion, it seem no improvement even increasing buffer-pool-size.

I attached results of command you suggested.

could you provide the nvstreammux 1.log by the following cmd. wondering if there is abnormal printings.
export GST_DEBUG=nvstreammux:6 && gst-launch-1.0 rtspsrc location=rtsp://address ! rtph264depay ! h264parse ! nvv4l2decoder ! mux.sink_0 nvstreammux name=mux batch-size=1 width=1280 height=720 batched-push-timeout=71000 live-source=1 ! fpsdisplaysink video-sink= nv3dsink text-overlay=false sync=true -v >1.log 2>1.log

1.log (483.5 KB)

could you provide the nvstreammux 2.log by the following cmd. I increased the bufferpool size.
export GST_DEBUG=nvstreammux:6 && gst-launch-1.0 rtspsrc location=rtsp://address ! rtph264depay ! h264parse ! nvv4l2decoder ! mux.sink_0 nvstreammux name=mux batch-size=1 width=1280 height=720 batched-push-timeout=71000 live-source=1 buffer-pool-size=10 ! fpsdisplaysink video-sink= nv3dsink text-overlay=false sync=true -v >2.log 2>2.log

2.log (633.8 KB)

Best regards,

fanzh · June 12, 2024, 7:19am

|0:00:04.002979019 e[36m46163e[00m 0xffff68002310 e[37mDEBUG e[00m e[00m nvstreammux gstnvstreammux.cpp:532:gst_nvstreammux_chain:e[00m Got buffer 0xffff340a8720 from source 0 pts = 0:00:02.678010103
0:00:12.136815518 e[36m46163e[00m 0xffff68002310 e[37mDEBUG e[00m e[00m nvstreammux gstnvstreammux.cpp:532:gst_nvstreammux_chain:e[00m Got buffer 0xffff340a8c00 from source 0 pts = 0:00:11.618252071

as the logs in 1.log, shown, nvstreammux did not receive any buffer during about 8 seconds. it is not nvstreammux’s issue. it is related to upstream plugins.

form the camera configuration tool, what is the fps? with nvstreammux, how often did the video stutter? how long will the video stuttering last?
without nvstreammux, as tested on May 22, if running for a long time, will the video stuttering happen? we need to rule out the accidental RTSP receiving issue.
to narrow down this issue, could you provide the 3.log by the following cmd? Thanks! we need more logs of the upstream plugins.

export GST_DEBUG=3,rtpjitterbuffer:6,nvstreammux:6,v4l2videodec:6  &&  gst-launch-1.0 rtspsrc location=rtsp://10.19.226.223/media/video1 latency=8000    ! rtph264depay ! h264parse ! nvv4l2decoder ! mux.sink_0 nvstreammux name=mux batch-size=1 width=1280 height=720 batched-push-timeout=71000 live-source=1 ! nv3dsink

masaki_yamagishi · June 16, 2024, 4:51pm

Hi, @fanzh

form the camera configuration tool, what is the fps? with nvstreammux, how often did the video stutter? how long will the video stuttering last?

The fps is 15. I use below camera, and it has only 15 fps mode.

And, video stuttered is always occurred when I report it.

without nvstreammux, as tested on May 22, if running for a long time, will the video stuttering happen? we need to rule out the accidental RTSP receiving issue.

I ran below command for 1 or more minutes.

gst-launch-1.0 rtspsrc location=rtsp://address ! rtph264depay ! h264parse ! nvv4l2decoder ! nv3dsink

And as attached video, there was no problem at any times.

to narrow down this issue, could you provide the 3.log by the following cmd? Thanks! we need more logs of the upstream plugins.

I ran below command, and I attached results of it.

export GST_DEBUG=3,rtpjitterbuffer:6,nvstreammux:6,v4l2videodec:6 && gst-launch-1.0 rtspsrc location=rtsp://10.19.226.223/media/video1 latency=8000 ! rtph264depay ! h264parse ! nvv4l2decoder ! mux.sink_0 nvstreammux name=mux batch-size=1 width=1280 height=720 batched-push-timeout=71000 live-source=1 ! nv3dsink >3.log 2>3.log

3.log (10.3 MB)

I’m confused little because nvstreammux may not be a cause of stuttered.
Do you have any idea?

Best regards,

fanzh · June 17, 2024, 4:36am

Thanks for the sharing! In the new log 3.log, there are some issues in nvtreammux. nvstreammux received buffers at almost even PTS interval, but sometimes there is a big gas between output PTS. for example, the log at 0:00:13.063609988.

there is no more detailed log. I suppose it is because of the resolution conversion in nvstreammux. please try the following cmd without resolution conversion and provide the log 4.log. the resolution setting of nvstreammux is same with the source.

export GST_DEBUG=3,rtpjitterbuffer:6,nvstreammux:6,v4l2videodec:6 && gst-launch-1.0 rtspsrc location=rtsp://10.19.226.223/media/video1 latency=8000  ! rtph264depay ! h264parse ! nvv4l2decoder ! mux.sink_0 nvstreammux name=mux batch-size=1 width=640 height=360 batched-push-timeout=67000 live-source=1 ! nv3dsink >4.log 2>4.log

to narrow down this issue, can we access the camera remotely? we want to add log to debug. did you try on other devices?

masaki_yamagishi · June 27, 2024, 6:15am

Hi, @fanzh

Sorry for my late reply.

I’ll try to below.

there is no more detailed log. I suppose it is because of the resolution conversion in nvstreammux. please try the following cmd without resolution conversion and provide the log 4.log. the resolution setting of nvstreammux is same with the source.
export GST_DEBUG=3,rtpjitterbuffer:6,nvstreammux:6,v4l2videodec:6 && gst-launch-1.0 rtspsrc location=rtsp://10.19.226.223/media/video1 latency=8000 ! rtph264depay ! h264parse ! nvv4l2decoder ! mux.sink_0 nvstreammux name=mux batch-size=1 width=640 height=360 batched-push-timeout=67000 live-source=1 ! nv3dsink >4.log 2>4.log

Then, what do you mean

can we access the camera remotely?
did you try on other devices?

The IP-cam, Tapo C510w, I’m using has only wireless connection.
This camera can feed TP-Link app, and it works fine.

And I have IP-cam only this, so I can’t check with another IP-cam.

Best regards,

masaki_yamagishi · July 7, 2024, 8:04pm

Hi, @fanzh

This issue is still severe, so please help me to solve it.

I tried below command.

export GST_DEBUG=3,rtpjitterbuffer:6,nvstreammux:6,v4l2videodec:6 && gst-launch-1.0 rtspsrc location=rtsp://10.19.226.223/media/video1 latency=8000 ! rtph264depay ! h264parse ! nvv4l2decoder ! mux.sink_0 nvstreammux name=mux batch-size=1 width=640 height=360 batched-push-timeout=67000 live-source=1 ! nv3dsink >4.log 2>4.log

And attached the result of it.

4.log (10.1 MB)

The stuttered was occurred.

Then, I also confirmed behavior of IP cam I using with Tplink App.

It worked fine, so network connection of IP cam and function of its own have no problem.

Best regards,

fanzh · July 9, 2024, 2:58am

Thanks for the sharing! As the log shown in 4.log, from 0:00:12.00 to 0:00:13.12, there is only one output frame from decoder. that is why nvstreammux sometimes can’t not receive any buffer. we need more decoder logs. could you provide 5.log after doing the following test? Thanks!

export GST_DEBUG=3,rtpjitterbuffer:6,h264parse:6,v4l2videodec:6,videodecoder:6,nvstreammux:6 &&  gst-launch-1.0 rtspsrc location=rtsp://10.19.226.223/media/video1 latency=8000 ! rtph264depay ! h264parse ! nvv4l2decoder ! mux.sink_0 nvstreammux name=mux batch-size=1 width=640 height=360 batched-push-timeout=67000 live-source=1 ! nv3dsink >5.log 2>5.log

we also need a log without nstreammux. could you provide 6.log after doing the following test? Thanks!

export GST_DEBUG=3,rtpjitterbuffer:6,h264parse:6,v4l2videodec:6,videodecoder:6 &&  gst-launch-1.0 rtspsrc location=rtsp://10.19.226.223/media/video1 latency=8000 ! rtph264depay ! h264parse ! nvv4l2decoder  ! nv3dsink >5.log 2>5.log

To rule out the hardware decoder issue. we can try “software decoder + nvvideoconvert”. will the following cmd stutter? Thanks!

gst-launch-1.0 rtspsrc location=rtsp://10.19.226.223/media/video1 latency=8000 ! rtph264depay ! h264parse ! avdec_h264  ! nvvideoconvert ! mux.sink_0 nvstreammux name=mux batch-size=1 width=640 height=360 batched-push-timeout=67000 live-source=1 ! nv3dsink

fanzh · July 22, 2024, 2:10pm

please refer to this fix in this topic rtsp-shows-stuttering-image-every-2-seconds, which has the same issue with your app. the output video stutters every some times.

use gst-launch to confirm the solution. I added " sync-inputs=1" for nvstreammux.

gst-launch-1.0 rtspsrc location=rtsp://10.19.226.223/media/video1 latency=2000 ! rtph264depay ! h264parse ! nvv4l2decoder ! mux.sink_0 nvstreammux name=mux  width=640 height=360 batched-push-timeout=67000 batch-size=1 live-source=1 sync-inputs=1 ! nv3dsink

port the solution to app.
add “sync-inputs=1” in [streammux] configuration of deepstream.

masaki_yamagishi · July 23, 2024, 8:10am

Thank you a lot, @fanzh

I’m checking your posts, but I’m too busy now, so please wait a while.
I’m going to validate commands you instructed by this weekend.

I still need your kindly support.

Best regards,

masaki_yamagishi · July 29, 2024, 1:15am

Hi, @fanzh.

Sorry for my late reply.

My developing environment is updated to Jetson Linux 36.3.
But this issue is still appeared.

Then, I shared some results.

export GST_DEBUG=3,rtpjitterbuffer:6,h264parse:6,v4l2videodec:6,videodecoder:6,nvstreammux:6 && gst-launch-1.0 rtspsrc location=rtsp://tapoc510w:tapoc510w@192.168.1.29/stream2 latency=8000 ! rtph264depay ! h264parse ! nvv4l2decoder ! mux.sink_0 nvstreammux name=mux batch-size=1 width=640 height=360 batched-push-timeout=67000 live-source=1 ! nv3dsink >5.log 2>5.log

5.log (13.7 MB)

export GST_DEBUG=3,rtpjitterbuffer:6,h264parse:6,v4l2videodec:6,videodecoder:6 && gst-launch-1.0 rtspsrc location=rtsp://tapoc510w:tapoc510w@192.168.1.29/stream2 latency=8000 ! rtph264depay ! h264parse ! nvv4l2decoder ! nv3dsink >6.log 2>6.log

6.log (9.5 MB)

gst-launch-1.0 rtspsrc location=rtsp://tapoc510w:tapoc510w@192.168.1.29/stream2 latency=8000 ! rtph264depay ! h264parse ! avdec_h264 ! nvvideoconvert ! mux.sink_0 nvstreammux name=mux batch-size=1 width=640 height=360 batched-push-timeout=67000 live-source=1 ! nv3dsink

gst-launch-1.0 rtspsrc location=rtsp://tapoc510w:tapoc510w@192.168.1.29/stream2 latency=2000 ! rtph264depay ! h264parse ! nvv4l2decoder ! mux.sink_0 nvstreammux name=mux width=640 height=360 batched-push-timeout=67000 batch-size=1 live-source=1 sync-inputs=1 ! nv3dsink

Then, I tried to add setting of “sync-inputs=1” to Deepstream Reference Application of PeopleNet.
But if adding that column, the sink window was blacked out.
So I couldn’t confirm there were any improvement or not.

Result of Integrated “sync-inputs=1”.

Result of comment outed “sync-inputs=1” (please ignore inference performance).

What should I do to improve it?

Thank you very much for your kindly support.

Best regards,

fanzh · July 29, 2024, 5:26am

Great! from the gst-launch command-line test in the point 4, there is no the “jerky” issue again. we only port the properties to app.
About " if adding that column, the sink window was blacked out.", from the screenshot, there is rtpjitterbuffer “Queue full, dropping old packet” abnormality. please add/correct the following configurations.

[source0]
......
latency=2000

[streammux]
......
batch-size=1
live-source=1
batched-push-timeout=70176
sync-inputs=1

“latency=2000” is for rtpjitterbuffer “Queue full…” error. “batched-push-timeout=70176” is for fps 14.25. 70176(ms)=1000000/14.25.
if still can’t work. please share your configuration file rtsp_test.txt. and please share the test log 7.log by the following command-line.

deepstream-app -c rtsp_test.txt >7.log 2>7.log

masaki_yamagishi · August 4, 2024, 4:47pm

Hi, @fanzh

Hooray!

Following your configuration, I found DeepStream showed smooth output from RTSP source.
I cannot thank you enough.

Result of modified PeopleNet file with 720p RTSP stream from my IP camera.

test_rtsp_20240805.txt (3.9 KB)
I attached log file for your reference.
7.log (4.2 KB)

Using modified TrafficCamNet file with 1080p RTSP stream from the IP camera, it also worked fine.

test_rtsp_20240805_trafficcamnet.txt (5.7 KB)

Then, I want to summarize this issue and solution.
Is this problem occurred by nvstreammux element, or rtpjitterbuffer element, or another?

I found method of updating rtpjitterbuffer in newest DeepStream installation web page.
Can we solve this issue of DeepStream stutterred with RTSP stream to apply only this method?
https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_Installation.html#install-prerequisite-packages
(See “Note” column)

Best regards,

fanzh · August 5, 2024, 2:12am

Here is the reason for stuttering.
the pipeline looks like “rtspsrc(rtpjitterbuffer) → nvv4l2decoder → nvstreammux → nvinfer → …”. from the comment on Jun 10 and Jun 12, sometimes the decoder output many frames at the same time. sometimes it did not output frame for a long time. this is related to the receiving. video I frame is bigger than other frames. I receiving cost too much time. if no sync-inputs in nvstreammux, some packets which entered to nvstreamux at the same time will be attached the similar PTS, which cause sink renders quickly. if with sync-inputs in nvstreammux, nvstreammux will attach a new PTS based on fps. then sink can render evenly.
here is the reason for “Queue full, dropping old packet”.
this is the receiving issue in Gstreamer Opensource plugin rtpjitterbuffer. Increasing latency of rtpjitterbuffer can improve. In deepstream-app, the default value is 100ms, and this value is configurable.

No. the link is for the stuck issue on reaching EOS. AYK, live source will not EOS when running, unless you quit the application.

masaki_yamagishi · August 5, 2024, 4:52pm

Hi, @fanzh

Thank you for your really dedicated support.

I understood this issue is caused by nvstreammux without sync-inputs.
But fundamental cause is the bug of rtpjitterbuffer related receiving i frame (key frame) whose size is larger.
And, this is solved to increase latency of rtpjitterbuffer.

OK, that’s it.
Thank you so much for a long time.

I’m glad if I contributed something to DeepStream developer community.

system · August 19, 2024, 4:53pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
RTSP latency does not work with NVSTREAMMUX DeepStream SDK nvbugs	37	5148	January 23, 2022
Instance with invalid (NULL) class pointer DeepStream SDK	58	3032	December 29, 2022
TX2 H264 RTSP Stream decoding issues Jetson TX2	27	11557	October 18, 2021
RTSP stream not work "Stream format not found, dropping the frame" DeepStream SDK rtsp , gstreamer , nvbugs	24	1580	January 3, 2024
Deepstream 5.1 rtsp and nvinfer issues DeepStream SDK	18	327	July 9, 2024
Problem using gstreamer with opencv and cuda on TX2 with Jetpack 4.3 Jetson TX2 rtsp , opencv , cuda , gstreamer	21	3302	October 18, 2021
Gstreamer TCPserversink 2-3 seconds latency Jetson Xavier NX camera , opencv , gstreamer	12	17735	October 3, 2021
Reconnection Issue DeepStream SDK	40	1499	January 25, 2024
jetson.utils.videoSource.Capture() latency issue Jetson Xavier NX jetson-inference	14	3408	October 18, 2021
Problems about the image resolution of rtsp stream DeepStream SDK deepstream	18	29	December 14, 2024

How can I improve instability of FPS when I use RTSP stream

Related topics