Improving Gstreamer RTSP and/or Nvcompositor latency

Robotics & Edge Computing Jetson & Embedded Systems Jetson Xavier NX
1

I’m streaming a video via an RTSP stream and then compositing it with another video using nvcompositor at 60FPS (or as close as possible to that).

When I run the RTSP stream like so I get the desired framerate:

./test-launch "v4l2src device=/dev/video2 ! video/x-raw, width=2560, height=720, framerate=60/1 ! nvvidconv ! nvv4l2h265enc maxperf-enable=1 ! h265parse ! rtph265pay name=pay0"

gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test latency=0 ! rtph265depay ! h265parse ! nvv4l2decoder ! nvvidconv ! xvimagesink sync=0

and when I run nvcompositor with direct streaming I also get the desired latency:

gst-launch-1.0 nvcompositor name=comp  \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=2560 sink_0::height=720 sink_0::zorder=1 sink_0::alpha=1 \
sink_1::xpos=0 sink_1::ypos=0 sink_1::width=2560 sink_1::height=720 sink_1::zorder=2 sink_1::alpha=0.1 ! \
'video/x-raw(memory:NVMM),format=RGBA' ! nv3dsink sync=0  \
v4l2src device=/dev/video2 ! video/x-raw,format=YUY2,width=2560,height=720,framerate=30/1 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! comp.sink_0    \
videotestsrc ! videoconvert ! video/x-raw,format=RGBA ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA,width=2560,height=720' ! comp.sink_1

But when I put it all together there’s latency that looks like it’s around 20-30FPS:

gst-launch-1.0 nvcompositor name=comp  \ 
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=2560 sink_0::height=720 sink_0::zorder=1 sink_0::alpha=1 \ 
sink_1::xpos=0 sink_1::ypos=0 sink_1::width=2560 sink_1::height=720 sink_1::zorder=2 sink_1::alpha=0.1 !  \ 
'video/x-raw(memory:NVMM),format=RGBA' ! nv3dsink    \ 
rtspsrc location=rtsp://127.0.0.1:8554/test latency=0 ! rtph265depay ! h265parse ! nvv4l2decoder ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! comp.sink_0    \ 
videotestsrc ! videoconvert ! video/x-raw,format=RGBA ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA,width=2560,height=720' ! comp.sink_1

I realize that the compositor examples use nv3dsink whereas the other one uses xvimagesink and adds the sync=0 property. I couldn’t figure out how to use xvimagesink in the compositor so I didn’t test if the latency came from the sink I was using. I’ve also already tried jetson_clocks, and adding the maxperf-enable property to the encoder. Any advice on how to improve latency would be much appreciated

3

Hi.
Please add is-live=1 to videotestsrc:

videotestsrc is-live=1

So that the source generates frames in fixed interval. It is more close to live-source case.

And do you check frame rate with fpsdisplaysink?

4

I get an error when I try to add is-live:

gst-launch-1.0 nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=2560 sink_0::height=720 sink_0::zorder=1 sink_0::alpha=1 sink_1::xpos=0 sink_1::ypos=0 sink_1::width=2560 sink_1::height=720 sink_1::zorder=2 sink_1::alpha=0.1 ! 'video/x-raw(memory:NVMM),format=RGBA' ! nv3dsink rtspsrc location=rtsp://127.0.0.1:8554/test latency=0 ! rtph265depay ! h265parse ! nvv4l2decoder ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! comp.sink_0  videotestsrc is-live=1 ! videoconvert ! video/x-raw,format=RGBA ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA,width=2560,height=720' ! comp.sink_1
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE 
Pipeline is live and does not need PREROLL ...
Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://127.0.0.1:8554/test
ERROR: from element /GstPipeline:pipeline0/GstRTSPSrc:rtspsrc0: Could not open resource for reading and writing.
Additional debug info:
gstrtspsrc.c(7469): gst_rtspsrc_retrieve_sdp (): /GstPipeline:pipeline0/GstRTSPSrc:rtspsrc0:
Failed to connect. (Generic error)
ERROR: pipeline doesn't want to preroll.
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

How would I use the FPS sink? I tried replacing it with my current sink and I get

 gst-launch-1.0 nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=2560 sink_0::height=720 sink_0::zorder=1 sink_0::alpha=1 sink_1::xpos=0 sink_1::ypos=0 sink_1::width=2560 sink_1::height=720 sink_1::zorder=2 sink_1::alpha=0.1 ! 'video/x-raw(memory:NVMM),format=RGBA' ! fpsdisplaysink rtspsrc location=rtsp://127.0.0.1:8554/test latency=0 ! rtph265depay ! h265parse ! nvv4l2decoder ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! comp.sink_0  videotestsrc is-live=1 ! videoconvert ! video/x-raw,format=RGBA ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA,width=2560,height=720' ! comp.sink_1
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE 
Pipeline is live and does not need PREROLL ...
Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://127.0.0.1:8554/test
ERROR: from element /GstPipeline:pipeline0/GstRTSPSrc:rtspsrc0: Could not open resource for reading and writing.
Additional debug info:
gstrtspsrc.c(7469): gst_rtspsrc_retrieve_sdp (): /GstPipeline:pipeline0/GstRTSPSrc:rtspsrc0:
Failed to connect. (Generic error)
ERROR: pipeline doesn't want to preroll.
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
5

Hi,
From the print:

ERROR: from element /GstPipeline:pipeline0/GstRTSPSrc:rtspsrc0: Could not open resource for reading and writing.

Looks like the RTSP source is not correctly set up. Please check if you stop the RTSP server.

7

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.