Gstreamer RSTP server with nvcompositor latency increases as time goes by

Robotics & Edge Computing Jetson Systems Jetson AGX Xavier
1

Hi, I build gst-rtsp-server and run the test-launch with below command

./test-launch "( rtspsrc buffer-mode=none location=rtsp://192.168.123.33:8554/media latency=0 ! rtph265depay ! nvv4l2decoder enable-max-performance=true ! comp.sink_0 nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1280 sink_0::height=720 sink_1::xpos=1280 sink_1::ypos=0 sink_1::width=1280 sink_1::height=720 sink_2::xpos=2560 sink_2::ypos=0 sink_2::width=1280 sink_2::height=720 sink_3::xpos=0 sink_3::ypos=720 sink_3::width=1280 sink_3::height=720 sink_4::xpos=1280 sink_4::ypos=720 sink_4::width=1280 sink_4::height=720 sink_5::xpos=2560 sink_5::ypos=720  sink_5::width=1280 sink_5::height=720 sink_6::xpos=0 sink_6::ypos=1440 sink_6::width=1280 sink_6::height=720 sink_7::xpos=1280 sink_7::ypos=1440 sink_7::width=1280 sink_7::height=720 comp. ! video/x-raw(memory:NVMM),format=(string)RGBA,width=3840,height=2160,framerate=(fraction)20/1 ! nvvidconv interpolation-method=2 ! video/x-raw(memory:NVMM),format=(string)NV12,width=3840,height=2160,framerate=(fraction)20/1 ! nvv4l2h264enc maxperf-enable=true control-rate=1 preset-level=1 insert-sps-pps=true ! h264parse ! rtph264pay name=pay0 )"

IP camera send H265 stream with 30fps and I composit it and convert to H264.

and get the media stream with below command

gst-launch-1.0 rtspsrc location=rtsp://Xavier_IP:8554 latency=0 ! rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! autovideosink sync=false

At fist latency was very low and it worked okay but latency increases as time goes by.
When I use test-launch with single camera without nvcompositor there isn’t increasement.

It doesn’t depend on number of cameras, I tried it with few cameras and single camera but the phenomenon was same.
Also I checked stream from the IP camera but there wasn’t latency increasment.

What is the reason for latency increasement an what can I do to keep the latency?

Thank you,

3

Hi,
Please share which release you are using. And for further check, you may try videotestsrc, use USB camera to try v4l2src, or use Bayer camera to try nvarguscamerasrc. See if the issue is seen with other types of sources.

And you may also try to run VIC engine at maximum clock:
Nvvideoconvert issue, nvvideoconvert in DS4 is better than Ds5? - #3 by DaneLLL
This enables nvcompositor in maximum throughput.

4

Hi,
I’m using 1.16.3 found by gst-inspect-1.0 --gst-version.
I don’t have usb or bayer right now, I can try with videotestsrc but how can I check latency with that?

I applied vic frequency as maximum and I’m testing it now. If it resolve this problem I’ll let you know.

And I’m planning to apply below patches

I’ll download source code at below site

5

This one makes latency lower but as time goes by latency increased once again.
I applied two patches I commented. I’ll let you know if it works.

6

Hi,
Please run the command and check if you use r35.4.1

cat /etc/nv_tegra_release

7

Hi,
This is output of command

# R35 (release), REVISION: 2.1, GCID: 32413640, BOARD: t186ref, EABI: aarch64, DATE: Tue Jan 24 23:38:33 UTC 2023

Can I just build gstreamer part from r35.4.1 and apply it?

And I just found that when I use multiple IP camera each latency of component of nvcompositor varies.

8

Hi,
You may try to build nvcompositor of r35.4.1 and apply it to r35.2.1 for a try. The plugins are open source and packetized into
https://developer.nvidia.com/embedded/jetson-linux-r3541

Driver Package (BSP) Sources

9

Hi,
I applied r35.4.1 and also r35.2.1 but both of them has increasment in latency as time goes by.

I wonder that if I make 9 configure(comp0~comp8) to nvcompositor and get fewer stream, does it make problem?

10

Hi,
Please try UDP and set to sync=false. Here is a reference setup:
Gstreamer TCPserversink 2-3 seconds latency - #5 by DaneLLL

To see if it is same phenomenon in UDP, or specific to RTSP.

11

Hi,
I tried the reference and I got this error

WARNING: erroneous pipeline: could not link comp to nvvconv0, comp can't handle caps video/x-raw, format=(string)RGBA, width=(int)3840, height=(int)2160, framerate=(fraction)20/1
12

Hi,
CPU buffer is not supported in nvcompositor. Please try

video/x-raw(memory:NVMM), format=(string)RGBA, width=(int)3840, height=(int)2160, framerate=(fraction)20/1
13

Hi,
I tested with udpsink. It seems like increasement of latency doesn’t occur in upd communication.
I’m sending stream to web server which only can take rtsp not rtp.
In this case, is there something I can do?
Can I send udp stream with gst-rtsp-server?

14

Hi again,

I noticed that below address shows how to send udp stream

When I applied it and see it with my laptop with below command

gst-launch-1.0 rtspsrc location=rtsp://192.168.123.50:8554/test latency=0 ! rtph264depay ! h264parse ! avdec_h264 ! videoconvert ! autovideosink sync=false

It has 2second of latency. By any chance, do you know about how to send stream with gst-rtsp-server without latency like udp?

15

Hi,
The issue seems to be in synchronization mechanism in gstreamer. In gst-launch-1.0, we can set sync=0 to disable it, but I am not sure how to disable it in test-launch. This would need other users to share experience.

16

Hi,
I just found that thre IDR frame of nvv4l2h264enc was the problem.
When I use default IDR frame frquency there wasn’t increasment of latency.
Thank you for all your help!
By any chance, do you have any recommandation for the key frame configuration?

17

Hi,
The default IDR interval is 128 and this is a bit larger for streaming. It may be set to 60, 30 , or 15. So that when certain frames are corrupted it does not last for seconds.

And by default IDR and I frames are enabled. You may also try IDR only.

18

Hi,
Thank you for letting me know!
I’ll try it and find out optimized IDR invtervals for my case.

Thank you :)