Improve performance Gstreamer pipeline for webrtc in Jetson AGX

[vcb1]

I have one applications in c++ to get the video using gstreamer from a camera and then send the video via UDP to another application in c++ that gets the video and makes the restreaming using webrct. Everything under a jetson AGX.

If i get the data from the camera in H264 and send it dirrectlly the videos works perfect in 4k:

First pipeline to get the video

pipe_source = "rtspsrc location=rtsp://192.168.1.162/z3-1.mp4 ! application/x-rtp,encoding-name=H264,profile=baseline ! ";
pipe_sink = "udpsink host=224.1.1.1 port=5000 sync=false auto-multicast=true";
launch_pipeline = pipe_source + pipe_sink;

Second pipeline to get the video and send it via webrtc

pipeline = "udpsrc multicast-group=224.1.1.1 auto-multicast=true port=5000 ! application/x-rtp,encoding-name=H264,profile=baseline,media=video,clock-rate=90000,payload=96 ! webrtcbin async-handling=true name=sendrecv";

However I can not do it in 4K if i want to make some precessing in the input video as i need to decode (and then encode) the frames prior sending the video by udp

pipe_source = "rtspsrc location=rtsp://192.168.1.162/z3-1.mp4 ! application/x-rtp,encoding-name=H265 !";
pipe_decode = "rtph265depay ! video/x-h265 ! nvv4l2decoder enable-max-performance=true ! ";
pipe_process = "nvvidconv output-buffers=5 name=myconv ! video/x-raw(memory:NVMM), format=RGBA ! nvvidconv output-buffers=5 ! video/x-raw(memory:NVMM), format=NV12 ! queue max-size-bytes=0 max-size-time=500 !";
pipe_encode ="nvv4l2vp9enc maxperf-enable=true ! video/x-vp9 ! rtpvp9pay !";
pipe_sink = "udpsink host=224.1.1.1 port=5000 sync=false auto-multicast=true";
launch_pipeline = pipe_source + pipe_decode + pipe_process + pipe_encode + pipe_sink;

In this pipeline for the source i have tried both h264/h265. Morevoere, for the encode I have tried using h264 instead of VP9, but it looks like H264 is much more slower. This is why i have used VP9 in the encoding part.

In this case the second pipeline is:

pipeline = "udpsrc multicast-group=224.1.1.1 auto-multicast=true port=5000 ! application/x-rtp,media=video,clock-rate=90000,encoding-name=VP9,payload=96, framerate=25/1 ! queue max-size-bytes=0 max-size-time=0 ! webrtcbin async-handling=true name=sendrecv";

My problem is that with this configuration i can not get video in 4k with good quality. I get the video but in a poor quality, i assume that the VP9 is changing the bitrate to have a continous video without losing frames. I have tried by giving the bit rate in the encoding part, this makes an improvement of the image quaity but i lose some frames.

If i use 1080 then i get the video in a good quality, therefore i have the feeling that is a matter of the processing capability of the hardware (i am using a jetson AGX) on doing the decoding/encoding.

Someone knows a way to improve the performance of the pipeline? I am not sure if i am doing something “useless” in the pipeline that is making the whole process slow for a 4k video.

[DaneLLL]

Hi,
We have a webrtc package but it is not able to be applied to gstreamer. This would need other users to check and share suggestion.

As a comparison you can try UDP streaming like this:
Gstreamer TCPserversink 2-3 seconds latency - #5 by DaneLLL

To check if the performance issue is specific to using webrtc.

[vcb1]

Hi,

We have a webrtc package but it is not able to be applied to gstreamer

What do you mean? . The pipeline works using gstreamer and webrtcbin (to use the webrct). May be i was wrong in my question saying webrtc instead of webrctbin.

The problem is a matter of performance of the pipeline (not working in 4k).

Thanks for the reference i will take a look.

Regarding to this topic, I have checked that in the offical docuemntation (NVIDIA Jetson Xavier Series) says that the Jetson AGX is able to encode/decode “(2x) 4Kp60 | HEVC/(2x) 4Kp60 | 12-Bit Support”.

However if i try a pipeline that makes the encoding/decoding (without any processing or UDP sink) i still do not get the image in 4k with good quality.

The pipeline is:

pipeline = "rtspsrc location=rtsp://192.168.1.162/z3-1.mp4 ! application/x-rtp,encoding-name=H265 !rtph265depay ! video/x-h265 ! nvv4l2decoder enable-max-performance=true ! nvv4l2vp9enc maxperf-enable=true ! video/x-vp9 ! rtpvp9pay ! application/x-rtp,media=video,clock-rate=90000,encoding-name=VP9,payload=96, framerate=25/1 ! queue max-size-bytes=0 max-size-time=0 ! webrtcbin async-handling=true name=sendrecv";

@DaneLLL is this a matter of the pipeline of gstreamer? What does it mean then the Encoder/Decoder in 4K of the documentation ?

Many thanks!!

Victor

[rsc44]

@vcb1 Per optimization of the encoding step, refer to this post.

How to achieve 4K@30fps recording in NVIDIA Jetson platforms

We get this requirement from all our customers who use 4K cameras like e-CAM130A_CUXVR for their surveillance applications. While our cameras can stream 4K@30fps in the MIPI interface and NVIDIA Jetson platforms also support hardware encoding for the same, there are certain limitations in the existing application software. Especially in cases when Gstreamer handles the frames for H.264 encoding . In this blog, you get to see how we helped one of our customers to overcome these limitations and a…

---

However, there are alot of other factors to consider when streaming with webrtc.

On the fly transcoding from h265 to vp9 is very compute intensive, especially at 4k.

All browsers support h264, so if possible, you should set your camera to output in h264, and just use that.

If you must stream at 4k, use TCP instead of UDP.

Though in my humble opinion, there are very few scenarios in which you should stream via webrtc at 4k. Consider the outside factors:

Most home/business networks won’t have the bandwidth to handle the 4k feed. So once you leave the local network, and begin remotely streaming from the server to an outside client then it’s almost certain that you’ll experience choppy and laggy feeds.

Consider the client screen size, and the size of the video on the screen. As most monitors are not 4k, even if you have a 4k monitor, then depending on your UI design the video that’s being displayed will most likely be alot smaller than 4k, this will cause the browser to drop a bunch of pixels and downsize the video stream.

If your camera offers a sub stream, then use that as your webrtc video feed, and use the 4k stream for your video processing needs.

If you just need to remotely stream a video file, and not a live stream, then don’t use WebRTC.

[DaneLLL]

Hi,
Thanks for your suggestion. It is practical if users would like to set up streaming on Jetson platforms. We focus more on RTSP/UDP protocols and probably the performance may not be good in using webrtc. Will check with teams and see if we can make enhancement on this in future release(s).

[vcb1]

Thanks for sharing the article. Do you know if there is a way to get Gstreamer app source and the sample application source mentioned in the article?

Thanks for all the comments they are really helpfull to understand the whole thing. After some more tests it looks like the transcoding from h265 to vp9 plus the use of webrtc in real time is not possible in 4k (or at least i was not able to do it with that piplines)

If you just need to remotely stream a video file, and not a live stream, then don’t use WebRTC.

The application requieres live stream, so this is why i’ve tried WebRTC. @rsc44 do you know a better/different way to send the live stream? I am not expert on that but after some research looks like WebRTC is the most common way to work with live stream to be used in a browser.