Hosting Multiple Remote Streams Using Compositor Gstreamer

Robotics & Edge Computing Jetson & Embedded Systems Jetson Xavier NX
, , ,
1

Having issues hosting multiple remote streams. We have Jetsons as Host and Remote both on local network with bandwidth limited to 50 Mb/s where low latency is important working to host four streams. When hosting a singular stream with the compositor,
Host

gst-launch-1.0 udpsrc port=9000 caps="application/x-rtp, media=video, encoding-name=H265, clock-rate=90000, payload=96" ! rtph265depay ! h265parse ! nvv4l2decoder ! nvvidconv ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080' ! queue ! comp. \
nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 ! nv3dsink

Remote

gst-launch-1.0 v4l2src device=/dev/video0 ! image/jpeg, format=MJPEG,width=1920,height=1080,framerate=15/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! 'video/x-raw(memory:NVMM), width=960, height=540, format=NV12' ! queue max-size-buffers=200000 leaky=2 ! nvv4l2h265enc bitrate=200000 slice-header-spacing=8 bit-packetization=0 name=video_enc ! rtph265pay mtu=1400 ! udpsink host=<Hostname> port=9000

Works fine with little latency issues. However, even when hosting two streams utilizing the compositor,
Host

gst-launch-1.0 udpsrc port=9000 caps="application/x-rtp, media=video, encoding-name=H265, clock-rate=90000, payload=96" ! rtph265depay ! h265parse ! nvv4l2decoder ! queue leaky=2 ! nvvidconv ! video/x-raw, format=RGBA ! nvvidconv ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080' ! queue ! comp. udpsrc port=9001 caps="application/x-rtp, media=video, encoding-name=H265, clock-rate=90000, payload=96" ! rtph265depay ! h265parse ! nvv4l2decoder ! queue leaky=2 ! nvvidconv ! video/x-raw, format=RGBA ! nvvidconv ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080' ! queue ! comp. nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=960 sink_0::height=540 sink_1::xpos=960 sink_1::ypos=0 sink_1::width=960 sink_1::height=540 sink_2::xpos=0 sink_2::ypos=540 sink_2::width=960 sink_2::height=540 ! nvvidconv ! nvegltransform ! nveglglessink

Remote_1

gst-launch-1.0 v4l2src device=/dev/video0 ! image/jpeg, format=MJPEG,width=1920,height=1080,framerate=15/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! 'video/x-raw(memory:NVMM), width=960, height=540, format=NV12' ! queue max-size-buffers=200000 leaky=2 ! nvv4l2h265enc bitrate=200000 slice-header-spacing=8 bit-packetization=0 name=video_enc ! rtph265pay mtu=1400 ! udpsink host=<Hostname> port=9000

Remote_2

gst-launch-1.0 v4l2src device=/dev/video0 ! image/jpeg, format=MJPEG,width=1920,height=1080,framerate=15/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! 'video/x-raw(memory:NVMM), width=960, height=540, format=NV12' ! queue max-size-buffers=200000 leaky=2 ! nvv4l2h265enc bitrate=200000 slice-header-spacing=8 bit-packetization=0 name=video_enc ! rtph265pay mtu=1400 ! udpsink host=<Hostname> port=9001

When running the debugger, it says the pipelines are successfully set to playing, but there is no video stream displayed. Other times when running it gets caught on SIGSEGV. The goal is to use the compositor to have four remote video streams playing in the same window. Any help is greatly appreciated.

2

Hello, can you please tell me what version of Jetson you are using? I can then move this topic to the proper category.

3

Thank you for your response, we are currently on Jetson 5.1.2. We would not be opposed to upgrading to a newer version of 5.X.X to stay on Ubuntu 20.04 for ROS compatibility.

4

Sorry, that does not help me.

There are different Jetson’s.

image
image442×723 14.2 KB

5

We have some Xavier NXs, but are transitioning to OrinNXs with 16Gb memory.

7

Thanks for the reply. I moved this topic to the Jetson Xavier NX category.

9

Hi,
In the pipeline,

gst-launch-1.0 udpsrc port=9000 caps="application/x-rtp, media=video, encoding-name=H265, clock-rate=90000, payload=96" ! rtph265depay ! h265parse ! nvv4l2decoder ! queue leaky=2 ! nvvidconv ! video/x-raw, format=RGBA ! nvvidconv ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080' ! queue ! comp. udpsrc port=9001 caps="application/x-rtp, media=video, encoding-name=H265, clock-rate=90000, payload=96" ! rtph265depay ! h265parse ! nvv4l2decoder ! queue leaky=2 ! nvvidconv ! video/x-raw, format=RGBA ! nvvidconv ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080' ! queue ! comp. nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=960 sink_0::height=540 sink_1::xpos=960 sink_1::ypos=0 sink_1::width=960 sink_1::height=540 sink_2::xpos=0 sink_2::ypos=540 sink_2::width=960 sink_2::height=540 ! nvvidconv ! nvegltransform ! nveglglessink

Frame data is copied from NVMM buffer to CPU buffer and then copied back to NVMM buffer. This looks redundant. Please have the frame data in NVMM buffer.

And you may construct the pipeline with filesrc ! qtdemux ! h265parse to ensure it works first. And then replace with udpsrc ! rtph265depay ! h265parse.

10

Thank you for the reply,
I believe I removed the redundancies with the below pipeline, would you mind confirming?

gst-launch-1.0 udpsrc port=9000 caps="application/x-rtp, media=video, encoding-name=H265, clock-rate=90000, payload=96" ! rtph265depay ! h265parse ! nvv4l2decoder ! nvvidconv ! 'video/x-raw(memory:NVMM), format=RGBA, width=(int)1920, height=(int)1080' ! queue ! comp. udpsrc port=9001 caps="application/x-rtp, media=video, encoding-name=H265, clock-rate=90000, payload=96" ! rtph265depay ! h265parse ! nvv4l2decoder ! nvvidconv ! 'video/x-raw(memory:NVMM), format=RGBA, width=(int)1920, height=(int)1080' ! queue ! comp. nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=960 sink_0::height=540 sink_1::xpos=960 sink_1::ypos=0 sink_1::width=960 sink_1::height=540 ! nvvidconv ! nvegltransform ! nveglglessink

Do you also have any recommendations on using nveglglessink and x11 or should this be avoided?

11

Hi,
On the other Jetson device, we set up UDP sources:

$ gst-launch-1.0 videotestsrc is-live=1 ! video/x-raw,width=1920,height=1080 ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! nvv4l2h265enc insert-sps-pps=1 idrinterval=15 ! h265parse ! rtph265pay ! udpsink host=10.19.115.105 port=9000 sync=0
$ gst-launch-1.0 videotestsrc is-live=1 pattern=18 ! video/x-raw,width=1920,height=1080 ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! nvv4l2h265enc insert-sps-pps=1 idrinterval=15 ! h265parse ! rtph265pay ! udpsink host=10.19.115.105 port=9001 sync=0

And the command works well:

$ export DISPLAY=:0
$ gst-launch-1.0 udpsrc port=9000 caps="application/x-rtp, media=video, encoding-name=H265, clock-rate=90000, payload=96" ! rtph265depay ! h265parse ! nvv4l2decoder ! nvvidconv ! 'video/x-raw(memory:NVMM), format=RGBA, width=(int)1920, height=(int)1080' ! queue ! comp. udpsrc port=9001 caps="application/x-rtp, media=video, encoding-name=H265, clock-rate=90000, payload=96" ! rtph265depay ! h265parse ! nvv4l2decoder ! nvvidconv ! 'video/x-raw(memory:NVMM), format=RGBA, width=(int)1920, height=(int)1080' ! queue ! comp. nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=960 sink_0::height=540 sink_1::xpos=960 sink_1::ypos=0 sink_1::width=960 sink_1::height=540 ! nvvidconv ! nvegltransform ! nveglglessink sync=0

Please give it a try.

12

Thanks again for the help.
I am not able to get the test sources to work, however, I was able to get live feed from a more updated (Jetpack v1.5.3) Orin as host, with two remote streams on one composited screen. This was what I wanted, but when I tried to replicate the set up using the same devices and descriptions I was unable to. It simply says “Failed to composite frames”.

The host description was:
gst-launch-1.0 udpsrc port=9000 caps="application/x-rtp, media=video, encoding-name=H265, clock-rate=90000, payload=96" ! rtph265depay ! h265parse ! nvv4l2decoder ! nvvidconv ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080' ! queue ! comp. udpsrc port=9001 caps="application/x-rtp, media=video, encoding-name=H265, clock-rate=90000, payload=96" ! rtph265depay ! h265parse ! nvv4l2decoder ! nvvidconv ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080' ! queue ! comp. nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=960 sink_0::height=540 sink_1::xpos=960 sink_1::ypos=0 sink_1::width=960 sink_1::height=540 ! nvvidconv ! nvegltransform ! nveglglessink

the remote devices description are identical to the ones in my first query.

13

Hi,
You may try

  1. Adjust bitrate of the sources
  2. Set sync=0 to udpsink

If videotestsrc works, it should be working the same for v4l2src.

15

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.