Use gstreamer to decode and play 18 videos

Autonomous Machines Jetson & Embedded Systems Jetson Nano
1

In the “Technical Specifications” section of https://developer.nvidia.com/embedded/jetson-nano-developer-kit
Jetson Nano Developer Kit is able to decode 18x 720p @ 30 (H.264/H.265).
This is perfect for me, because my application needs to play many videos at the same time. For example, 16 videos in 4x4 grid on one monitor, or 18 videos in 3x3 grid on two monitors.
I’m trying to implement it on my Jetson Nano Developer Kit, but I haven’t found a good solution yet.

One of the strategies is to start a pipeline for each video, but I have to choose a sink first.

I like nvoverlaysink because “overlay-x/y/w/h” and “display-id” make it very convenient to play multiple videos on dual monitors, but there is hardware limitation and it can’t play enough videos for me.

I tried nvdrmvideosink. Its properties “offset-x/y”, “plane-id”, and “conn-id” are useful. But there are errors when I play 5 or 6 videos.

nveglglessink seems better. The properties “window-x/y/width/height” are nice. But I didn’t find any way to play video on the second monitor. I tried something like “DISPLAY=:1” but it didn’t work. Another problem is it requires either X11 or wayland backend, which I’m trying to avoid, but I can go with it if there is no better option.

Another strategy is to use one pipeline for all videos. I’m still not sure how the whole pipeline will be affected by individual videos. For example, some videos might become unavailable while playing. Or, the user may change the source of some videos at any time. I’m hoping the whole pipeline doesn’t need to be restarted in those cases. I tried glvideomixer and compositor, but the performance is unacceptable. Hopefully nvstreammux should have better performance, I might give it a try.

Any advice will be appreciated.

2

Hi,
If you need each video playback to be independent, you only can use nveglglessink. Please execute ‘sudo jetson_clocks’ and run 18 pipelines like:

$ gst-launch-1.0 uridecodebin uri=file:///home/nvidia/test.mp4 ! nvvidconv ! nveglglessink &
3

I use nvcompositor

gst-launch-1.0 -e nvcompositor name=comp \
    sink_0::alpha=1 sink_0::xpos=0 sink_0::ypos=0 \
    sink_1::alpha=1 sink_1::xpos=0 sink_1::ypos=1080 \
    sink_2::alpha=1 sink_2::xpos=1920 sink_2::ypos=0 \
    sink_3::alpha=1 sink_3::xpos=1920 sink_3::ypos=1080 \
    ! nvoverlaysink display-id=1 \
    filesrc location=1.mkv ! matroskademux  name=demux0 demux0.video_0! h264parse ! omxh264dec ! nvvidconv ! comp.sink_0 \
    filesrc location=2.mkv ! matroskademux  name=demux1 demux1.video_0! h264parse ! omxh264dec ! nvvidconv ! comp.sink_1 \
    filesrc location=3.mkv ! matroskademux  name=demux2 demux2.video_0! h264parse ! omxh264dec ! nvvidconv ! comp.sink_2 \
    filesrc location=4.mkv ! matroskademux  name=demux3 demux3.video_0! h264parse ! omxh264dec ! nvvidconv ! comp.sink_3
4

Thank you, Dane.
I tried nveglglessink, but I didn’t find any way to play the video on the DP monitor. It always plays on the HDMI monitor. Do you have any suggestion?

For example, when I use nvoverlaysink, I can use “display-id=1” to play on the DP monitor, “display-id=0” to play on the HDMI monitor. I can play two 4K 30fps videos on two monitors at the same time.

This “display” property looks close, but I haven’t figured out what value should be passed to it.

$ gst-inspect-1.0 nveglglessink
......
  display             : If set, the sink will use the passed X Display for rendering
                        flags: readable, writable
                        Pointer.
5

Thank you, jocover.

I didn’t know there is such a plugin. I’ll give it a try. Hopefully it’s better than glvideomixer and compositor. When I use them with rtspsrc, the video has a 3-second delay.

6

The only current issue with nvcompositor is that it seems to be limited to 8 streams. I’ve attempted 3x3 and 4x4 for 9 and 16 streams, but nvcompositor only renders 8. I have another post inquiring about this, but has anyone else experienced the same issue?

https://devtalk.nvidia.com/default/topic/1071764/jetson-nano/nvcompositor-appears-to-only-render-lowest-8-zorder-streams/

7

Hi,

It is a limitation. Please check this post