Pixelated and weird color-drift video when sending h265encoding via nvcompositor to rtspserver

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
1

I am trying to display 2 videos as a quad using nvcompositor and then send that composite image to an rtsp-server. My videos are pulled from a udpsrc (30 fps, bitrate=1000000).

When I send a single video through the rtspserver, it plays beautifully. But the composite image only looks good for 2 seconds before pixelation and blockiness and weird color gradients start to accumulate in the stream. I inherited these pipelines, so I’m not entirely sure the rhyme or reason for how the composite pipeline is written. Most tutorials I have found for the compositor has had very simple examples. Nothing like the acrobatic pipeline I have inherited :-p

I have 2 questions.

  1. Why is the pipeline for a single video so simple: src–> caps → queue → depay -->pay
    But the pipeline for the composite so complicated with decodebins, nvvidconvs, and so many caps?
  2. Any ideas on how to improve the video quality in the composite?

Here is the pipeline I use for the single video that is streamed beautifully.

“udpsrc buffer-size=1310702 port=10010 ! application/x-rtp,media=video,encoding-name=H265,payload=96 ! queue ! rtph265depay ! rtph265pay name=pay0 pt=96”

Here is the pipeline I use for the composite that is goofing up. I have formatted it to try to make it easier to read.

char fstr0[1500] = “nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=960 sink_0::height=540”
" sink_1::xpos=960 sink_1::ypos=0 sink_1::width=960 sink_1::height=540"
" ! nvvidconv ! video/x-raw(memory:NVMM)"
" ! nvv4l2h265enc iframeinterval=100 vbv-size=66666 ! video/x-h265,stream-format=byte-stream"
" ! h265parse ! rtph265pay name=pay0 pt=96 "
" udpsrc buffer-size=1310702 port=10100 ! application/x-rtp,media=video,encoding-name=H265,payload=96 ! rtph265depay ! queue ! decodebin ! nvvidconv ! video/x-raw(memory:NVMM),width=960,height=540,format=(string)I420 ! comp. "
“udpsrc buffer-size=1310702 port=10102 ! application/x-rtp,media=video,encoding-name=H265,payload=96 ! rtph265depay ! queue ! decodebin ! nvvidconv ! video/x-raw(memory:NVMM),width=960,height=540,format=(string)I420 ! comp.”;

I would also like to add that if I were to change the encoding on the udpsrc to be H264, and replace all the H265s in the above pipeline to H264, the streaming quality instantly improves.
I’m attaching a screenshot so you can get an idea of what I mean by pixelation and weird colors. I’m holding a rainbow stuffy in the screenshot.

pixelated
pixelated1248×690 39.8 KB

3

Hi,
It seems like iframeinterval=100 may triggers the issue. For real-time streaming the value looks to be too large. Would suggest try small value such as 15, 30, 60.

For further investigation please break down the pipeline and also try gst-launch-1.0 command. Such as use nvoverlaysink to display directly:

... ! nvcompositor ! nvoverlaysink

Or save to a video file and check if it is good:

... ! nvcompositor ! nvvidconv ! nvv4l2h265enc ! h265parse ! matroskamux ! filesink location=test.mkv
4

Hi Dale,

Thanks for the suggestions. I tried removing the iframeinterval=100, but it did not help.

I also tried a smaller pipeline launched using gst_parse_launch and streamed to the nveglglessink to see if the issue remained. The issue was still there.

Here is the smaller pipeline without using the composite, and the stream looks fine:

sprintf (pipe_string, “udpsrc port=10106 ! application/x-rtp, media=video, clock-rate=90000, encoding-name=H265,playload=96 ! rtph265depay ! h265parse ! video/x-h265,alignment=au ! nvv4l2decoder disable-dpb=true enable-max-performance=1 ! queue max-size-bytes=0 ! identity name=elm_id ! queue ! nvvidconv ! nveglglessink”);

Here is the smaller pipeline using the composite element (1x1 composite). The pixelation and color drift issue remains.

sprintf (pipe_string, “nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=960 sink_0::height=540 ! nvvidconv ! video/x-raw(memory:NVMM) ! nvv4l2h265enc ! video/x-h265,stream-format=byte-stream ! h265parse ! video/x-h265,alignment=au ! nvv4l2decoder disable-dpb=true enable-max-performance=1 ! queue ! nvvidconv ! nveglglessink udpsrc buffer-size=1310702 port=10106 ! application/x-rtp,media=video,encoding-name=H265,payload=96 ! rtph265depay ! queue ! decodebin ! nvvidconv ! comp.”);

Any other ideas on how I can debug this issue?

Thanks!
Mercy

5

Hi,
Do you see the issue if you record to test.mkv? We have seen issues while rtph265pay and rtph265depay are present in the command:
Does nvv4l2decoder fully support 4K resolution?

This probably is a similar topic. Please save to a local file and check if the stream is good.

6

Hi Dane,
I have tried saving to a local file as you suggested. And the pixilation is still present in the recorded stream. Do you have any suggestions on how to work around NOT using rtph265pay and rtph265depay? Are there other plugins/elements I could use?

-Mercy

7

Hi,
Please try this command:

$ gst-launch-1.0 nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=960 sink_0::height=540 ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! nvoverlaysink udpsrc port=10106 ! application/x-rtp,media=video,encoding-name=H265,payload=96 ! rtph265depay ! queue ! decodebin ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA' ! comp.

We don’t see issue in running the command. Would like to know if you can also run it without issue.

8

H Dane,

I really appreciate all your help so far. Thanks for the much more concise pipeline to try. Unfortunately, the video stream still shows pixilation and weird colors. The stream starts off looking fine, but within a few seconds it begins to deteriorate.

I also tried replacing the nvcompositor element with the generic compositor element. That did not make things better.

I also tried feeding a videotestsrc to port 10106 to see if that would make the composite feed look better. It did look better!

This made me suspect my video source, so I tried this pipeline to check that the video source is fine. This pipeline produced a fine stream.

gst-launch-1.0 udpsrc port=10106 ! application/x-rtp,media=video,encoding-name=H265,payload=96 ! rtph265depay ! h265parse ! nvv4l2decoder ! nvoverlaysink

I’m truly stumped as to what the problem could be. Do you have any other suggestions I could try?
I’m running from a Xavier board:

apps@apps-desktop:~/staleVideo$ cat /etc/nv_tegra_release
R32 (release), REVISION: 4.3, GCID: 21589087, BOARD: t186ref, EABI: aarch64, DATE: Fri Jun 26 04:34:27 UTC 2020