Nvv4l2h265enc iframe distortion during streaming on AGX Orin

[waep]

Hi
I am trying to encode a full HD UYVY video stream from an nvv4l2camerasrc with the nvv4l2h265enc on a AGX Orin and then stream it to another AGX Orin via an udp sink. There the stream is decoded again using nvv4l2decoder and output in an nvdrmvideosink. This basically works well and I get the desired 60 fps with very low network utilization. Nevertheless, I have a distorted image on my display about every 5 seconds. I have an iframe interval of 300 which would result in exactly the 5 seconds (60 fps). If I make the iframe interval of the encoder smaller, the event becomes more frequent.

This is the Gstreamer pipeline that I use:
Sender:

gst-launch-1.0 -v nvv4l2camerasrc device=/dev/video0 ! 'video/x-raw(memory:NVMM), format=(string)UYVY,framerate=(fraction)60/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)NV12, framerate=(fraction)60/1' ! queue max-size-buffers=3 leaky=downstream  ! nvv4l2h265enc maxperf-enable=1 bitrate=500000000 iframeinterval=300 vbv-size=33333 insert-sps-pps=true control-rate=constant_bitrate profile=Main num-B-Frames=0 ratecontrol-enable=true preset-level=UltraFastPreset EnableTwopassCBR=false ! queue ! h265parse ! rtph265pay ! udpsink host=10.42.0.189 port=5005

Receiver:

gst-launch-1.0 -v udpsrc port=5005 ! 'application/x-rtp, payload=(int)96' ! rtph265depay ! h265parse ! queue ! nvv4l2decoder enable-max-performance=1 ! queue ! nvdrmvideosink plane_id=0

I have also tested the whole thing on the AGX Orin where I have connected the v4l2 source and output the stream directly to an autovideosink. Then there are no problems, so I assume it can’t be the source:

gst-launch-1.0 -v nvv4l2camerasrc device=/dev/video0 ! 'video/x-raw(memory:NVMM), format=(string)UYVY,framerate=(fraction)60/1' ! nvvidconv !  autovideosink

The problem also exists if I omit the nvv4l2decoder in the pipeline and save the stream directly in an mkv file (on the AGX Orin to which the v4l2 source is connected):

gst-launch-1.0 -v nvv4l2camerasrc device=/dev/video0 ! 'video/x-raw(memory:NVMM), format=(string)UYVY,framerate=(fraction)60/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)NV12, framerate=(fraction)60/1' ! queue max-size-buffers=3 leaky=downstream  ! nvv4l2h265enc maxperf-enable=1 bitrate=500000000 iframeinterval=300 vbv-size=33333 insert-sps-pps=true control-rate=constant_bitrate profile=Main num-B-Frames=0 ratecontrol-enable=true preset-level=UltraFastPreset EnableTwopassCBR=false ! queue ! h265parse ! queue ! matroskamux name=mux ! filesink location=h265Streamer_iframe_problem.mkv

The disturbance of the picture only occurs in the lower two thirds of the picture. It is also noticeable that the interference is stronger with a complex input stream. If you have an input stream in which a large area of the image displays the same color, for example, the interference is less severe.

I therefore assume that it must be due to the nvv4l2h265enc and its settings. However, I can’t find out what the problem might be. Help would be very welcome. Also general suggestions to improve my pipeline would be greatly appreciated.

I have attached the saved video streams in this post. One stream was recorded with an iframeinterval of 300 and the other with an iframeinterval of 60.

There you can clearly see what the problem is.

h265Stream_problem_iframe.zip (25.5 MB)

[DaneLLL]

Hi,
Setting bitrate to 500Mbps seems to be a very large value for 1080p60. Please try like 10Mbps, 20Mbps, 40Mbps. And vbv-size looks small. Please refer to example in
Random blockiness in the picture RTSP server-client -Jetson TX2 - #5 by DaneLLL

And for further debugging, you can save the steam to a file:

gst-launch-1.0 -v nvv4l2camerasrc device=/dev/video0 ! 'video/x-raw(memory:NVMM), format=(string)UYVY,framerate=(fraction)60/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)NV12, framerate=(fraction)60/1' ! queue max-size-buffers=3 leaky=downstream  ! nvv4l2h265enc maxperf-enable=1 bitrate=500000000 iframeinterval=300 vbv-size=33333 insert-sps-pps=true control-rate=constant_bitrate profile=Main num-B-Frames=0 ratecontrol-enable=true preset-level=UltraFastPreset EnableTwopassCBR=false ! queue ! h265parse ! video/x-h265 ! filesink location=test.h265

And decode it by 00_video_decode. To ensure video quality is good without effect of network.

[waep]

Hi
Thank you for your help! I was able to eliminate the error by increasing the vbv-size. 2*(bitrate/fps) didn’t really work for me, I had to increase it to 10’000’000 which is about 10*(bitrate/fps).
This is my current pipeline:

Sender:

gst-launch-1.0 -v nvv4l2camerasrc device=/dev/video0 ! 'video/x-raw(memory:NVMM), format=(string)UYVY,framerate=(fraction)60/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)NV12, framerate=(fraction)60/1' ! queue max-size-buffers=10 ! nvv4l2h265enc maxperf-enable=1 bitrate=60000000 iframeinterval=60 vbv-size=10000000 insert-sps-pps=true control-rate=constant_bitrate num-B-Frames=0 preset-level=UltraFastPreset ! queue ! h265parse ! rtph265pay ! udpsink host=192.168.100.189 port=5005

Receiver:

gst-launch-1.0 -v udpsrc port=5005 ! 'application/x-rtp, payload=(int)96' ! rtph265depay ! h265parse ! queue ! nvv4l2decoder enable-max-performance=1 !  queue ! nvdrmvideosink

This works well and runs smoothly at 60fps. But now I want to upscale the full hd stream coming from the nvv4l2decoder to 1920x1440 with a nvvidconv. When I do this, the stream that the nvdrmvideosink outputs no longer runs smoothly, so it is no longer 60fps:

gst-launch-1.0 -v udpsrc port=5005 ! 'application/x-rtp, payload=(int)96' ! rtph265depay ! h265parse ! queue ! nvv4l2decoder enable-max-performance=1 ! 'video/x-raw(memory:NVMM)' ! nvvidconv interpolation-method=5 ! 'video/x-raw, width=(int)1920, height=(int)1440' ! queue ! nvdrmvideosink

But if I add an fpsdisplaysink with video-sink=fakesink at the end of the pipeline instead of nvdrmvideosink, the fpsdiplaysink indicates that I have 60fps. If I then

gst-launch-1.0 -v udpsrc port=5005 ! 'application/x-rtp, payload=(int)96' ! rtph265depay ! h265parse ! queue ! nvv4l2decoder enable-max-performance=1 ! 'video/x-raw(memory:NVMM)' ! nvvidconv interpolation-method=5 ! 'video/x-raw, width=(int)1920, height=(int)1440' ! queue ! fakesink video-sink=fakesink

If I then set sync=false for the nvdrmvideosink, it is smooth again, but then I have a latency of about 1 second between input at the sender pipeline and output via nvdrmvideosink:

gst-launch-1.0 -v udpsrc port=5005 ! 'application/x-rtp, payload=(int)96' ! rtph265depay ! h265parse ! queue ! nvv4l2decoder enable-max-performance=1 ! 'video/x-raw(memory:NVMM)' ! nvvidconv interpolation-method=5 ! 'video/x-raw, width=(int)1920, height=(int)1440' ! queue ! nvdrmvideosink sync=false

With sync=true it is only about 100ms. Do you have any idea what the problem could be? Even if I insert an rtpjitter buffer in the receiver pipeline, I have a latency of about 1 second. Do I still have a general error in the pipeline. The lowest possible latency would be desirable. Nevertheless, the frames should arrive in the correct order because they are to be saved and evaluated later as a file. Can you find any errors in my pipeline?

[DaneLLL]

Hi,
Please feed NVMM buffer to nvdrmvideosink like:

... ! nvvidconv interpolation-method=5 ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1440' ! queue ! nvdrmvideosink

This eliminates additional buffer copy. Please give it a try.

[waep]

Hi,
Sorry, I’ve already tried that. But the problem persists. I have tried it with the following pipeline:
Sender:

gst-launch-1.0 -v nvv4l2camerasrc device=/dev/video0 ! 'video/x-raw(memory:NVMM), format=(string)UYVY,framerate=(fraction)60/1' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)NV12, framerate=(fraction)60/1' ! queue max-size-buffers=10 ! nvv4l2h265enc maxperf-enable=1 bitrate=60000000 iframeinterval=60 vbv-size=10000000 insert-sps-pps=true control-rate=constant_bitrate num-B-Frames=0 preset-level=UltraFastPreset ! queue ! h265parse ! rtph265pay ! udpsink host=192.168.100.189 port=5005

Receiver:

gst-launch-1.0 -v udpsrc port=5005 ! 'application/x-rtp, payload=(int)96' ! rtph265depay ! h265parse ! queue ! nvv4l2decoder enable-max-performance=1 ! 'video/x-raw(memory:NVMM)' ! nvvidconv interpolation-method=5 ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1440' ! queue ! nvdrmvideosink

Any other ideas? Maybe somewhere else in the pipeline? What about the huge delay with the sync flag?

(By the way, up at the fourth pipeline I posted in my reply. At the very end it should be fpdsiplaysink and not fakesink)

[DaneLLL]

Hi,
You may break down the pipeline with fakesink. To clarify which plugin triggers the failure.

[waep]

Hi,
What could be the reason that the latency with nvdrmvideosink sync=false increases to about one second?

[DaneLLL]

Hi,
Please run the sender with videotestsrc+time overlay like:
Gstreamer TCPserversink 2-3 seconds latency - #5 by DaneLLL

And check if you still observe the one second delay in nvdrmvideosink. We think certain latency is from the camera source. Comparing to videotestsrc should provide some information.