Problem decoding H.265 video with nvv4l2decoder

[pedro9]

Hello,

We are building a gateway to convert between several media formats. The gateway is based on a Jetson AGX Xavier and the software builds different GStreamer pipelines, depending on the media types it needs to process.

We were recently asked to support RTP streams generated by a Xilinx-based board and we could not decode them using nvv4l2decoder. We recorded the H.265 payload of those streams as Matroska files and decoding them using gst-launch-1.0 yields these errors:

> gst-launch-1.0 filesrc location=10-bit-2.mkv ! matroskademux ! h265parse ! nvv4l2decoder ! nvvidconv ! nv3dsink
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE
Opening in BLOCKING MODE 
Pipeline is PREROLLING ...
NvMMLiteOpen : Block : BlockType = 279 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 279 

(gst-launch-1.0:9110): GStreamer-CRITICAL **: 12:02:13.441: gst_caps_remove_structure: assertion 'idx < gst_caps_get_size (caps)' failed

(gst-launch-1.0:9110): GStreamer-CRITICAL **: 12:02:13.441: gst_caps_remove_structure: assertion 'idx < gst_caps_get_size (caps)' failed

(gst-launch-1.0:9110): GStreamer-CRITICAL **: 12:02:13.442: gst_caps_remove_structure: assertion 'idx < gst_caps_get_size (caps)' failed
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:00.000689824
Setting pipeline to NULL ...
Freeing pipeline ...

Replacing the nv3dsink with a filesink gets rid of the errors but generates an empty file:

> gst-launch-1.0 filesrc location=10-bit-2.mkv ! matroskademux ! h265parse ! nvv4l2decoder ! nvvidconv ! filesink location=tmp.raw
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE
Opening in BLOCKING MODE 
Pipeline is PREROLLING ...
NvMMLiteOpen : Block : BlockType = 279 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 279 
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:00.000488992
Setting pipeline to NULL ...
Freeing pipeline ...


> ll tmp.raw 
-rw-rw-r-- 1 bisect bisect 0 Dec  2 12:09 tmp.raw

However, avdec_h265 decodes the files correctly with a similar pipeline:

gst-launch-1.0 filesrc location=10-bit-2.mkv ! matroskademux ! h265parse ! avdec_h265 ! videoconvert ! ximagesink

This happens with 4:2:0 content, both 8 and 10 bit. Here is the output of mediainfo for the 10-bit file:

> mediainfo 10-bit-2.mkv 
General
Unique ID                                : 215386674912513167907874724445387356410 (0xA209F6F581247B1005F50F408E55F8FA)
Complete name                            : 10-bit-2.mkv
Format                                   : Matroska
Format version                           : Version 2
File size                                : 190 KiB
Encoded date                             : UTC 2021-09-27 09:51:34
Writing application                      : GStreamer Matroska muxer
Writing library                          : GStreamer matroskamux version 1.14.5
IsTruncated                              : Yes

Video
ID                                       : 1
Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Main 10@L4.1@Main
Codec ID                                 : V_MPEGH/ISO/HEVC
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 3.175 FPS
Original frame rate                      : 50.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0 (Type 0)
Bit depth                                : 10 bits
Title                                    : Video
Language                                 : English
Default                                  : Yes
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709

I appreciate any suggestion on how to overcome this problem.

Thanks in advance,

Pedro

[pedro9]

We seem to have found a solution for this, changing the pipeline this way:

... ! h265parse config-interval=-1 ! 'video/x-h265, stream-format=(string)byte-stream' ! nvv4l2decoder ! ...

This looks odd because the caps of the parser src pad are the same with and without the above.

[DaneLLL]

Hi,
It sounds like the stream does not have VPS/SPS/PPS information at the very beginning. Please try to put VPS/SPS/PPS for every IDR frame. If you use nvv4l2h265enc, you can enable this property:

  insert-sps-pps      : Insert H.265 SPS, PPS at every IDR frame
                        flags: readable, writable
                        Boolean. Default: false

[pedro9]

Hi,

Thanks for the feedback.

Indeed, the VPS/SPS/PPS is not at the beginning. However, there is not much we can do about that. We are receiving an RTP stream that is generated by some other device and, since we can start receiving the stream at any moment in time, we can never be sure of what will be the kind of the first frame that we receive.

And I don’t really understand why forcing the caps solves this.

[DaneLLL]

Hi,
Probably setting this property to h265parse helps:

  config-interval     : Send VPS, SPS and PPS Insertion Interval in seconds (spr
op parameter sets will be multiplexed in the data stream when detected.) (0 = di
sabled, -1 = send with every IDR frame)
                        flags: readable, writable
                        Integer. Range: -1 - 3600 Default: 0

Setting config-interval=1, VPS/SPS/PPS is inserted every second. This may help decoder for initializing decoding tasks such as deciding resolution and creating buffers.

[pedro9]

Hello,

Thanks for your suggestion.

I am now able to decode the MP4 files. The next step is to simulate if we would be able to decode the same H.265 if it was sent over RTP. To simulate that, I started with a MPEG TS file that contains an H.265 stream generated by the said encoder.
I demux it and send it over RTP with the following pipeline:

gst-launch-1.0 filesrc location=sample.ts ! tsdemux ! h265parse config-interval=1 ! rtph265pay config-interval=1 mtu=1400 ! udpsink auto-multicast=true host=239.100.1.1 port=5000 sync=true

Then, I launch the receiver:

gst-launch-1.0 -vv udpsrc multicast-group=239.100.1.1 auto-multicast=true port=5000 caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H265, payload=(int)96" ! rtph265depay ! h265parse config-interval=1 ! queue ! nvv4l2decoder ! nvvidconv ! nv3dsink

This works but if and only if I start the receiver before I start the sender. If I do it the other way around, it starts the pipeline, a short period afterwards logs new caps (probably, when it receives the VPS/SPS/PPS sent every second) and opens a window. The problem is that the window is totally transparent and shows no image at all.

Any suggestion?

Many thanks in advance,

Pedro

[DaneLLL]

Hi,
The decoding begins from IDR frame and probably sample.ts has large idr interval so decoder cannot start decoding if first IDR frame is missed. You may try to re-encode with idrinterval=15 like:

$ gst-launch-1.0 filesrc location=sample.ts ! tsdemux ! h265parse config-interval=1 ! nvv4l2decodr ! nvv4l2h265enc idrinterval=15 insert-sps-pps=1 !  rtph265pay config-interval=1 mtu=1400 ! udpsink auto-multicast=true host=239.100.1.1 port=5000 sync=true

May also try videotestsrc for comparison:
Gstreamer TCPserversink 2-3 seconds latency - #5 by DaneLLL