Nvv4l2vp9enc doesn't flag keyframes

[armando-guereca]

When nvv4l2vp9encis used as part of a Gstreamer pipeline, the resulting stream doesn’t properly flag keyframes which derives on issues on downstream components relying on the correct flag for its operation. In contrast, omxvp9enc element correctly flags key-frames, the usage of this alternative element wouldn’t be a problem if not for it being marked as deprecated, and even scheduled to be removed on new versions, this increases the relevance to fix the behavior on nvv4l2vp9enc or to delay the removal of omxvp9enc.
An example of two downstream elements that break when the keyfames are not correctly flagged, is matroskamux which is unable to write Cues element on the resulting files, also when used with splitmuxsink, is unable to properly split the output files.

How to reproduce the issue:

• The easiest way to identify this issue is through the use of the element identity and filtering the output for non delta-unit buffers, on the following examples, the expectation is to see a printed output message for each correctly flagged keyframe.

Example of a broken pipeline:
gst-launch-1.0 -v videotestsrc is-live=false ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)NV12, framerate=(fraction)30/1' ! nvv4l2vp9enc ! identity silent=false ! matroskamux ! filesink location=nvv_test.mkv -e | grep chain | grep -v delta
Output:
<none>

Example of a working pipeline:
gst-launch-1.0 -v videotestsrc is-live=false ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)NV12, framerate=(fraction)30/1' ! omxvp9enc ! identity silent=false ! matroskamux ! filesink location=../../test-media/nvv_test.mkv -e | grep chain | grep -v delta
Output:

/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain   ******* (identity0:sink) (4621 bytes, dts: 0:00:00.000000000, pts: 0:00:00.000000000, duration: 0:00:00.033333333, offset: -1, offset_end: -1, flags: 00000040 discont , meta: none) 0x7f78183d20
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain   ******* (identity0:sink) (3586 bytes, dts: 0:00:02.000000000, pts: 0:00:02.000000000, duration: 0:00:00.033333333, offset: -1, offset_end: -1, flags: 00000000 , meta: none) 0x7f7400a360
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = chain   ******* (identity0:sink) (3557 bytes, dts: 0:00:04.000000000, pts: 0:00:04.000000000, duration: 0:00:00.033333333, offset: -1, offset_end: -1, flags: 00000000 , meta: none) 0x7f7400a360

As seen the only difference is the usage of omxvp9enc vs nvv4l2vp9enc

Related issues:
This issue was in essence recently reported on the topic: My pipeline with 'nvv4l2vp9enc ! appsink ' could not recognize key frame - #6 by zhang.dongchuan
However the “solved” response doesn’t seems to be related to the actual issue which is improper frame flags in the encoder, vp9parse is a different element, which correctly flags keyframes, but it is not available on the GStreamer version shipped with Jetson devices (1.14.5)

Platform:
# R32 (release), REVISION: 6.1, GCID: 27863751, BOARD: t186ref, EABI: aarch64, DATE: Mon Jul 26 19:36:31 UTC 2021

[DaneLLL]

Hi,
We think it is fine to link to vp9parse in the pipeline. There is similar linkage in using nvv4l2h264enc/nvv4l2h265enc. Would need to link to h264parse/h265parse.

The source code of gst-v4l2 is public and if you would like to link vp9 encoder and muxer directly without parser, you can refer to vp9parse and add the code to nvv4l2vp9enc plugin.

[armando-guereca]

Thanks for your reply. I do agree that adding vp9parse into the pipeline would correctly parse the stream and add the proper/expected flags, what is worth nothing here is that vp9parse is not available on GStreamer 1.14 (shipped with devices), I’ll definitely try building a newer version (maybe even the new 1.20), since `gst-v4l2is public I can build that one from source for the new version if needed.
Thanks for the help.