[H264/HEVC]Passing private pointers through enocders

Hi,

Has anyone validated passing private pointers(like discontinuity information) through H264 and HEVC encoders on Jetson TX1 platforms?

Regards,
durgeshmn

Hi durgeshmn,
Is your app coded in gstreamer pipeline, or MMAPI?

Hi DaneLLL,

We are using our own pipeline with omx plugins.

Regards,
durgeshmn

hi durgeshmn,
We don’t support omx-il layer on TX1. we have samples in gstreamer pipeline and MMAPI.

Hi DaneLLL,

We tried using omxh264encode Gstreamer plugin but it does not have provision to pass timestamp from input of encoder to output. This will be very necessary if B frames are used. Please let us know if there is a way to pass timestamp or pointers for every process call.

Regards,
durgeshmn

Hi durgeshmn,
The timestamp info is put in GST_BUFFER_PTS()
https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/GstBuffer.html#GST-BUFFER-PTS:CAPS

Hi DaneLLL,

Does H264(encoder & decoder) and HEVC(encoder & decoder) supports SEI/user data insertion on encode side and parsing/passing to application on decode side ?

Does MPEG2 decoder supports user data extraction?

Regards,
durgeshmn

hi durgeshmn,
We don’t support it. You need to implement it on your own. We suggest putting it in gstomxvideoenc.c/gstomxh264enc.c/gstomxh265enc.c

Source code gstomx1_src.tbz2 is in https://developer.nvidia.com/embedded/dlc/l4t-24-2-sources

Attach the patch for your reference.

If the user data is put per H264/HEVC/MPEG2 stream spec, the decoding should work fine since the decoder will simply discard the non-bitstream data.

Hi DaneLLL,

Thanks for your response and patch files.

Regards,
durgeshmn

Hi DaneLLL,

I need few more clarifications:

  1. Does H264 & HEVC encoders supports both Closed GOP and Open GOP?
  2. If Closed GOP is supported, does it support strict Closed GOP?
  3. Does GOP period of less than 2 seconds supported?
  4. Does H264 & HEVC encoders supports CBR, VBR and Lookahead bitrate control modes ?
  5. Are H264 & HEVC encoders VBV buffer compliant?

Thanks,
durgeshmn

Hi dureshmn,

  1. Does H264 & HEVC encoders supports both Closed GOP and Open GOP?
    Closed GOP is supported by default. Open GOP NOT supported.

  2. If Closed GOP is supported, does it support strict Closed GOP?
    What is strict Closed GOP?

  3. Does GOP period of less than 2 seconds supported?
    Yes, using I frame interval.

  4. Does H264 & HEVC encoders supports CBR, VBR and Lookahead bitrate control modes ?
    CBR & VBR are supported. Lookahead bitrate control mode is NOT supported.

  5. Are H264 & HEVC encoders VBV buffer compliant?
    We are checking further.

Hi DaneLLL,

Thank you!

Please update me on the VBV compliance after your testing.

Regards,
durgeshmn

Hi durgeshmn,
We should be VBV buffer compliant. You can configure it via the property ‘vbv-size’.
Please refer to the source code gstomx1_src.tbz2 for detail.

Hi DaneLLL,

For our application, handling variable number of B frames is a must on decoding side. So we cannot maintain and match timestamp from application. The support has to be there from Codec library itself wherein the library gives the corresponding timestamp when the frame is returned to application after decoding.
Does TX1 platform support this feature?
If not, how can we get this features? This being very critical to us.
Since we have our own framework, we may not use Gstreamer framework. Does the codec support OMX APIs?

Does codec support automatic detection of input stream properties like Framerate, profile, level, coding type etc. Does the codec provides these info to application when IDR/I frames is decoded? How can application get these info?

Is it possible to change codec properties- framerate, bitrate, resolution runtime/dynamic ?

Thanks and regards,
durgeshmn

Hi durgeshmn,
We don’t support OMX APIs. Beside gstreamer, you can also choose MM APIs. Please install it via Jetpack and refer to samples at /home/ubuntu/tegra_multimedia_api

The decoder does not assign timestamps as the timestamp should be obtained from the container(mp4, ts, avi). For the case:
$ gst-launch-1.0 filesrc location=a.ts ! tsdemux ! h264parse ! omxh264dec ! nvoverlaysink
tsdemux can get the timestamps for each frame

The codec info is in SPS/PPS/(VPS of HEVC) NALs. You have to parse them to get the info.
For H264, you can refer to FindAVCDimensions() at
https://android.googlesource.com/platform/frameworks/av/+/master/media/libstagefright/avc_utils.cpp

For changing encoder parameters dynamically, please refer to
https://devtalk.nvidia.com/default/topic/963451

Hi DaneLLL,

Yes, Decoders does not assign time stamps, but has to come with the incoming stream. But, my query was, incase if the incoming streams has B frames, the frame order at the output of the decoder will be different from the input order. No of B frames also varies from streams to streams. So TSDMUX cannot maintain and assign time stamp corresponding to each frames. If the codec can pass through the time stamp info associated with a frame and return at the output, it is easy for the application to use the time stamp as the application gets in presentable order. So, we were analyzing if this is supported in any of the above mentioned frameworks.

Thanks and regards,
durgeshmn

Hi durgeshmn,
The decoder can pass through the timestamp. The frames may not be outputted as the input orders due to decoding algorithm, but the timestamps are matched.

If you experience issues with B-frame deocding. Please share the stream and steps so that we can do further check.