MMAPI H.264 VUI Aspect Ratio

Hello,

I am testing JetPack 3.2 on a TX2 development board. I was excited to see that the new version of the MMAPI supports VUI insertion in H.264 streams. However, I noticed two things about the VUI that I hope can be enhanced:

  1. The VUI does not include aspect ratio information, and there doesn’t seem to be a way to set it.
  2. The num_units_in_tick and time_scale values in the VUI do not correspond to the frame rate set on the encoder. The values are always 1000/60000 respectively.

Does NVIDIA plan on adding support for more parameters to the VUI (aspect ratio, accurate timing information, color primaries etc, video full range flag, etc. etc.)?

Thank you,

Chris Richardson

Hi Chris,
I need to check with our developer team about the plan, will update to you soon.

Thanks
wayne zhu

Hi cjwti,
1 What is the issue you will meet without aspect ratio information in VUI?
2 we will check the issue about num_units_in_tick and time_scale. The timestamp of each frame is correct, so why you need to read timestamp info in VUI?

Hi all,

Thanks for checking on this and for the quick responses. I really appreciate your consideration on these requests. To answer Dane’s questions:

  1. We have options in our software for performing scaling, cropping and aspect ratio conversion. For the viewer (e.g. VLC, our own client software, many other 3rd party programs we don’t control) to display the video correctly (even without cropping enabled), the aspect ratio must be in the VUI data. We could generate the whole VUI ourselves, but that will involve either refactoring our other platform agnostic code to accept out-of-band VUI data, or will require us to copy/move the encoded buffer around in memory to insert VUI data in the SPS after encoding. Neither option is really great.

  2. The time_scale issue is similar to the aspect ratio. Many clients/players will rely on the timing information in the VUI data, and if it isn’t correct then for example re-muxing to mp4 produces files that play at the wrong speed. We perform frame rate decimation ourselves during capture and set the frame rate appropriately in the encoder. However, the encoder doesn’t seem to use the frame rate when inserting the timing information as far as I can tell.

For the aspect ratio, it would be great to be able to just set the aspect_ratio_idc, sar_width and sar_height fields. Sometimes we use the predefined aspect ratios and sometimes we use the sar_width/sar_height directly.

For the time_scale, it would be nice if the encoder would support fractional frame rates (e.g. 59.94) as well, but even with whole number frame rates, it would be great if it calculated the VUI timing values based on the specified frame rate.

Thank you,

Chris Richardson