TX2 H.264 invalid SPS and no options for VUI

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

Hello,

I’m working with a TX2 development board, using the MMAPI to capture and encode video in H.264 format. This is mostly working fine. However, I notice two things:

  1. The SPS NAL unit seems to be invalid. The (fairly old) JM build I am using to test this does not accept the SPS due to the reserved_zero_4bits not actually being zero. A sample NAL unit looks like this:
0  1  2  3  4  5  6  7  8  9  10 11 12 13 14 15
00 00 00 01 67 64 0C 32 AC 2C A8 07 80 22 7E 54

As you can see, two of the lower 4 bits of byte 6 are 1. Other encoders that I have access to from different vendors produce zeros in these bits. Any ideas what is going on here?

  1. I can’t find a way to insert VUI information in the SPS. I am aware that I could generate my own VUI data if I wanted to, but I don’t really want to have to do this. Does anyone at NVIDIA know if there are plans to allow VUI insertion via the MMAPI? The specific parameters I am interested in are the timing info, aspect ratio, color primaries and pixel range.

Thank you for your time,

Chris Richardson

2

Hi Chris,
For 1 please attach the correct SPS for our reference. What is the revision of your JM decoder? Does the non-zero padding lead to failure of decoding in any PC/Android/iOS platform?

For 2, does absence of VUI lead to failure of decoding in any platform?

Please provide more information.

3

Hi DaneLLL,

Thank you for the response.

  1. I looked at the newer JM software, version 19.0, since mine was quite old. It looks like the two bits set in the NV SPS are constraint_set4_flag and constraint_set5_flag, which the JM software will allow if the MVC_EXTENSION_ENABLE macro is defined (https://www.hhi.fraunhofer.de/en/departments/vca/research-groups/image-video-coding/research-topics/mvc-extension-of-h264avc.html). I’m not sure if the bits should only be set if sending MVC profile data or not, as my expertise with MVC is non-existent.

I don’t notice any problem decoding the stream with ffmpeg on my PC, but I like to test with JM as well to make sure I am not sending out invalid streams. Additionally, my customers have an unknown array of hardware decoders, and I won’t be able to test them all. Assuming we end up using this NVIDIA module, I will eventually get feedback from my customers.

Anyways, for completeness, below you can see an example SPS generated by TI hardware which we currently use. Please ignore that the start code is not present at the beginning; it is from a live stream that doesn’t include them. You can see that byte 3 is all zeros in this case, and that much more information (including the VUI) is included in the SPS.

00000000 67 64 00 28 AD 84 05 45  62 B8 AC 54 74 20 2A 2B   gd.(...Eb..Tt *+
00000010 15 C5 62 A3 A1 01 51 58  AE 2B 15 1D 08 0A 8A C5   ..b...QX.+......
00000020 71 58 A8 E8 40 54 56 2B  8A C5 47 42 02 A2 B1 5C   qX..@TV+..GB...\
00000030 56 2A 3A 10 24 85 21 39  3C 9F 27 E4 FE 4F C9 F2   V*:.$.!9<.'..O..
00000040 79 B9 B3 4D 08 12 42 90  9C 9E 4F 93 F2 7F 27 E4   y..M..B...O..'.
00000050 F9 3C DC D9 A6 B4 03 C0  11 3F 2E 02 84 04 04 05   .<.......?......
00000060 00 00 03 00 01 00 00 03  00 3C E0 40 00 3D 09 00   .........<.@.=..
00000070 00 22 55 1B DE F8 5E 11  08 D4 ·· ·· ·· ·· ·· ··   ."U...^...
  1. It’s not that decoding fails without the VUI, but display is not good, since the video comes out with the wrong color levels and aspect ratio. Additionally, the timing information is used by several broadcast decoders I am testing with, and I would like to continue sending this information in our streams if we switch our hardware over from TI to NVIDIA. It seems very surprising to me that NVIDIA does not have software support for this basic feature on such an expensive module.

Thanks,

Chris Richardson

4

Hi Chris,
For 1, it looks fine with JM decoder1.9.0

@0     SPS: profile_idc                                       01000010 ( 66) 
@8     SPS: constrained_set0_flag                                    0 (  0) 
@9     SPS: constrained_set1_flag                                    1 (  1) 
@10    SPS: constrained_set2_flag                                    0 (  0) 
@11    SPS: constrained_set3_flag                                    0 (  0) 
@12    SPS: constrained_set4_flag                                    0 (  0) 
@13    SPS: constrained_set5_flag                                    0 (  0) 
@14    SPS: reserved_zero_2bits                                     00 (  0) 
@16    SPS: level_idc                                         00110100 ( 52) 
@24    SPS: seq_parameter_set_id                                     1 (  0) 
@25    SPS: log2_max_frame_num_minus4                            00101 (  4) 
@30    SPS: pic_order_cnt_type                                     011 (  2) 
@33    SPS: num_ref_frames                                         010 (  1) 
@36    SPS: gaps_in_frame_num_value_allowed_flag                     0 (  0) 
@37    SPS: pic_width_in_mbs_minus1                      0000001111000 (119) 
@50    SPS: pic_height_in_map_units_minus1               0000001000100 ( 67) 
@63    SPS: frame_mbs_only_flag                                      1 (  1) 
@64    SPS: direct_8x8_inference_flag                                1 (  1) 
@65    SPS: frame_cropping_flag                                      1 (  1) 
@66    SPS: frame_crop_left_offset                                   1 (  0) 
@67    SPS: frame_crop_right_offset                                  1 (  0) 
@68    SPS: frame_crop_top_offset                                    1 (  0) 
@69    SPS: frame_crop_bottom_offset                             00101 (  4) 
@74    SPS: vui_parameters_present_flag                              0 (  0)

It shows reserved_zero_2bits instead of reserved_zero_4bits

For 2, we will evaluate to add inserting VUI into MM APIs.

5

Hi DaneLLL,

Thanks for the response. Regarding 1, thanks. It looks fine with the newer JM. Thanks for consideration on adding support for number 2. VUI insertion will be very helpful for us.

Thanks,

Chris Richardson