Colorspace issue while encoding bt709 source to h264

lgonzalez · August 22, 2024, 4:25pm

Hi, I am having trouble to correctly generate an h264 stream that has the correct information to inform the decoder about the colorspace. This happens on an Orin dev kit running r36.3.

We experience the issue with gstreamer, but it can be nailed down to the encoder itself by running the 01_video_encoder sample. A raw source is encoded, option is given to notify the encoder it is bt709. The produced h264 stream has only partial information about the colorspace, that the decoder cannot correctly identify as bt709.

$ ./video_encode colorbar.raw 1920 1080 H264 test.h264 --color-space 2 --insert-vui
Creating Encoder in blocking mode 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 4 
===== NvVideo: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
875967048
842091865
H264: Profile = 66 Level = 51 
NVMEDIA: Need to set EMC bandwidth : 846000 
NvVideo: bBlitMode is set to TRUE 
Could not read complete frame from input file
File read complete.
Got 0 size buffer in capture 
App run was successful

$ ./video_decode H264 -o out.raw --report-input-metadata --disable-rendering test.h264
Set governor to performance before enabling profiler
Creating decoder in blocking mode 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NvMMLiteBlockCreate : Block : BlockType = 261 
Setting frame input mode to 1 
NVMMLITE_NVVIDEODEC, <NvMMLiteNvVideoDecSetAttribute:9359> Error status reporting set to 1
Input file read complete
Starting decoder capture loop thread
Got EoS at output plane
Video Resolution: 1920x1080
Video SAR width: 0 SAR height: 0
Decoder colorspace ITU-R BT.601 with standard range luma (16-235)
Query and set capture successful
Got EoS at capture plane
Exiting decoder capture loop thread
App run was successful

A deeper analysis of the h264 stream show that some parameter might not be properly set:

$ ffmpeg -i test.h264 -c copy -bsf:v trace_headers -f null - 2>&1 | grep -A 10 vui
[trace_headers @ 0xaaaac6be6e30] 87          vui_parameters_present_flag                                 1 = 1
[trace_headers @ 0xaaaac6be6e30] 88          aspect_ratio_info_present_flag                              0 = 0
[trace_headers @ 0xaaaac6be6e30] 89          overscan_info_present_flag                                  0 = 0
[trace_headers @ 0xaaaac6be6e30] 90          video_signal_type_present_flag                              1 = 1
[trace_headers @ 0xaaaac6be6e30] 91          video_format                                              101 = 5
[trace_headers @ 0xaaaac6be6e30] 94          video_full_range_flag                                       0 = 0
[trace_headers @ 0xaaaac6be6e30] 95          colour_description_present_flag                             1 = 1
[trace_headers @ 0xaaaac6be6e30] 96          colour_primaries                                     00000000 = 0
[trace_headers @ 0xaaaac6be6e30] 104         transfer_characteristics                             00000000 = 0
[trace_headers @ 0xaaaac6be6e30] 112         matrix_coefficients                                  00000001 = 1
[trace_headers @ 0xaaaac6be6e30] 120         chroma_loc_info_present_flag                                0 = 0

Turns out that colour_primaries and transfer_characteristics are set to 0 (reserved value according to the norm), while other encoders set them to 1.

What could be done to help us getting a full bt709 support?

DaneLLL · August 23, 2024, 1:48am

Hi,
Please try ffprobe:

$ ffprobe -i test.h264

And check if it shows bt709 like:

  Stream #0:0: Video: h264 (Constrained Baseline), yuv420p(tv, bt709/reserved/reserved, progressive), 1920x1080, 30 fps, 30 tbr, 1200k tbn, 60 tbc

lgonzalez · August 23, 2024, 2:53pm

Hi, thank you for your answer.
ffprobe returns the following:

Input #0, h264, from 'test.h264':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: h264 (Constrained Baseline), yuv420p(tv, bt709/reserved/reserved, progressive), 1920x1080, 30 tbr, 1200k tbn, 60 tbc

As a reference, we have an external encoder that produces an h264 stream that has no trouble in the decoder. The use of “reserved” keyword by ffprobe (for h264 produced by video_encode) looks to match colour_primaries and transfer_characteristics, which turns out to prevent the decoder to correctly identify the colorspace as being bt709.

$ ffprobe -hide_banner -i test.ssg.h264 
Input #0, h264, from 'test.ssg.h264':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: h264 (Main), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 56 fps, 59.94 tbr, 1200k tbn, 119.88 tbc

$ ffmpeg -i test.ssg.h264 -c copy -bsf:v trace_headers -f null - 2>&1 | grep -A 10 vui
[trace_headers @ 0xaaaaf8226250] 84          vui_parameters_present_flag                                 1 = 1
[trace_headers @ 0xaaaaf8226250] 85          aspect_ratio_info_present_flag                              1 = 1
[trace_headers @ 0xaaaaf8226250] 86          aspect_ratio_idc                                     00000001 = 1
[trace_headers @ 0xaaaaf8226250] 94          overscan_info_present_flag                                  0 = 0
[trace_headers @ 0xaaaaf8226250] 95          video_signal_type_present_flag                              1 = 1
[trace_headers @ 0xaaaaf8226250] 96          video_format                                              010 = 2
[trace_headers @ 0xaaaaf8226250] 99          video_full_range_flag                                       0 = 0
[trace_headers @ 0xaaaaf8226250] 100         colour_description_present_flag                             1 = 1
[trace_headers @ 0xaaaaf8226250] 101         colour_primaries                                     00000001 = 1
[trace_headers @ 0xaaaaf8226250] 109         transfer_characteristics                             00000001 = 1
[trace_headers @ 0xaaaaf8226250] 117         matrix_coefficients                                  00000001 = 1

$ ./video_decode H264 -o out.ssg.raw --report-input-metadata --disable-rendering test.ssg.h264
Set governor to performance before enabling profiler
Creating decoder in blocking mode 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NvMMLiteBlockCreate : Block : BlockType = 261 
Setting frame input mode to 1 
NVMMLITE_NVVIDEODEC, <NvMMLiteNvVideoDecSetAttribute:9359> Error status reporting set to 1
Starting decoder capture loop thread
Video Resolution: 1920x1080
Video SAR width: 1 SAR height: 1
Decoder colorspace ITU-R BT.709 with standard range luma (16-235)
Query and set capture successful
Input file read complete
Got EoS at output plane
Got EoS at capture plane
Exiting decoder capture loop thread
App run was successful

DaneLLL · August 27, 2024, 5:13am

Hi,
Please try

Apply attached lib
Set color space in calling VIDIOC_S_FMT

@@ -136,6 +136,7 @@ NvVideoEncoder::setOutputPlaneFormat(uint32_t pixfmt, uint32
_t width,
     format.fmt.pix_mp.height = height;
     format.fmt.pix_mp.pixelformat = pixfmt;
     format.fmt.pix_mp.num_planes = num_bufferplanes;
+    format.fmt.pix_mp.colorspace = V4L2_COLORSPACE_REC709;

     return output_plane.setFormat(format);

Rebuild and run 01 sample

r36_3_TEST_libtegrav4l2.zip (85.6 KB)

lgonzalez · August 27, 2024, 2:39pm

Hi,
thank you for providing the test library. Now we have a better VUI in the h264 stream:

$ ffprobe -hide_banner -i test.h264 
Input #0, h264, from 'test.h264':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: h264 (Constrained Baseline), yuv420p(tv, bt709, progressive), 1920x1080, 30 tbr, 1200k tbn, 60 tbc

The decoder from sample code also detects bt709 correclty.

Any chance to have this library upgrade back-ported for Xavier ?

DaneLLL · August 28, 2024, 11:44pm

Hi,
Please apply attached lib to r35.5.0 and give it a try.

r35_5_TEST_libtegrav4l2.zip (83.2 KB)

lgonzalez · September 3, 2024, 7:31pm

Thank you for providing that upgrade.
Our Xavier is running an older version of the SDK (32.7.3), so we need to upgrade before we can test it.
I will give feedback when done.