Colorspace issue with gstreamer while decoding h264 stream with bt709 VUI

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
,
1

Hello,
I am having troubles decoding a bt709 h264 stream and get correct colors. It can be reproduced on Orin AGX running R36 (release), REVISION: 2.0, as well as on Xavier NX running R32 (release), REVISION: 7.3

We encoded a steady colorbar pattern (grabbed from a 1080p HD source) using x264enc:

gst-launch-1.0 multifilesrc location=img.png caps=image/png,framerate=60000/1001 num-buffers=200 ! pngdec ! videoconvert ! x264enc bitrate=10000 ! video/x-h264,profile=high ! mpegtsmux ! tsdemux ! video/x-h264 ! filesink location=test.h264

The h264 stream has VUI flags as follows (matching gstreamer definition of bt709 colorspace): limited value range, bt709 matrix, bt709 transfer and bt709 color primaries. This can be validated by ffmpeg with trace_headers.

[trace_headers @ 0x55e62e646b40] 84          vui_parameters_present_flag                                 1 = 1
[trace_headers @ 0x55e62e646b40] 95          video_signal_type_present_flag                              1 = 1
[trace_headers @ 0x55e62e646b40] 96          video_format                                              010 = 5
[trace_headers @ 0x55e62e646b40] 99          video_full_range_flag                                       0 = 0
[trace_headers @ 0x55e62e646b40] 100         colour_description_present_flag                             1 = 1
[trace_headers @ 0x55e62e646b40] 101         colour_primaries                                     00000001 = 1
[trace_headers @ 0x55e62e646b40] 109         transfer_characteristics                             00000001 = 1
[trace_headers @ 0x55e62e646b40] 117         matrix_coefficients                                  00000001 = 1

The encoded video is decoded to an output device that requires UYVY format and bt709 colorimetry:

gst-launch-1.0 filesrc location=test.h264 ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw,colorimetry=bt709 ! v4l2sink

Unfortunately, the color rendering is bad (darker than expected). One way to solve the color issue is to first decode to RGBA, then use sw converter (videoconvert) to UYVY. This is not an ideal solution because of the SW conversion.

To help to reproduce and analyse the issue, one frame of the video output can be decoded in different ways to different files, then files are compared. As a reference I also used libav h264 as a software decoder:

  • decoder can be avdec_h264(I420)+videoconvert or nvv4l2decoder(NV12)+nvvidconv
  • the first conversion is done to an intermediate format: RGBA or UYVY
  • a second conversion is done with videoconvert to UYVY (required by the output device)
  • one frame is store to a file for each case (decoder crossed by RGBA/UYVY intermediate format)

The common parts of such a pipeline is as follows:

filesrc location=test.h264 ! h264parse ! <decoder> ! video/x-raw,format=[RGBA,UYVY] ! videoconvert ! video/x-raw,format=UYVY,colorimetry=bt709 ! multifilesink location=<filename> max-files=1

Once having 4 files (avdecRGBA, avdecUYVY, nvdecRGBA, nvdecUYVY), they can be compared using ffmpeg psnr filter as follows:

ffmpeg -pix_fmt uyvy422 -s:v 1920x1080 -i <fileA> -pix_fmt uyvy422 -s:v 1920x1080 -i <fileB> -filter_complex psnr -f null -

Results show that only nvdecUYVY differs significatively from the others: PSNR is under 31dB when nvdecUYVY is compared to another file, while it goes higher than 42dB when comparing other files to each other’s.

The nvconv conversion from NV12 to UYVY looks to be broken. Are there additional tests that can be made ? nvconv parameters to check ?

4

Hi,
Please try to decode the stream via the sample:

/usr/src/jetson_multimedia_api/samples/00_video_decode

And check if format.fmt.pix_mp.colorspace is set to correct format. Please confirm whether the stream can be correctly decoded while running the sample.

5

Thank you for the suggestion.
I compiled and ran 00_video_decode on the Orin AGX (R36.2.0):

$ ./video_decode H264 -o out.raw --disable-rendering colorbar-x264enc-i420.h264 
Set governor to performance before enabling profiler
Creating decoder in blocking mode 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NvMMLiteBlockCreate : Block : BlockType = 261 
Setting frame input mode to 1 
Input file read complete
Starting decoder capture loop thread
Video Resolution: 1920x1080
Video SAR width: 1 SAR height: 1
Decoder colorspace ITU-R BT.709 with standard range luma (16-235)
Query and set capture successful
Got EoS at output plane
Got EoS at capture plane
Exiting decoder capture loop thread
App run was successful

I can confirm that the output frames (I first split out.raw into frames, that were all identical) needed to undergo a bt601 to bt709 to have the good colorimetry, something I did with gstreamer:

gst-launch-1.0 -v filesrc location=/tmp/frame000 ! rawvideoparse format=23 width=1920 height=1080 colorimetry=bt601 ! videoconvert ! video/x-raw, format=UYVY, colorimetry=bt709 ! filesink location=/tmp/frame.raw

Sounds like despite the decoder is configured to output bt709 (as per decoder traces), it still generates bt601.

6

Hi,
It looks like decoder can identify the format correctly:

Decoder colorspace ITU-R BT.709 with standard range luma (16-235)

By default bt601 NvBufSurface is allocated in 00_video_decode sample so the frame data is converted from bt709 to bt601. If yo modify the sample to allocate bt709 NvBufSurface, you should get desired frame data.

And r36.2 is not production release. Please upgrade to r36.3/Jetpack 6.0GA.

8

Thanks for your answer.
I upgraded to 36.3 via using apt (added 36.3 sources, update, dist-upgrade).
I recompiled the video_decoder application, it now outputs the same thing:

$ ./video_decode H264 -o out.raw --disable-rendering colorbar-x264enc-i420.h264 
Set governor to performance before enabling profiler
Creating decoder in blocking mode 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NvMMLiteBlockCreate : Block : BlockType = 261 
Setting frame input mode to 1 
Input file read complete
Starting decoder capture loop thread
Video Resolution: 1920x1080
Video SAR width: 1 SAR height: 1
Decoder colorspace ITU-R BT.709 with standard range luma (16-235)
Query and set capture successful
Got EoS at output plane
Got EoS at capture plane
Exiting decoder capture loop thread
App run was successful

After splitting the output in individual frames, verified they are all the same and transformed into UYVY, the result is identical, colorspace is still wrong and can be fixed with an additional bt601 to bt709 software post-processing.

According the code of the video_decoder application and the traces, the decoder detects bt709 correctly and reports colorspace REC709 in its v4l2 pixfmt for capture plane, then the nv surface allocated to receive the data is correclty created setting colorFormat to NVBUF_COLOR_FORMAT_NV12_709 (this is all in the function query_and_set_capture of the sample code).

9

Hi,
For UYVY we support NVBUF_COLOR_FORMAT_UYVY_709 and NVBUF_COLOR_FORMAT_UYVY_709_ER. Please convert decoded YUV to the format and check. You can modify 00_video_decode sample to allocate NvBufSurface in the format, call NvBufSurfTransform(), and check if converted data is correct.

11

Hi. Using r36.3 both sample application (modified to convert to UYVY BT709) and gstreamer works as expected.
Is there a plan to backport the UYVY BT709 support for Xavier NX as well ?

12

Hi,
The formats NVBUF_COLOR_FORMAT_UYVY_709 are added in Jetpack 6. Please consider use Orin series to have the capability.

13

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.