Unexpected color space conversion with h.264 AYUV format (444 yuv)

I’m encoding h.264 video using the NV_ENC_BUFFER_FORMAT_AYUV buffer format. The input frames are of type DirectX and in DXGI_FORMAT_AYUV format. I configure the encoder to work with 4:4:4 YUV (no chroma subsampling) by setting chromaFormatIDC = 3 and by using the NV_ENC_H264_PROFILE_HIGH_444_GUID profile.

Contrary to what I expect, the encoder apparently performs color space conversion for the frames prior to encoding them, as if I was inputting the frames in RGB format. I even tried to avoid it manually by specifying identity color matrix for the transformation in the VUI parameters (h264VUIParameters.colourMatrix = 0 and colourDescriptionPresentFlag = 1) but it seems to have no effect.

I haven’t encountered this problem when inputting frames in NV12 format (NV_ENC_BUFFER_FORMAT_NV12) and, hence, using 4:2:0 chroma subsampling.

Am I doing something wrong or what is going on here?

Hello @patsku79 and welcome to the NVIDIA developer forums!

First of all, can you tell me which driver version you are using?

The behavior you are describing is very likely related to a driver issue we found in the NVIDIA driver starting with versions 470.x and later, where AYUV is interpreted as RGB by the encoder. The issue is known and fixed internally, but it is not confirmed yet what future public release will include this fix.

One thing to check on your side would be to try and revert the NVIDIA driver to 462.59 for example, if that is possible, and check if your code works as expected then. And it would be great if you could share your result with us.

Thank you!

Dear @MarkusHoHo ,

Thanks a lot for your message! I’m indeed using a driver version 472.12 and reverting back to the older driver you suggested fixes the issue. I would never have thought about trying that…

As I was looking for ways to go around this, I came up with a related question: Which DXGI format can be used to input YUV444 frames in planar format (NV_ENC_BUFFER_FORMAT_YUV444)? Or is it so that there is no such option to input planar YUV444 frames (8-bits per channel) to the encoder with DirectX? I didn’t find anything that would work. I thought maybe I could just use DXGI_FORMAT_R8_UNORM with three times the frame height but the encoder didn’t agree with me (NV_ENC_ERR_UNSUPPORTED_PARAM while calling nvEncRegisterResource()). There is probably the same issue with 10-bit YUV444 as there seems to be no packed NV_ENC_BUFFER_FORMAT and no planar DXGI format available.

Hi again,

Interesting that you ask. There was a similar request recently here in the forum, check this topic for reference: YUV4:4:4 HEVC Encoding in D3D12

In short, the answer is No, you cannot use planar 444 in DX12 with NvEnc.