PyNvVideoCodec v2.0 - DecodedFrame's nvcv_image() API support for P016

Gaming and Visualization Technologies Video Processing & Optical Flow
,
1

Hi, I am new here, I am facing a challenge with PyNvVideoCodec. Documentation says PyNvVideoCodec can decode frame in various color formats. Generally, it works fine with NV12format but lately I have been testing with 4K HDR10 video sample. PyNvVideoCodec can decode it in P016 format but after which my code fails:

with open(out_file_path, "wb") as nvcFile:
    for packet in demuxer:        
        for decode2d in nvdec.Decode(packet):
            
            img2d = nvcv.as_image(decode2d.nvcv_image(), nvcv.Format.U8)
            ten2d = nvcv.as_tensor(img2d)
            if ten2d.layout == "NCHW":
                yuv_batch = cvcuda.reformat(ten2d, "NHWC", stream=cvcuda_stream)
            else:
                yuv_batch = ten2d

for NV12 format, I can use the yuv_batch to load into:

cvcuda.cvtcolor_into(rgb_out, yuv_batch, cvcuda.ColorConversion.YUV2RGB_NV12, stream=cvcuda_stream)

However, with P016 frame format, I am facing this error:

PyNvVCExceptionUnsupported                Traceback (most recent call last)
Cell In[3], line 158
    155     break    
    156 for decode2d in frames:
--> 158     img2d = nvcv.as_image(decode2d.nvcv_image(), nvcv.Format.U8)
    159     ten2d = nvcv.as_tensor(img2d)
    160     if ten2d.layout == "NCHW":

PyNvVCExceptionUnsupported: operator() : 
Error code : 801
Error Type : only nv12 and yuv444 supported as of now
at /project/src/PyNvVideoCodec/src/PyNvDecoder.cpp:539

Which brings me to the part that if nvcv_image() only supports NV12 and YUV444, how do I work with the remaining formats supported by the decoders: P016, YUV444_16Bit, NV16 & P216? perhaps there is a workaround to convert this formats to NV12for CVCUDA operations?

2

Hi @escalate.fast
you can pass the decoded frame directly to cvcuda and then reformat it

3

well yes but then I can’t convert it color to one of the supported format for encoding.
here is an example with PyNvVideoCodec 2.0.2 (because 2.0.3 package is not yet added here.

Decoder:

        nvdec = nvc.ThreadedDecoder(

            input_path,

            buffer_size=30, #hold 30 frames at once

            cuda_context=self.pycuda_ctx.handle,

            cuda_stream=self.cvcuda_stream,

            use_device_memory=True,

            output_color_type=nvc.OutputColorType.NATIVE

        )

Loop through Frames:

            while True:
                frames = nvdec.get_batch_frames(15)
                if len(frames) == 0:
                    break  #exit loop

                for decoded2d in frames:
                    pixelFormat = cvcuda.Format.U16 if decoded2d.format == nvc.Pixel_Format.P016 else cvcuda.Format.U8
                    pixelType = cvcuda.Type.U16 if decoded2d.format == nvc.Pixel_Format.P016 else cvcuda.Type.U8
                  #sample pixelFormat and pixelType- not for production

                    img2d = cvcuda.as_image(torch.from_dlpack(decoded2d).contiguous(), pixelFormat)
                    ten2d = cvcuda.as_tensor(img2d)
                    
                    if ten2d.layout == "NCHW":
                        yuv_batch = cvcuda.reformat(ten2d, "NHWC", stream=self.cvcuda_stream)
                    else:
                        yuv_batch = ten2d
                    
                    
                    rgb_tensor = cvcuda.Tensor(
                        (1, self.frame_height, self.frame_width, 3),
                        pixelType,
                        cvcuda.TensorLayout.NHWC,
                    )
                    
                    # --- color convert in place on GPU ---
                    cvcuda.cvtcolor_into(rgb_tensor, yuv_batch, cvcuda.ColorConversion.YUV2RGB_NV12, stream=self.cvcuda_stream)

cvtcolor_into will work great for normal input video that comes in as NV12 color format but when the video decodes frame in P016 this is where the cvcuda.ColorConversion.YUV2RGB_NV12fails. Looking back at CVCUDA supported color conversion. I don’t see any support for P016 or for 10-bit video (Please correct me if I’m wrong because I’m no expert in Video Color Format).

the purpose of this color conversion is to be able to further down the pipeline to encode the video frame into supported format. So far NV12 works great for 8-bit videos:

#sample encoder:
nvenc_resize = nvc.CreateEncoder(self.final_width, self.final_height, "NV12", False, tuning_info="low_latency", codec="h264", fps=self.frame_rate, rc="vbr") 


#For sake of example, we will just resize video frame and, we proceed to convert it back to NV12 for encoding:
resized_input = cvcuda.resize(rgb_tensor, (batch, self.final_height, self.final_width, channels), stream=self.cvcuda_stream)
cvcuda.cvtcolor_into(yuv_tensor, resized_input, cvcuda.ColorConversion.RGB2YUV_NV12, stream=self.cvcuda_stream)

#proceed with 
frame = AppFrameNV12(self.final_width, self.final_height, int(gpu_buffer))
bitstream = nvenc_resize.Encode(frame)
file.write(bytearray(bitstream))

so let me know if there is a way I can handle for P016 color format as well.

Many thanks.

4

Hi @escalate.fast
Apologies for replying late, it seems 10bit support does not work out of the box with current version (v2.0.3) , we are actively working on it. Meanwhile we will also try to find out workaround with current version

5

Thank you for the update. Please keep me posted as and when the support is available. Thank you