Output format of Gst-nvvideo4linux(nvv4l2decoder)

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 5.0.1
• JetPack Version (valid for Jetson only)
• TensorRT Version 7.2.2
• NVIDIA GPU Driver Version (valid for GPU only) 440.87
• Issue Type( questions, new requirements, bugs) question
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

I’m creating original convert which convert from NV12 to UYVU.
And the convert logic is as follows.

global void
d_nv12_uyvy(unsigned char *d_src_Y, unsigned char d_src_UV, unsigned char d_dest, int image_width, int image_height)
{
unsigned int tx = blockIdx.xblockDim.x + threadIdx.x;
unsigned int ty = blockIdx.yblockDim.y + threadIdx.y;

d_dest[(ty * image_width * 2) + (tx * 2)] = d_src_UV[(((ty - (ty & 0x1)) * image_width)/2) + tx];
d_dest[(ty * image_width * 2) + ((tx * 2) + 1)] = d_src_Y[(ty * image_width) + tx];

}

#define D_CUDA_BLOCKDIM_X 32
#define D_CUDA_BLOCKDIM_Y 8

extern “C”
double nv12_uyvy_cuda(unsigned char *d_src, unsigned char *d_dest, int image_width, int image_height)
{
int blockX = D_CUDA_BLOCKDIM_X;
int blockY = D_CUDA_BLOCKDIM_Y;
dim3 blockDim (D_CUDA_BLOCKDIM_X , D_CUDA_BLOCKDIM_Y);
dim3 gridDim ((image_width ) / blockX, (image_height) / blockY);

    unsigned char *d_src_Y = d_src;
    unsigned char *d_src_UV = d_src + image_width * image_height; 

    {
        d_nv12_uyvy<<< gridDim, blockDim >>>(d_src_Y, d_src_UV, d_dest, (image_width), (image_height));
    }
return 1;

}

However, I cannot convert the NV12 data, which is outputted from nvv4l2decoder, correctly.

For example, the ball image is converted to wide spreading ball image as attached image.

ball
converted_ball

Now, I find that there are some NV12 format in GStreamer.
https://gstreamer.freedesktop.org/documentation/video/video-format.html?gi-language=c
So I want to confirm which format is outputted from nvv4l2decoder.

GST_VIDEO_FORMAT_NV12 (23) – planar 4:2:0 YUV with interleaved UV plane
GST_VIDEO_FORMAT_NV12_64Z32 (53) – NV12 with 64x32 tiling in zigzag pattern (Since: 1.4)
GST_VIDEO_FORMAT_NV12_4L4 (97) – NV12 with 4x4 tiles in linear order.
GST_VIDEO_FORMAT_NV12_32L32 (98) – NV12 with 32x32 tiles in linear order.

Regards,
hiro

3

nvv4l2decoder is a hardware accelerated video decoder which using Nvidia VIC hardware. The output of nvv4l2decoder is not standard NV12 format as described in gstreamer. The output buffer is a hardware buffer which is customized for using Nvidia hardware and the format is proprietary. Your conversion function is not correct. We already have nvvideoconvert plugin(Gst-nvvideoconvert — DeepStream 6.1.1 Release documentation) to transfer the harware buffer to different format. Please using Nvidia deepstream plugin for your deepstream application.

4

Dear Fiona.Chen

Thank you for your information.
Your information is very helpful for us.

By the way, the output format of nvv4l2decoder is changed?
My colleague said that the output of DS4’s stream is GST_VIDEO_FORMAT_NV12.
And our converter works well on DS4.

Regards,
hiro

5

The format is not GST_VIDEO_FORMAT_NV12. It may change for different video format(JPEG, H264, HEVC). Gst-nvvideo4linux2 — DeepStream 6.1.1 Release documentation

6

Thank you for your information.

I will try to use nvvideoconvert plugin.

7

Dear Fiona.Chen,

We’ll try to use nvvideoconvert plugin for getting ‘video/x-raw(memory:NVMM),format=(string)NV12’ data as follows.

$ gst-launch-1.0 filesrc location=./ball.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! nvvideoconvert ! ‘video/x-raw(memory:NVMM),format=(string)NV12’ ! out_converter ! xvimagesink

However, nvvideoconvert doesn’t output GST_VIDEO_FORMAT_NV12 and seems to pass thorough the data.
Is it true?
And, Is there any suggestion to get GST_VIDEO_FORMAT_NV12 on NVMM memory.
If there isn’t good solution, could I ask about proprietary NV12 format?

Regards,
hiro

8

Hi Fiona, I am also facing same issue.
I want to create CUDA-based plug-in to directly access output of nvv4l2decoder. Is there anyway to use nvvideoconvert or other filter to convert proprietary format (I think it is NV12 block linear) to NV12 pitch linear format on NVMM without copying data from/to cpu memory?

9

@Hiromitsu.Matsuura nvvideoconvert can output GST_VIDEO_FORMAT_NV12, you need to change the caps to ‘video/x-raw,format=(string)NV12’. ‘video/x-raw(memory:NVMM),format=(string)NV12’ is the Nvidia HW format.

10

@Frona, Chen,

Thank you for your comment.

nvvideoconvert can output GST_VIDEO_FORMAT_NV12, you need to change the caps to ‘video/x-raw,format=(string)NV12’. ‘video/x-raw(memory:NVMM),format=(string)NV12’ is the Nvidia HW format.

I confirm that we can get GST_VIDEO_FORMAT_NV12 on CPU memory.
However, we want to continue GPU processing.
So we need GST_VIDEO_FORMAT_NV12 on GPU memory.

This is the reason we want to request to get GST_VIDEO_FORMAT_NV12 on NVMM memory and reveal proprietary NV12 format.

Regards,
hiro

11

No. There is no GST_VIDEO_FORMAT_NV12 on GPU memory.

The HW buffer(what you can think it is GPU buffer) is encapsulated as NvBufSurface. There are sample source codes in gst-dsexample plugin and deepsteam_image_meta_test app to show how to access NvBufSurface with GstBuffer.

The layout of the NvBufSurface buffer is described in nvbufsurface.h. There are many different buffer types.

12

Dear Fiona, Chen

Thank you for your information.
I will check it.

Regards,
hiro