I am currently working on a plugin that use a buffer pool, which instantiates NvBufSurface buffers with NvBufSurfaceCreate. These buffers must have the exact same properties than the buffers that pass through the pipeline.
With gst_nvquery_batch_size_new it’s easy to get batch size, and the GstCaps gives us width and height. However, for color format, the GstCaps “format” field is not enough: when NV12 is provided, I observed that the NvBufSurfaceParams.colorFormat can randomly be NVBUF_COLOR_FORMAT_NV12_709 instead of NVBUF_COLOR_FORMAT_NV12.
I guess the color format is defined in nvstreammux, but it is not open source. How nvstreammux choose between NVBUF_COLOR_FORMAT_NV12_709 and NVBUF_COLOR_FORMAT_NV12, and how can I do the same in my plugin ?
• Hardware Platform: Jetson
• DeepStream Version: 5.0 GA
• JetPack Version: 4.4
The pixel format is decided by video decoder(nvv4l2h264dec,nvv4l2h265dec,…) which will be different with different Nvidia platforms.
The NvBufSurface pixel format should be decided with the consideration of how will it be used (to be used by GPU, VIC,…) . I think there is sample with gst-nvinfer plugin which is open source.
My pipeline does not use a video decoder but nvvideoconvert, I deduce that this is where NvBufSurface buffers are created in my case. I don’t set any property of nvvideoconvert, and only ensure its sink capabilities are video/x-raw(memory:NVMM), format=NV12 with a capsfilter element.
Because every property of nvvideoconvert are set to default (on Jetson), I would expect that the output buffer color format is always the same. However, depending on the other elements of the pipeline, I sometimes get NVBUF_COLOR_FORMAT_NV12_709, and sometimes NVBUF_COLOR_FORMAT_NV12.
How can I know by advance which one will be use, or how can I force one color format to be used ?
Edit: there’s nothing about NVBUF_COLOR_FORMAT_NV12_709 in the gst-nvinfer source code.
I’m curious of where you get the " sometimes NVBUF_COLOR_FORMAT_NV12_709 , and sometimes NVBUF_COLOR_FORMAT_NV12 " from Caps? In which plugin and what pipeline?
You’ll find attached a minimal, reproducible example.
There is a pad probe attached to the src pad of the streammux element, where I display the colorFormat value of the buffer.
I observed that:
- When
WIDTH=1280 and HEIGHT=720, the program will display : colorFormat = 33, which means NVBUF_COLOR_FORMAT_NV12_709.
- When
WIDTH=848 and HEIGHT=480, the program will display : colorFormat = 6, which means NVBUF_COLOR_FORMAT_NV12.
Back to my original question, what’s the underlying rules behind this behavior ?
main.cpp (5.5 KB)
In case of nvstreammux, We check for the buffer format on the input pad of streammux and allocate buffer matching to input colorimetry. Irrespective of resolution input and buffer transformed will have colorimetry. This check takes place runtime.