Nvvideoconvert different resolutions support (input and output)

[tvai]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU): GTX1650
• DeepStream Version: 6.1
• TensorRT Version: 8.2.5
• NVIDIA GPU Driver Version (valid for GPU only): 510.85.02

I have a simple pipeline using new nvstreammux like this:

filesrc1 -> decodebin2 -> } nvstreammux -> nvinfer -> nvvideoconvert (nvbuf-memory-type=nvbuf-mem-cuda-unified) -> capsfilter (video/x-raw(memory:NVMM) + format=RGBA) -> appsink
filesrc2 -> decodebin2 -> }

filesrc1 has resolution 1920x1080
filesrc2 has resolution 1280x720

When I receive the surface in appsink, I get this:

Just FYI if I remove nvvideoconvert+capsfilter I get this:

It seems nvvideoconvert scales the biggest surface to the same resolution as the smallest one.
How can I convert input surfaces to CUDA UNIFIED, converting to RGBA, but respecting the original resolution.
I don’t want to make resolution uniform between all the buffers in the batch.

I also checked doing it manually with NvBufSurfaceCopy(), but it seems the NvBufSurface API doesn’t permit support of multiple resolution.
When I call NvBufSurfaceCreate() with batch of 2, I cannot specify a different resolution for the buffers.

Thanks in advance for the help/advices.

[fanzh]

please refer to Gst-nvstreammux New — DeepStream 6.1.1 Release documentation, currently new streammux can’t accept different resolutions.

[tvai]

You refer to the documentation and say new streammux can’t accept different resolutions.
In the documentation it is said:

New nvstreammux do not transform/scale batched buffers to a single color-format/resolution unlike the default nvstreammux. A batch can have buffers from different streams of different resolutions and formats.

Also:

When we have plugins that could transform the input buffers (example: change resolution or color format of video buffers in the batch) between nvstreammux and nvstreamdemux, we need to ensure nvstreammux output is homogeneous (meaning buffers from all sources in the batch shall have same resolution and color format). This is a limitation in the new nvstreammux and shall be addressed in upcoming releases.

I am not using nvstreamdemux, so the documentation is not clear at all.

[fanzh]

Here is the key description “a single resolution for this heterogeneous batched buffer is invalid and the muxer’s source-pad-caps is not valid either.” , the solution is converting different resolutions to the same resolution, like this:
gst-launch-1.0
uridecodebin ! nvvideoconvert ! “video/x-raw(memory:NVMM), width=1920, height=1080, format=NV12” ! m.sink_0
uridecodebin ! nvvideoconvert ! “video/x-raw(memory:NVMM), width=1920, height=1080, format=NV12” ! m.sink_1
nvstreammux name=m batch-size=2 ! fakesink async=0

[tvai]

With new streammux what would be the way to add black border to keep the original image aspect ratio?
Old streammux had “enable-padding” property, how should I do to simulate the same behaviour?

[fanzh]

new streammux dose not support scaling and color conversion, please refer to Gst-nvstreammux New — DeepStream 6.1.1 Release documentation,
enable-padding N/A; Scaling and color conversion support Deprecated.

[tvai]

@fanzh I know that.
But if I want to move to new streammux and have the same behaviour as the old one, is there a way to add padding with videoconvert or any other component?

[fanzh]

Sorry for the late reply, currently, you can use nvvideoconvert’s dest-crop to do scaling with keeping the original image aspect ratio. please find it in Gst-nvvideoconvert — DeepStream 6.1.1 Release documentation.

[tvai]

Thanks @fanzh, I’ll try it, I now understand the dest-crop :)
Do you think support for multiple resolution will be added soon to the new streammux? Is it something in your roadmap?

[fanzh]

it is on our roadmap.