Convert NV12 to RGB in DeepStream pipeline

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
AGX Xavier
• DeepStream Version
5.0
• JetPack Version (valid for Jetson only)
4.5
• TensorRT Version
7.1.3
• Issue Type( questions, new requirements, bugs)
question

I have a DeepStream pipeline with a custom plugin in it that only accepts RGB input. I didn’t find a simple solution to move from the NV12 output from the nvv4l2decoder to RGB for the custom plugin.
I’m currently stuck using the pipeline below which results in a internal datastream error.

Pipeline:

gst-launch-1.0
filesrc location=“input.mp4” ! qtdemux ! h264parse ! nvv4l2decoder !
nvvideoconvert ! ‘video/x-raw,format=(string)RGBA’ !
videoconvert ! ‘video/x-raw,format=(string)RGB’ !
custom_plugin ! nvvideoconvert !
‘video/x-raw(memory:NVMM),format=(string)NV12’ !
m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 enable-padding=0 !
nvinfer config-file-path=“config_infer_primary_yolo.txt” !
nvvideoconvert ! nvdsosd ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! mp4mux !
filesink location=“result.mp4”

Could you provide a simple pipeline that loads a h264 mp4 file, converts to RGB and performs inference with nvinfer after that?

3

Can you use “gst-inspect-1.0 custom_plugin” to query the information of the custom_plugin?

4

The pipeline will not be simple because deepstream plugins uses Nvidia HW buffers while custom_plugin uses software buffers, the HW buffer to SW buffer transferring and SW buffer to HW buffer transferring will make the pipeline complicated. Deepstream pipeline needs deepstream plugins.

5

The accepted input/output format from gst-inspect is video/x-raw format: {GRAY8,RGB}.

6

Okay I see, I was under the impression that nvvideoconvert could switch between HW/SW buffers as it has src and sinkpads for memory:NVMM and without. Unfortunately the custom_plugin is non-deepstream and I cannot change it. My plan was to do all the preprocessing in a regular GStreamer pipeline and then switch to DeepStream for the neural networks/visualization.

7

nvvideoconvert does not support GRAY8 or RGB. Please add videoconvert before nvvideoconvert(the second one).

8

Thanks for the lightning fast answer!
You were right, I didn’t consider to convert back using another videoconvert after custom_plugin, now it works!
For anyone wanting do something similar, here is the full pipeline that is working now:

gst-launch-1.0
filesrc location=“input.mp4” ! qtdemux ! h264parse ! nvv4l2decoder !
nvvideoconvert ! ‘video/x-raw,format=(string)RGBA’ !
videoconvert ! ‘video/x-raw,format=(string)RGB’ !
custom_plugin ! videoconvert ! ‘video/x-raw,format=(string)NV12’ ! \ nvvideoconvert !
‘video/x-raw(memory:NVMM),format=(string)NV12’ !
m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 enable-padding=0 !
nvinfer config-file-path=“config_infer_primary_yolo.txt” !
nvvideoconvert ! nvdsosd ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! mp4mux !
filesink location=“result.mp4”