How to Use an Image Fusion Model in DeepStream

• Hardware Platform: Jetson
• DeepStream Version : 6.3
• JetPack Version : 5.1.2
• TensorRT Version : 8.5.2.2
• Issue Type : questions

My image fusion model outputs a single RGB image, and this model is utilized as a secondary GIE model. It seems that gst-nvinfer does not support image fusion models. Which plugins do I need to modify to process the model’s output?

You need to customize the postprocessing. We have customization samples for the models which are not detector, classifier or segmentation. The metadata, postprocessing and pipeline should be customized. Please refer to deepstream_tao_apps/apps/tao_others at master · NVIDIA-AI-IOT/deepstream_tao_apps (github.com)

I have examined the examples within the deepstream_tao_apps/apps/tao_others directory, and it appears that most of them utilize a custom video-template plugin for video stream processing and leverage cvcore functions to perform computations. Would it be feasible for me to incorporate my own custom cvcore methods?Or should I create a custom postprocess plugin to process the video stream? I would like to display the model’s fused image output in real-time on the sink.
image

nvvideotemplate is open source, you can customized your own library as you like.

It depends on your model. For different model, the customization may be different.

Please describe the model input layers and output layers in details so that we can give proper suggestions.

“I have a model with two 1x3x640x640 inputs (which I’ve modified from ONNX to combine into a 2x3x640x640 input), and the output is 1x3x640x640. That is, it takes two images as input and produces one image as output. I am now preparing to add an NvDsPostProcessFusionOutput type in the postprocess plugin to save the model’s output image. Could you tell me specifically which parts I need to modify to output the image to the sink?”

What kind of sink do you want? Display sink? Message broker? File sink?..

Display sink

The proposed pipeline may look like:

source → nvstreammux → nvdspreprocess → nvinfer → nvvideoconvert → nvvideotemplate → nvmultistreamtiler → nvdsosd → nv3dsink

The nvdspreprocess library should be customized to generate the 2x3x640x640 tensor as the model needed. The nvvideoconvert helps you to scale the video resolution to the resolution you want to display and convert the video format to RGBA. In the nvvideotemplate, you should customize the following functions:

  1. get the output 1x3x640x640 tensor and convert it to RGB data
  2. scale the RGB data to the resolution you want to display
  3. you need to replace the RGB data inside NVBufSurface with the RGB data you generated.
1 Like