Apply opencv frame to Sgie

[nomeis]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
RTX2080
• DeepStream Version
6.1
• JetPack Version (valid for Jetson only)
• TensorRT Version
8.1
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Im working on RetinaFace model, which has 3 outputs, bbox, conf and landmarks. I’m doing alignment using the extracted landmarks from the model . So, I have converted the cropped image of the face to mat and I did the alignment using Opencv. Next step is feed this mat to a recognition model.

How to feed the opencv mat to SGIE?? I’m doing all of these in osd_sink_pad_buffer_probefunction from deepstream_infer_tensor_meta_test.cpp

[Fiona.Chen]

@nomeis

How to feed the opencv mat to SGIE??

There is no way to feed such things since the buffer should be GPU memory.

@nomeis

I’m doing alignment using the extracted landmarks from the model . So, I have converted the cropped image of the face to mat and I did the alignment using Opencv.

Can you elaborate the detailed algorithm? Do you mean you cropped the image of the face with bbox and then do some alignment according to the landmarks?

There is already bbox based image cropping inside nvinfer plugin, so we need to know what kind of preprocessing is missing.

[nomeis]

@Fiona.Chen

Can you elaborate the detailed algorithm? Do you mean you cropped the image of the face with bbox and then do some alignment according to the landmarks?

Yes

@Fiona.Chen

There is already bbox based image cropping inside nvinfer plugin, so we need to know what kind of preprocessing is missing.

I did the preprocessing for the alignment using opencv and now I want to pass the aligned face to SGIE for feature extraction

[Fiona.Chen]

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

@nomeis

I did the preprocessing for the alignment using opencv and now I want to pass the aligned face to SGIE for feature extraction

You should add your processing into the preprocessing part in nvinfer source code. nvinfer needs GPU buffer for processing(including inferencing by TensorRT), if you need to use opencv, you need to copy the CUDA memory to CPU memory, processing by opencv and then copy the result back to CUDA memory. It will make the whole processing be very slow. The better way is to implement the algorithm with CUDA. Gst-nvinfer — DeepStream 6.3 Release documentation
Another way is to customize your own nvdspreprocess libs to generate tensor data with your algorithms. Gst-nvdspreprocess (Alpha) — DeepStream 6.3 Release documentation
All source code can be found in /opt/nvidia/deepstream/deepstream/sources/