How to custom preprocess in SGIE base on Deepstream 5.0?

Accelerated Computing Intelligent Video Analytics DeepStream SDK
#1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
dGPU
• DeepStream Version
deepstream 5.0
• JetPack Version (valid for Jetson only)
• TensorRT Version
7.0
• NVIDIA GPU Driver Version (valid for GPU only)
cuda 10.2
• Issue Type( questions, new requirements, bugs)
questions
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

my question is similar like Image pre-processing between PGIE and SGIE

i plan to build a pipeline for face deetcting, the pipeline like this:
PGIE(face detection)---->SGIE1(face landmark predict)---->SGIE2(face attribute predict (ages? male or female? with mask?))

preprocess of SGIE2 need the SGIE1 result (landmark points) to do face algin(similar transform or affine transform). Assume that I can save the SGIE1 results to ObjectMetaData.

How to do the preprocess of SGIE2?
the answer of Image pre-processing between PGIE and SGIE
is not clearly for me . and I did not find the code memtion in that question(may be sth changed in deepstream 5.0 ?)

#3

Pls check the comment #6 in that topic.
BTW, you can refer DeepStream 5.0 nvinferserver how to use upstream tensor meta as a model input if you are using nvinferserver

#4
  1. you suggested add probe between pgie and sgie,and do Affine transform on every box in that probe func on the comment #6 , but for our requirement, we need to do the affine transformation on the ROI IMAGE(the image croped from frame based on the bbox),using the landmark points as the “source points”.Then send the transformed ROI image as input image to sgie to do infer. So the transform not on bbox,it’s on the cropped image.
    2.we plan to use nvdsinfer-plugin
    3.for the comment #11 we cannot find the code in Deepstream 5.0
#5

The sgie will crop the ROI from original image based on your modified bbox, if you need to do some affine transformation based on the croped image, then you need to modify the nvinfer preprocess code gstnvinfer.cpp -> gst_nvinfer_process_objects

#6

thanks！BTW，where can I find the implement details of gstnvinfer？ I‘m afraid can not understand the code clearly without other information.

#7

Check the function code and all the nvinfer is open sourced, you can refer DeepStream SDK FAQ and the nvinfer plugin manual https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html

#8

thanks! let me check