How to preprocess image in SGIE Deepstream 5.0?

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

Hello,

I’m trying to build a pipeline for vehicles detection, the pipeline look like this:
Vehicle Detection (PGIE) ----> Plate Status Classifier (SGIE1)
In order to get the plate status, the output of the PGIE (the vehicle) need to be cropped based on the bounding boxes coordination.

Scenario Example:

  1. We have this picture entering the piepline:
    22fc4d46266991.5607ca7211350

  2. After vehicle detection
    22fc4d46266991.5607ca7211350(1)

  3. Crop the vehicle before SGIE model
    22fc4d46266991.5607ca7211350(2)

  4. Classify plate status

How can I perform that? the preprocessing to the image?

Note: I can’t postprocess the vehicle detection model, because I have other SGIEs depending on it. so it has to be preprocessing between PGIE and SGIE

• Hardware Platform (Jetson / GPU): GPU
• DeepStream Version: 5.0
• JetPack Version (valid for Jetson only)
• TensorRT Version: 7.1
• NVIDIA GPU Driver Version (valid for GPU only): 440

3

I think SGIE will crop the image based on the bbox from pgie.

4

@bcao Yes true, the detected object will be cropped and sent to SGIE, but I want to do extra cropping to the detected object before entering SGIE

5

Is it possible to install a probe on the src pad of pgie and apply your extra cropping in that probe?

6

@bcao Can you give me more details on how to do that?

7

Are you familiar with gstreamer probe?

8

@bcao No, can you explain it?
Also would it be possible to perform the cropping through gstnvinfer.cpp?

9

Refer Probes , also you can find how to use probe inside deepstream for deepstream-test1.

10

Yes, you can check gstnvinfer.cpp-> gst_nvinfer_process_objects

11

@bcao I didn’t understand the code quite well.

I added this code inside gst_nvinfer_process_objects inside the objects iteration loop; to access the classifier

 if (object_meta->unique_component_id == 1)
     {
        if (object_meta->class_id == 0){
              NvOSD_RectParams rect_params = object_meta->rect_params;
         for (l_classifier = object_meta->classifier_meta_list; l_classifier != NULL;
               l_classifier = l_classifier->next)
          {
            class_meta = (NvDsClassifierMeta *)(l_classifier->data);
              if (class_meta->unique_component_id == 7) {
                  //some code
           }
        }
    }

but I still can’t figure out how to perform the preprocessing, what part and method is responsible for editing the image?

12

You can check convert_batch_and_push_to_input_thread

13

I checked it. I’m still confused about how to perform the necessary modifications.
Would you mind explaining it?

14

Yeah, you need to be familiar with NvBufSurfTransform , and also you can refer dsexample for how to use it