Preprocessing between pgie and sgie

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
• DeepStream Version
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)

Hi team,

I’m using the pgie-sgie detection network in deepstream. Basically, the pgie reads in the image and do the primary detection, and then passes the cropped image to sgie to continue the detection based on the pgie’s output.
My problem is how can I do the preprocessing before the sgie after pgie. I need to transpose the input image before the sgie network, but default preprocessing functions only include normalization and subtraction.
I checked the code do the normalization in “nvdsinfer_context_impl.cpp”, it processes on the first layer of the network. But I need to do the transpose on the input image before the image been resized to the network input size.
Any help would be appreciated, thanks.

3

@kobe2410811

What kind of sgie model are you using? pytorch? tensorflow? or caffe?
I think you can add a transpose layer into your sgie model to transpose input before deploying the model onto DeepStream 5.0

4

@ersheng:
I am facing the same problem. I need to rotate the detected object a little bit before transferring to the secondary gie. Could you please show me the source code where the frame is cropped, so maybe I can customize it?
Thank you very much!

1 Like
5

Thanks for the reply.
I’m using caffe model as sgie.
That might be a way, but is there a transpose layer in caffe?
I’m not very familiar to caffe, and I didn’t find a transpose layer in caffe layer parameters.

6

@kobe2410811

This is an example of how a caffe transpose layer may look like:

layer {
  name: "conv9_2_mbox_conf_perm"
  type: "Permute"
  bottom: "conv9_2_mbox_conf"
  top: "conv9_2_mbox_conf_perm"
  permute_param {
    order: 0
    order: 2
    order: 3
    order: 1
  }
}
7

Do I need to retrain model if I add a permute layer?

8

@kobe2410811

No, you don’t have to.

You can try to add this layer to prototxt and use the same caffemodel. Don’t change names of other layers.

9

Thanks, I got the way to do it and now it can be flipped before conv layer.
The order of the param should be
permute_param {
order: 0
order: 1
order: 3
order: 2
}
Thanks for your help, but I’m still wondering if there’s a way to directly preprocess on the image in case user needs to do some complex process steps.
I notice there’s a code snippet sort of similar with this: Deepstream sample code snippet
In this code, image can be extracted from a sink pad before pgie, but I didn’t quite understand the surface & opencv part and how does a image be inserted back to buffer after extracted to preprocess.

10

@kobe2410811

I think this issue is solved.
You can open a new topic on forum to discuss your new concerns or more complex scenarios.

11

Yeah, I’ve created a new topic here: Image preprocessing before nvinfer

12

Sorry to restart this topic again.
Previously I tested it in opencv and it works good if I add a transpose layer in prototxt.
But now I found it still not works good in deepstream, where deepstream prompts an error when I try to create a trt engine.
The error is here:

err
err1015×53 19.2 KB

It shows that the volume is not matched.
Appreciate if you could help me with that.

13

@kobe2410811

It seems this error is reported from TensorRT.
You can try the following command to check if TensorRT can handle the modified prototxt properly. trtexec is located in /usr/src/tensorrt/bin

trtexec --deploy=<prototxt_file> --model=<caffemodel_file> --output=<output_blob_names_separated_by_comma> --saveEngine=<engine_file>
14

I tried tensorrt and the engine can be created using trtexec.
And now deepstream can load the engine generated by trt.
It’s very weird that the error was reported when I tried to create engine directly from deepstream.
Thanks for help!

16

hi, the issue sloved? iam deploying face recognition, so i need to transpose image before infer it on sgie. how can i do it?

17

Hi phuong1998bn,

Please help to open a new topic for your issue. Thanks