How to implement secondary GIE in order to conduct inference only in the cropped region from primary GIE bbox

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

Hardware Platform (Jetson / GPU) Jetson Xaxier
• DeepStream Version 6.1.0
• JetPack Version (valid for Jetson only)
• TensorRT Version 8.4
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs) Memory overload
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Hello, I am trying to implement a deepstream pipeline in each I have a primary and secondary GIE , The secondary GIE should take the cropped objects using bbox from the primary GIE as input and conduct inference on that cropped image. I have tried to get inspiration from
GitHub - preronamajumder/deepstream-lpr-python-version: Python version for NVIDIA Deepstream's LPR. https://developer.nvidia.com/blog/creating-a-real-time-license-plate-detection-and-recognition-app/ and GitHub - NVIDIA-AI-IOT/deepstream_lpr_app: Sample app code for LPR deployment on DeepStream, but I couldn’t find where the cropping of objects was done in those two examples.

From the description of the pipeline, it can be clearly seen that the detected car plate was cropped and given to the LPR secondary GIE.

Any suggestions or clarifications are warmly welcome , thanks

3

You can refer to our Guide about nvinfer below. Try to set gie-unique-id on pgie, and set the same id with operate-on-gie-id on sgie.
https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html

4

Thanks a lot this is very helpful,

My question is by doing so is the original image cropped and fed to the sgie or does the application preserve the original image size and conduct inference only inside the bbox?

Thank you

5

You can copy the bbox image data to another buffer and use it to the next inference.

6

Do you have a sample code that does something similar? By the way, I am using the python API.

7

What are your exact user case? Maybe you can refer our source code directly: sources\gst-plugins\gst-nvdspreprocess or sources\gst-plugins\gst-nvinfer. There is a detailed processing flow in the code. Thanks

8

We have a very classical use case. We want to detect workers in a construction site then crop the worker image within the bbox and feed it to the sgie to detect if they are wearing the right protection gear like (Helmet and Reflective vest).
We previously had a pipeline with only one GIE but the model failed to detect the gear when the worker are far away and thus very small.

9

You can change the primary object bbox parameters (NvDsObjectMeta rect_params) or create new objects (NvDsObjectMeta) with the required bbox and use them as primary objects.

10

You can refer to the link below. Just set the gie-unique-id para in your pgie cofing file. Then set the operate-on-gie-id para in your sgie config file.
https://github.com/NVIDIA-AI-IOT/deepstream_python_apps/tree/master/apps/deepstream-test2

11

This sounds to be the best solution, I will try that.

Thank you

12

@yuweiw

This seems to be working but how can I make sure the SGIE is operating on the cropped bbox image only? One way to do this is to save the image fed into the SGIE but how can I do that?

13

Please refer to our Guide:
https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html

The Gst-nvinfer plugin can work in three modes:
...

When you set the process-mode to 2, it will operate the object from previous nvinfer.

14

hi, I have tried to use gie-unique-id and operate-on-gie-id params in my PGIE and SGIE respectively but when I run the program i got the following error .

1681199706135
16811997061351439×189 51.1 KB

Surprisingly, this error comes after running for a while. It does not come directly just after launching the program.

Also I am not seeing any object being detected by the SGIE.

15

It means the scaled ratio between the origin and dest video is too large. Could you attach your demo app(models, videos, code, config files) to us? You can message all of that to me.

18

Hey ,

I am sending you the demo app, this is how you can run it.

  1. Extract the file
  2. run python3 ppe_app_demo.py --no-display -i /your-demo-dir/Tower.mp4

PS :

  1. I have already included the required lib in the project dir, so there is no need for any config.
  2. Our pipeline consists of one PGIE based on NVIDIA peopleNet and SGIE is a yolov5 model that detects helmet and reflective vest
  3. The rtps streaming is turned ON for visualization

demo.zip (62.9 MB)

Let me know if you have any issues with it.

Thank you.

19

You need to modify the open source code if you use the deepstream 6.1, please refer to the link below:
https://github.com/zhouyuchong/face-recognition-deepstream/issues/24
If you update the deepstream version to 6.2, you can just set the crop-objects-to-roi-boundary=1 in the config file.

20

Did you face the same issue while running the code ?

21

No. Because the engine file can only be used in your environment. If I want to use it in my jetson env, I need to use the origin model file to generate engine file again.
Based on previous experience, it should be the issue of bbox crossing the boundary.
You can also try to set the scaling-compute-hw=1 in your config file to use gpu instead of VIC.

1 Like
22

Hey
Thank you very much, setting scaling-compute-hw=1 fixes the issue. Now the pipeline is running fine.

23

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.