Affine transformation/warp on bbox

Accelerated Computing Intelligent Video Analytics DeepStream SDK
#1

• Hardware Platform: GPU
• DeepStream Version: 5.0.0
• TensorRT Version: 7.0.0.11
• NVIDIA GPU Driver Version (valid for GPU only): 460.32.03

Hi, I want to get something done that was discussed quite a few times in this forum but was never really answered or the provided answer was for an outdated deepstream version.

My pipeline looks something like this: pgie (detector, outputs bboxes) → sgie1 (network-type=100, output-tensor-meta=1; outputs landmarks which i parse and attach to the respective bbox meta) → sgie2

sgie2 expects a bbox that is aligned in respect to the landmarks from sgie1’s output. So the question is how to realize that?

I found the following topics covering this:

https://forums.developer.nvidia.com/t/image-pre-processing-between-pgie-and-sgie/111350/6

Here it is suggested to do an affine transformation on every bbox. I have no idea how this should be realized within deepstream API as a bbox is defined by it’s upper left corner as well as width and height, so it gives a rectangle but applying an affine transformation on a rectangle not necessarily yields a rectangle.

Next thing is that the piece of code which is suggested to be changed in /opt/nvidia/deepstream/deepstream-4.0/sources/libs/nvdsinfer/nvdsinfer_context_impl.cpp is not available anymore in my deepstream version.

https://forums.developer.nvidia.com/t/need-advice-image-pre-processing-between-pgie-and-sgie-custom-sgie-output-feature-vector/107444/4

This suggests to follow the dsexample plugin and to use NPP API. However I didn’t manage to find any example how to use the NPP API. Additionally it would be nice to see an example how to connect DS and NPP API. Maybe someone can point to a resource.

Appreciate any comment

#3

You can update/change the meta data (left corner as well as width and height) via installing the probe on sgie1 src pad in your case, then sgie2 will do infer on your updated bbox, you need to configure sgie2 do inference based on sgie1 through operate-on-gie-id. Referring Gst-nvinfer — DeepStream 5.1 Release documentation

#4

Let me show you what I want to achieve:

Say I have this input picture from the chancellor:

m
m4876×2984 2.48 MB

Now I run a detection model on it which returns this bbox:

m_bbox
m_bbox4876×2984 1.88 MB

Now I compute an affine transformation(just a rotation in this case) which maps the box in the above picture to this one:

m_bbox_warped
m_bbox_warped4876×2984 1.89 MB

Unfortunately the above picture cannot be mapped to DS API via left corner, width and height. However, I want to crop this box like this (Excuse my crappy free hand paint cutting)

m_bbox_warped_croped
m_bbox_warped_croped1243×1223 246 KB

and transform this into something I can feed into an nvinfer:

m_bbox_warped_croped_rectangle
m_bbox_warped_croped_rectangle1023×976 96.5 KB

I don’t see how this could be accomplished using left corner, width and height. You might be able to do it in OpenCV but I guess NPP API is also capable of that. Any idea how? I would like to stay in the NVIDIA stack and ideally handle all this or as much as possible on the GPU. Additionally I would probably need to not only change the rect_params fields in the probe callbacks but also some buffers I guess?

Any idea on how to solve this?

The source picture is taken from wikipedia: https://de.wikipedia.org/wiki/Datei:Angela_Merkel_-_World_Economic_Forum_Annual_Meeting_2011.jpg

#5

RIght, you can refer NVIDIA 2D Image And Signal Performance Primitives (NPP): Affine Transform

Yeah, in gstnvinfer.cpp , instead of calling NvBufSurfTransform inside convert_batch_and_push_to_input_thread, need to call affine transformation

Populate NvBufSurface from cv::Mat
Populate NvBufSurface from cv::Mat
#7

Could you deepen on that?

#9

NvBufSurfTransform and convert_batch_and_push_to_input_thread is two function in gstnvinfer.cpp, I mean you need to replace current NvBufSurfTransform with “affine transformation” you need to implement it yourself.