Can User convert Camera image in DeepStream?

[h3a1n7]

• Hardware Platform (Jetson / GPU)
Jetson Nano
• DeepStream Version
DeepStream 5.0 SDK
• JetPack Version (valid for Jetson only)
Jetpack 4.4.1
• TensorRT Version
7.1.3.0

Hello.

We made our application using Jetson Nano.

At the camera setting site, the camera rotates 90 degrees and gives input.

When we apply AI detection with this image, the performance is poor because the human rotates 90 degrees and comes in as input.

I want to rotate the camera image coming in as input by 90 before inferring it from DeepStream and reverse it when outputting it again.

Is there any way?

In DeepStream, can the user infer by converting the camera image (ex. rotating, resize)?

[DaneLLL]

Hi,
There is a sample of doing 180-degree rotation:

Jetson Nano CSI Raspberry Pi Camera V2 upside down video when run an example with deepstream-app

Hi, Please use NvBufSurface APIs. Below is a patch for reference: sources\apps\apps-common\src\deepstream_source_bin.c: @@ -22,6 +22,8 @@ #include <string.h> +#include "nvbufsurface.h" +#include "nvbufsurftransform.h" #include "gstnvdsmeta.h" #include "deepstream_common.h" #include "deepstream_sources.h" @@ -62,6 +64,64 @@ set_camera_v4l2_params (NvDsSourceConfig * config, NvDsSrcBin * bin) return TRUE; } +static GstPadProbeReturn +nvargus_src_pad_buffer_probe (GstPad * pad, Gst…

Since the resolution is the same after 180-degree rotation, it is a simple case.

90-degree rotation is a bit complicated. Possible solution is to run the pipeline:

nvarguscamerasrc ! 'video/x-raw(memory:NVMM),width=1920,height=1080' ! nvvideoconvert ! 'video/x-raw(memory:NVMM),width=1920,height=1920' ! nvvideoconvert ! 'video/x-raw(memory:NVMM),width=1080,height=1920' ! nvstreammux ! nvinfer ! ...

You can add prob callback to source pad of first nvvideoconvert plugin and do 90-degree rotation.

[h3a1n7]

Thank you DaneLLL,

If I somehow run PipeLine to convert the image, will it affect the inference speed?

[DaneLLL]

Hi,
The nvvideoconvert plugin leverages hardware converter. Should not have impact to doing inference.

[h3a1n7]

That’s right.

Sorry,
I mean the process speed when image conversion is performed using nvvideoconvert plugin.

One more question, can the user rotate the image 90 degrees to make an inference and get the image and information rotated 90 degrees back to the original as output?

[DaneLLL]

Hi,
Probably it is better to have the AI detection model trained from 90-degree rotated images. This should be easier to construct the pipeline.

[h3a1n7]

Yes that is correct.
It is a burden to learn the normal and rotated images separately.
So I wanted to integrate it into one non-rotated learning model.

[DaneLLL]

Hi,
Since the case is not supported by default, may need to check the source code and do customization.

The nvinfer plugin is open source, one more possible solution is to apply 90-degree rotation inside the plugin. The source code is in

/opt/nvidia/deepstream/deepstream-5.0/sources/gst-plugins/gst-nvinfer/