12 bit colour depth in deepstream gstreamer plugins

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) Jetson Xavier AGX
• DeepStream Version 5.0
• JetPack Version (valid for Jetson only) R32, Revision 4.3
• TensorRT Version 7.1.3
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs) question, new requirement
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Hello,

I am working on a project where a high quality machine vision camera is used. This camera delivers RGB images with 12 bit per colour channel. I would like to use deepstream for video analytics. A first review of deepstream’s gstreamer plugins has shown that only 8-bit colour depth is currently supported. Is there an easy way to feed at least the nvinfer plugin with 12-bit colour depth?

We have already trained and tested a pytorch model for our application in TensorRT. With 12-bit colour depth of the input image and float16 inference it delivers much better results than with 8-bit colour depth.

BR CRR

3

How do you get the 12bit per channel RGB output from the camera? What is the software interface for the camera?

4

It is a Gigabit Ethernet Vision camera. We have a custom gstreamer source producing RGBA64 raw images.

5

Unfortuenately, currently we can support 10bit video input with DeepStream, but not 12bit input.

6

ok, that would be ok for us.
Can you show me how to setup a 10bit pipeline in deepstream? It looks like,that nvinfer just supports RGBA 8-bit!?

me@me-desktop:~$ gst-inspect-1.0 nvinfer
...
Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw(memory:NVMM)
                 format: { (string)NV12, (string)RGBA }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]

  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw(memory:NVMM)
                 format: { (string)NV12, (string)RGBA }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
7

We use nvvideoconvert to convert NV12(P010_10LE) to NV12 before input to nvstreammux and nvinfer.

8

I see, so everything after nvvideoconvert works then only with 8-bit per color channel?
There is no way to use nvinfer with more than 8-bit per color channel?

This is surprising because TensorRT itself is not limited to 8-bit per colour channel.

9

TensorRT is not limited but nvinfer is limited because the feature has not been integrated.

10

Is there any plan to integrate this in future?

11

Currently, we don’t have such plan.