colorFormat of NvBufSurface and its default value

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) dGPU: Tesla T4
• DeepStream Version 5.0.2
• JetPack Version (valid for Jetson only) N/A
• TensorRT Version 7.1.3.4
• NVIDIA GPU Driver Version (valid for GPU only) 450.51.06
• Issue Type( questions, new requirements, bugs) questions
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing) N/A
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description) N/A

In our application we are feeding YUV NV12 buffers via appsrc to a pipeline containing nvinfer and other necessary elements. The buffer is loaded into GstBuffer created by gst_buffer_new_and_alloc, and supposed to live in system memory. It is then converted by nvvideoconvert to provide caps specified as video/x-raw(memory:NVMM),format=NV12,width=1920,height=1080,framerate=1/0 before pushed to nvstreammux.

Currently the YUV buffer is direct dump from output of nvv4l2decoder, which marks colorFormat as NVBUF_COLOR_FORMAT_NV12_709 or NVBUF_COLOR_FORMAT_NV12 depending on different rtsp streams as input. However the previously mentioned nvvideoconvert always produces NVBUF_COLOR_FORMAT_NV12_709 surface.

Is there any chance for specifying the input color format somewhere, or the conversion output color format is consistant and reliable behavior?

3

Hi,
So you mean in the pipeline:

appsrc ! video/x-raw,format=NV12 ! nvvideoconvert ! video/x-raw(memory:NVMM),format=NV12

The video/x-raw(memory:NVMM),format=NV12 buffer is always NVBUF_COLOR_FORMAT_NV12_709 and cannot be configured? Please confirm it.

4

Hi DaneLLL,

Basically yes, we tested both DS 4.0 and 5.0.2. The input buffer into appsrc is a GstBuffer of 1920*1620 bytes without special color format annotation.
In the mentioned pipeline
appsrc ! video/x-raw,format=NV12 ! nvvideoconvert ! video/x-raw(memory:NVMM),format=NV12 ! identity
probing at the identity we’ve got mapped NvBufSurface with colorFormat = 33 (NVBUF_COLOR_FORMAT_NV12_709).

Maybe there could be something like colorformat=xxx in the caps to configure it?

5

Hi,
Please refer to the patch:

diff --git a/apps/deepstream/sample_apps/deepstream-appsrc-test/deepstream_appsrc_test_app.c b/apps/deepstream/sample_apps/deepstream-appsrc-test/deepstream_appsrc_test_app.c
index eb34308..7681b2d 100644
--- a/apps/deepstream/sample_apps/deepstream-appsrc-test/deepstream_appsrc_test_app.c
+++ b/apps/deepstream/sample_apps/deepstream-appsrc-test/deepstream_appsrc_test_app.c
@@ -27,6 +27,7 @@
 #include <stdio.h>
 #include <string.h>
 #include "gstnvdsmeta.h"
+#include "nvbufsurface.h"
 
 #define MAX_DISPLAY_LEN 64
 
@@ -206,6 +207,27 @@ stop_feed (GstElement * source, AppSrcData * data)
   }
 }
 
+static GstPadProbeReturn
+conv1_src_pad_buffer_probe (GstPad * pad, GstPadProbeInfo * info,
+    gpointer u_data)
+{
+    GstBuffer *buf = (GstBuffer *) info->data;
+    GstMapInfo in_map_info;
+    NvBufSurface *surface = NULL;
+
+    memset (&in_map_info, 0, sizeof (in_map_info));
+    if (!gst_buffer_map (buf, &in_map_info, GST_MAP_READ)) {
+      g_print ("Error: Failed to map gst buffer\n");
+      gst_buffer_unmap (buf, &in_map_info);
+      return GST_PAD_PROBE_OK;
+    }
+    surface = (NvBufSurface *) in_map_info.data;
+    g_print("colorFormat=%d, [%d %d] \n", surface->surfaceList[0].colorFormat, NVBUF_COLOR_FORMAT_NV12, NVBUF_COLOR_FORMAT_NV12_709);
+
+    gst_buffer_unmap (buf, &in_map_info);
+    return GST_PAD_PROBE_OK;
+}
+
 int
 main (int argc, char *argv[])
 {
@@ -375,9 +397,19 @@ main (int argc, char *argv[])
   g_object_set (G_OBJECT (nvvidconv1), "nvbuf-memory-type", 3, NULL);
 #endif
 
+  GstPad *conv1_src_pad = NULL;
+  conv1_src_pad = gst_element_get_static_pad (nvvidconv1, "src");
+  if (!conv1_src_pad)
+    g_print ("Unable to get src pad\n");
+  else
+    gst_pad_add_probe (conv1_src_pad, GST_PAD_PROBE_TYPE_BUFFER,
+        conv1_src_pad_buffer_probe, NULL, NULL);
+  gst_object_unref (conv1_src_pad);
+
   caps =
       gst_caps_new_simple ("video/x-raw", "format", G_TYPE_STRING,
-      vidconv_format, NULL);
+      vidconv_format,
+      "colorimetry", G_TYPE_STRING, "bt601", NULL);
   feature = gst_caps_features_new ("memory:NVMM", NULL);
   gst_caps_set_features (caps, 0, feature);
   g_object_set (G_OBJECT (caps_filter), "caps", caps, NULL);

Setting the caps colorimetry=bt601 works in patching deepstream-appsrc-test. Please apply it to your usecase and try again.

6

Thanks DaneLLL, that’s great help.

It seems working if specified ad the output side of nvvideoconvert, the observed NvBufSurface has colorFormat=6.

We’ll take some experiments to see if it does change the pixel value. Will update later.

The remaining questions are:

  • Does the colorimetry entry works at the input side of nvvideoconvert?
  • Guess the possible values are bt601, bt709, bt2020?
7

Hi,

You only need to configure it in the case:

appsrc ! video/x-raw ! nvvideoconvert ! video/x-raw(memory:NVMM)

And in the case, please configure the source pad of nvvideoconvert.

bt601 and bt709 are supported. bt2020 is not supported.
The two types are with range [16, 235]. If your source is in extension range [0, 255]. Please set

colorimetry="1:4:0:0" // bt601 extension range
colorimetry="1:3:0:0" // bt709 extension range

For detail about colorimetry, you can take a look at
https://github.com/GStreamer/gst-plugins-base/blob/master/gst-libs/gst/video/video-color.h

8

Thanks! That successfully changed the pixel values in buffer.

Here is a simple script to test it:

gst-launch-1.0 videotestsrc pattern=gamut num-buffers=1 ! video/x-raw,width=20,height=16,format=NV12,colorimetry=bt709 ! identity dump=1 ! nvvideoconvert ! 'video/x-raw(memory:NVMM),format=NV12,colorimetry=bt601' ! nvvideoconvert ! 'video/x-raw' ! identity dump=1 ! fakesink

Edit:

Works in DS 5.0.1 container.

Seems it does not work in DS 4.0.2. The script runs but failed to actually change pixel values.