No text renders when passing rgba buffers to nvstreammux

Accelerated Computing Intelligent Video Analytics DeepStream SDK
#1

**• Jetson Nano Devkit
**• DeepStream 5.0.0
**• Jetpack 4.3
**• TensorRT 7.1.3
**• Bug

When we passing nv12 buffers to nvstreammux and after that converting them to rgba all is working as expected:

gst-launch-1.0 filesrc location=streams/sample_720p.h264 ! h264parse ! nvv4l2decoder ! "video/x-raw(memory:NVMM),format=NV12" ! m.sink_0 nvstreammux name=m batch-size=1 width=1280 height=720 ! nvvideoconvert ! "video/x-raw(memory:NVMM),format=RGBA" ! nvinfer config-file-path= configs/deepstream-app/config_infer_primary.txt batch-size=1 unique-id=1 ! nvdsosd ! nvegltransform ! nveglglessink

Снимок экрана_2021-01-19_11-56-46
Снимок экрана_2021-01-19_11-56-461269×708 212 KB

Buf if we have ->rgba conversion before nvstreammux no text renders

gst-launch-1.0 filesrc location=streams/sample_720p.h264 ! h264parse ! nvv4l2decoder ! nvvideoconvert ! "video/x-raw(memory:NVMM),format=RGBA" ! m.sink_0 nvstreammux name=m batch-size=1 width=1280 height=720 ! nvinfer config-file-path= configs/deepstream-app/config_infer_primary.txt batch-size=1 unique-id=1 ! nvdsosd ! nvegltransform ! nveglglessink

Снимок экрана_2021-01-19_11-57-23
Снимок экрана_2021-01-19_11-57-231275×714 209 KB

#3

Yes, I can reproduce the problem with the pipeline you list. We will investigate this problem.

#4

The reason of the text disappearance with RGBA format is because the alpha blending, in nvinfer plugin(please refer to the source code in /opt/nvidia/deepstream/deepstream-5.0/sources/gst-plugins/gst-nvinfer/gstnvinfer_meta_utils.cpp, in attach_metadata_detector() function (the same case with attach_metadata_classifier() function) , the text color is set as white and background color is black. Black is the same to the alpha 0, so the text area is transparent in the display. This will not happen with NV12 format because there is no alpha for YUV format.

The resolution is to set background color to other colors (even (0.001, 0.001 , 0.001, 1) can work), or you can also disable the text background with setting “text_params.set_bg_clr = 0;”.

The nvinfer is open source for customization, you can modify the code and build the new gstnvinfer plugin for your case.

There is also another method to show text with RGBA format without modifying gstnvinfer plugin. You can write your own deepstream app just as our deepstream samples https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_C_Sample_Apps.html#c-c-sample-apps-source-details, with deepstream app, you can change the display color and text color with pad probe function.

#5

I moved nvvideoconvert before nvstreammux, set caps and changed background color in osd_sink_pad_buffer_probe function in deepstream-test-app1, but no success.

deepstream_test1_app.c (11.3 KB)

#6

Seems this is a nvdsosd problem. The following pipeline will cause the bbox disappear.

gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream-5.0/samples/streams/sample_720p.h264 ! h264parse ! nvv4l2decoder ! nvvideoconvert ! “video/x-raw(memory:NVMM),format=RGBA” ! m.sink_0 nvstreammux name=m batch-size=1 width=1280 height=720 ! nvinfer config-file-path=/opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/config_infer_primary.txt batch-size=1 unique-id=1 ! nvdsosd process-mode=0 ! nvegltransform ! nveglglessink

We will investigate into this issue.