How to add black and white filter in pipeline using videobalance

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

• Hardware Platform (Jetson / GPU)
Jetson Orin
• DeepStream Version
6.1.1
• JetPack Version (valid for Jetson only)
5.0.2
• TensorRT Version
8.4.1-1+cuda11.4

Hi, I have a model that was trained with greyscale images with 3 channels… I tried setting model-color-format=2 in my detector config but the application fails.

Error: gst-resource-error-quark: Failed to create NvDsInferContext instance (1): /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp(846): gst_nvinfer_start (): /GstPipeline:pipeline0/GstNvInfer:secondary2-nvinference-engine:
Config file path: config_infer_third.txt, NvDsInfer Error: NVDSINFER_CONFIG_FAILED
Exiting app

How can I use videobalance to convert the image to black and white by setting the saturation to 0.0?
https://gstreamer.freedesktop.org/documentation/videofilter/videobalance.html?gi-language=c
I tried:

videobalance = Gst.ElementFactory.make("videobalance","videobalance")
videobalance.set_property('saturation', 0.0)
pipeline.add(videobalance)

print("Creating pre filter1 \n ")
caps1_gray  = Gst.Caps.from_string("video/x-raw(memory:NVMM), format=I420")
#filter1 = Gst.ElementFactory.make("capsfilter", "filter1")
filter1_gray  = Gst.ElementFactory.make("capsfilter", "filter1_gray")
if not filter1_gray:
    sys.stderr.write(" Unable to get the caps filter1_gray \n")
filter1_gray.set_property("caps", caps1_gray)
pipeline.add(filter1_gray)

nvvidconvert_gray = Gst.ElementFactory.make("nvvideoconvert", "nvvidconvert_gray")
if not nvvidconvert_gray:
    sys.stderr.write(" Unable to create nvvidconvert_pre \n")
pipeline.add(nvvidconvert_gray)
...
...
pgie.link(tracker)
tracker.link(nvvidconvert_pre)
nvvidconvert_pre.link(filter1_pre)
filter1_pre.link(sgie1)
#sgie1.link(sgie2)
sgie1.link(videobalance)
videobalance.link(nvvidconvert_gray)
nvvidconvert_gray.link(filter1_gray)
filter1_gray.link(sgie2)

0:00:11.021672980 54491     0x2a5ac4c0 WARN                 nvinfer gstnvinfer.cpp:2300:gst_nvinfer_output_loop:<secondary1-nvinference-engine> error: Internal data stream error.
0:00:11.021716052 54491     0x2a5ac4c0 WARN                 nvinfer gstnvinfer.cpp:2300:gst_nvinfer_output_loop:<secondary1-nvinference-engine> error: streaming stopped, reason not-linked (-1)
Error: gst-stream-error-quark: Internal data stream error. (1): /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp(2300): gst_nvinfer_output_loop (): /GstPipeline:pipeline0/GstNvInfer:secondary1-nvinference-engine:
streaming stopped, reason not-linked (-1)
Exiting app
3

Can you provide the complete config file? The failure is not caused by model-color-format=2.

Why do you want to add “videobalance”?

4

Could it be because model-color-format=2 (Gray) is only for single-channel gray images?

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
model-color-format=0
custom-network-config=gie3/yolov4-tiny.cfg
model-file=gie3/yolov4-tiny.weights
model-engine-file=gie3/model_b16_gpu0_fp32.engine
#int8-calib-file=calib.table
labelfile-path=gie3/labels.txt
batch-size=16
network-mode=0
num-detected-classes=36
interval=0
gie-unique-id=3

operate-on-gie-id=2

process-mode=2
network-type=0
cluster-mode=2
maintain-aspect-ratio=0
parse-bbox-func-name=NvDsInferParseYolo
custom-lib-path=gie3/nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so
engine-create-func-name=NvDsInferYoloCudaEngineGet

[class-attrs-all]
nms-iou-threshold=0.45
pre-cluster-threshold=0.25
topk=300

I want to change the image to black and white for my third detector since my model was built with training images in black and white in 3 channels.

5

No. model-color-format=2 means the model needs gray input. gst-nvinfer will convert the video/image data to gray according to this parameter.

Where did you get the yolov4 model? Can you show us the input layer(s)’ dimension?

You don’t need to do this. The gst-nvinfer can do preprocess inrenally. Just fill correct information about your model in the configuration file, gst-nvinfer will handle the color space transformation. DeepStream SDK FAQ - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums

6

Config:

[net]
# Testing
#batch=1
#subdivisions=1
# Training
batch=64
subdivisions=16

width=224
height=96
channels=3
momentum=0.9
decay=0.0005
angle=0
saturation = 1.5
exposure = 1.5
hue=.1

learning_rate=0.00261
burn_in=1000
max_batches = 72000
policy=steps
steps=57600,64800
scales=.1,.1

[convolutional]
batch_normalize=1
filters=32
size=3
stride=2
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=64
size=3
stride=2
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=64
size=3
stride=1
pad=1
activation=leaky

[route]
layers=-1
groups=2
group_id=1

[convolutional]
batch_normalize=1
filters=32
size=3
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=32
size=3
stride=1
pad=1
activation=leaky

[route]
layers = -1,-2

[convolutional]
batch_normalize=1
filters=64
size=1
stride=1
pad=1
activation=leaky

[route]
layers = -6,-1

[maxpool]
size=2
stride=2

[convolutional]
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=leaky

[route]
layers=-1
groups=2
group_id=1

[convolutional]
batch_normalize=1
filters=64
size=3
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=64
size=3
stride=1
pad=1
activation=leaky

[route]
layers = -1,-2

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[route]
layers = -6,-1

[maxpool]
size=2
stride=2

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[route]
layers=-1
groups=2
group_id=1

[convolutional]
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=leaky

[route]
layers = -1,-2

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[route]
layers = -6,-1

[maxpool]
size=2
stride=2

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

##################################

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

[convolutional]
size=1
stride=1
pad=1
# (classes + 5)*3
filters=123
activation=linear



[yolo]
mask = 3,4,5
anchors = 10,14,  23,27,  37,58,  81,82,  135,169,  344,319
classes=36
num=6
jitter=.3
scale_x_y = 1.05
cls_normalizer=1.0
iou_normalizer=0.07
iou_loss=ciou
ignore_thresh = .7
truth_thresh = 1
random=1
resize=1.5
nms_kind=greedynms
beta_nms=0.6

[route]
layers = -4

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[upsample]
stride=2

[route]
layers = -1, 23

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[convolutional]
size=1
stride=1
pad=1
filters=123
activation=linear

[yolo]
mask = 0,1,2
anchors = 10,14,  23,27,  37,58,  81,82,  135,169,  344,319
classes=36
num=6
jitter=.3
scale_x_y = 1.05
cls_normalizer=1.0
iou_normalizer=0.07
iou_loss=ciou
ignore_thresh = .7
truth_thresh = 1
random=1
resize=1.5
nms_kind=greedynms
beta_nms=0.6

I get it, so why model-color-format=2 gives me error?

7

This yolov4 input has 3 channels, so it should be RGB or BGR, it could not be gray image. Your setting is wrong.

8

How can I convert the image to a gray image with 3 channels? I have accomplished this with openCV how can I accomplish this with deepstream?

9

Why do you need this? Is your original input video/image the GRAY color format?

10

I just need the image to be in black and white for my third detector. is this possible?

11

Is the yolov4 model the third detector? Please tell us what are your PGIE, SGIE1, SGIE2 models’ inputs?

12

All my detectors are yolov4 - 3 channels detectors.
Is it possible to have a black and white image in for my third detector?

13

yolov4 need 3 channels input data, so the input should be RGB/BGR/… data, why do you need GRAY image? Is the third yolov4 model trained by “black and white 3 channels” images? GRAY image has only one channel.

14

Yes.

It has happened to me before that my model would have trouble detecting images in 3-channel color. So I converted the image to 3-channel black and white image and it worked fine. I just want to see if the accuracy is better by doing this small change. Thanks for your help I jus need to know if:
Is it possible to have a black and white 3-channel image for my third detector?

15

Yes. You need to convert the color image to “black and white” image by yourself.
In the pipeline, after nvstreammux, all data are in batched frames (NvBufSurface NVIDIA DeepStream SDK API Reference: NvBufSurface Struct Reference | NVIDIA Docs). You need to implement the conversion algorithm on the batched video frames with customizing the code in nvdsexample plugin or videotemplate plugin. There are samples of how to get NvBufsurface inside the nvdsexample and videotemplate sample codes.

1 Like
16

Thanks, @Fiona.Chen for guiding me in the right direction. However, I am new to modifying deepstream plugins and I am having trouble understanding where to modify nvdsexample plugin.

I already compiled nvdsexample plugin with opencv:=1 and added it to my pipeline and tested the blurring:

#preprocessing  black and white 3 channels
preprocessing = Gst.ElementFactory.make("dsexample", "pre-processing")
preprocessing.set_property('processing-width', 640)
preprocessing.set_property('processing-height', 480)
preprocessing.set_property('full-frame', 0)
preprocessing.set_property('unique-id', 15)
preprocessing.set_property('gpu-id', 0)
preprocessing.set_property('blur-objects', 1)

pipeline.add(q)
pipeline.add(pgie)
pipeline.add(tracker)
pipeline.add(sgie1)

pipeline.add(preprocessing)

pipeline.add(sgie2)
pipeline.add(nvstreamdemux)
pipeline.add(nvvidconvert_pre)
pipeline.add(filter1_pre)
pipeline.add(queue1)
pipeline.add(queue2)
pipeline.add(tee)
pipeline.add(msgconv)
pipeline.add(msgbroker)
  
streammux.link(q)
q.link(pgie)
pgie.link(tracker)
tracker.link(nvvidconvert_pre)
nvvidconvert_pre.link(filter1_pre)
filter1_pre.link(sgie1)

sgie1.link(preprocessing)
preprocessing.link(sgie2)

sgie2.link(tee)
queue1.link(msgconv)
msgconv.link(msgbroker)
queue2.link(nvstreamdemux)

As you can see I am adding the nvdsexample plugin after sgie1 I would like to only blur the images being processed in sgie2 (similar to what I am trying to achieve) but all detectors are being blurred including pgie , sgie1 and sgie2. I tried setting the property full-frame = 1 but I get a segmentation fault.

  1. Why are all detectors getting blurred if I added it after sgie1?

  2. Where I can start modifying the nvdsexample code to convert the image to black and white 3 channels?

Edit:
I am trying to modify the blur_objects function, it compiles without errors but no change is made in the image. code below:

#ifdef WITH_OPENCV
/*
 * Blur the detected objects when processing in object mode (full-frame=0)
 */
static GstFlowReturn
blur_objects (GstDsExample * dsexample, gint idx,
    NvOSD_RectParams * crop_rect_params, cv::Mat in_mat)
{
  cv::Rect crop_rect;

  if ((crop_rect_params->width == 0) || (crop_rect_params->height == 0)) {
    GST_ELEMENT_ERROR (dsexample, STREAM, FAILED,
        ("%s:crop_rect_params dimensions are zero",__func__), (NULL));
    return GST_FLOW_ERROR;
  }

/* rectangle for cropped objects */
  crop_rect = cv::Rect (crop_rect_params->left, crop_rect_params->top,
  crop_rect_params->width, crop_rect_params->height);

/* apply gaussian blur to the detected objects */
  //GaussianBlur(in_mat(crop_rect), in_mat(crop_rect), cv::Size(15,15), 4);
  cv::Mat greyMat1;
  cv::Mat greyMat2;
  cv::Mat greyMat3;
  cv::Mat out;
  

  cv::cvtColor(in_mat(crop_rect), greyMat1, cv::COLOR_BGR2GRAY);
  cv::cvtColor(in_mat(crop_rect), greyMat2, cv::COLOR_BGR2GRAY);
  cv::cvtColor(in_mat(crop_rect), greyMat3, cv::COLOR_BGR2GRAY);
  
  cv::Mat in[3] = {greyMat1,greyMat2,greyMat3};
  cv::merge(in, 3, out);
  in_mat(crop_rect) = out.clone();

  return GST_FLOW_OK;
}
#endif
17

Have you seen the following code in gst_dsexample_start() function?

dsexample->cvmat = new cv::Mat (dsexample->processing_height, dsexample->processing_width, CV_8UC3, dsexample->host_rgb_buf, dsexample->processing_width * RGB_BYTES_PER_PIXEL);

dsexample can only handle RGB data. So you must convert the video data to RGBA before dsexample.

18

Please make sure you are familiar with GStreamer knowledge and coding skills before start with DeepStream,

Please read the DeepStream document to understand the features and usages of all DeepStream plugins and APIs. There are many details which need your investigation. Welcome to the DeepStream Documentation — DeepStream 6.1.1 Release documentation

19

@Fiona.Chen please look closely at my pipeline I am using nvvideoconvert plugin to convert the video data to RGBA before sgie1 and using dsexample after sgie1.

nvvidconvert_pre = Gst.ElementFactory.make("nvvideoconvert", "nvvidconvert_pre")
if not nvvidconvert_pre:
    sys.stderr.write(" Unable to create nvvidconvert_pre \n")
  
print("Creating pre filter1 \n ")
caps1_pre  = Gst.Caps.from_string("video/x-raw(memory:NVMM), format=RGBA")
#filter1 = Gst.ElementFactory.make("capsfilter", "filter1")
filter1_pre  = Gst.ElementFactory.make("capsfilter", "filter1_pre")
if not filter1_pre:
    sys.stderr.write(" Unable to get the caps filter1_pre \n")
filter1_pre.set_property("caps", caps1_pre)
20

@Fiona.Chen Thanks I will open a new topic.

21

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.