How to convert source to grayscale in deepstream?

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
3090ti
• DeepStream Version
6.2 triton (docker)

Hello

I am using a custom deepstream-test5 app for my research.

I want to convert the input video (mp4) to grayscale in the deepstream-test5 app and put it in pgie and receive the output in grayscale as well.

Can you edit the config file to enable this? Or, please tell me another way. Below is the config file

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=5
#gie-kitti-output-dir=streamscl

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI
type=2
uri=file://../../../../../samples/streams/sample_1080p_h264.mp4
num-sources=1
gpu-id=0
nvbuf-memory-type=0

[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File
type=2
sync=1
source-id=0
gpu-id=0
nvbuf-memory-type=0

[osd]
enable=1
gpu-id=0
border-width=1
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Arial
show-clock=0
clock-x-offset=800
clock-y-offset=820
clock-text-size=12
clock-color=1;0;0;0
nvbuf-memory-type=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=0
batch-size=4
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
## Set muxer output width and height
width=1920
height=1080
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0
## If set to TRUE, system timestamp will be attached as ntp timestamp
## If set to FALSE, ntp timestamp from rtspsrc, if available, will be attached
# attach-sys-ts-as-ntp=1

[primary-gie]
enable=1
gpu-id=0
batch-size=4
## 0=FP32, 1=INT8, 2=FP16 mode
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;1;1;1
bbox-border-color3=0;1;0;1
nvbuf-memory-type=0
interval=0
gie-unique-id=1
model-engine-file=../../../../../samples/models/Primary_Detector/resnet10.caffemodel_b4_gpu0_int8.engine
labelfile-path=../../../../../samples/models/Primary_Detector/labels.txt
config-file=../../../../../samples/configs/deepstream-app/config_infer_primary.txt
#infer-raw-output-dir=../../../../../samples/primary_detector_raw_output/

[tracker]
enable=1
# For NvDCF and NvDeepSORT tracker, tracker-width and tracker-height must be a multiple of 32, respectively
tracker-width=640
tracker-height=384
ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
# ll-config-file required to set different tracker types
# ll-config-file=../../../../../samples/configs/deepstream-app/config_tracker_IOU.yml
# ll-config-file=../../../../../samples/configs/deepstream-app/config_tracker_NvSORT.yml
ll-config-file=../../../../../samples/configs/deepstream-app/config_tracker_NvDCF_perf.yml
# ll-config-file=../../../../../samples/configs/deepstream-app/config_tracker_NvDCF_accuracy.yml
# ll-config-file=../../../../../samples/configs/deepstream-app/config_tracker_NvDeepSORT.yml
gpu-id=0
enable-batch-process=1
enable-past-frame=1
display-tracking-id=1
3

do you mean the model’s input is grayscale? not RGB? you can set model-color-format=2 in …/…/…/…/…/samples/configs/deepstream-app/config_infer_primary.txt, please find model-color-format in nvinfer

4

The input to the model is grayscale.

Therefore, I want to convert the RGB input image to grayscale.

5

please set model-color-format=2 in …/…/…/…/…/samples/configs/deepstream-app/config_infer_primary.txt to have a try.

6

Are you saying that setting “model-color-format=2” will automatically convert the RGB input source to greyscale into pgie’s input?

7

It says that you can convert the image to grayscale using “videobalance”, but I don’t know how to apply it, whether I need to modify the code or add it to the config file.

8

nvinfer plugin is opensource, please refer to get_converted_buffer in \opt\nvidia\deepstream\deepstream-6.2\sources\gst-plugins\gst-nvinfer\gstnvinfer.cpp, it can convert source to GRAY8 by NvBufSurfTransformAsync. please set model-color-format=2 to try.

9

thank you for kind reply!

Then, can I resize the input source in that part?

10

you only need to set right configuration in the configuration file, especially model-color-format and infer-dims should be correct, then nvinfer plugin will do the convertion and scaling. please refer to a configruation sample: opt\nvidia\deepstream\deepstream-6.2\sources\objectDetector_SSD\config_infer_primary_ssd.txt

1 Like
11

I’m really sorry, but if I have one more question

The custom resnet model I use needs to take 10 grayscale images as input.

Image conversion has been completed with your kind explanation, and now I want to save the converted images in the queue and push them to pgie by 10.

I can implement the queue structure, but I don’t know where to put the queue in the pipeline, so please advise.

12

you can add a nvpreprocess plugin before pgie, nvpreprocess can package multiple pictures as a tensor, then nvinfer will do inference on this tensor. please refer to \opt\nvidia\deepstream\deepstream-6.2\sources\apps\sample_apps\deepstream-3d-action-recognition, the model needs a sequence of pictures as input, especially nvinfer 's input-tensor-from-meta should be 1.

13

thank you!!!
have a nice day!!

14

I’m working with the app you recommended, but custom_sequence_preprocess is currently designed for rgb, so can I convert it to grayscale? if yes can you tell me how

15
  1. nvpreprocess also supports converting RGB to gray8, please find network-color-format in nvdspreprocess.
  2. custom_sequence_preprocess is opensource, you can customize it. NvDsPreProcessConvert_C1ToP1Float in NvDsPreProcessTensorImpl::prepare_tensor is used to do gray8 normalization.
16

did you meet any problems? fpenet model also needs gray8 input, please refer to its sample faciallandmark_sgie_config

17

thank you
I resolved!!