How to enable asymetric padding in deepstream?

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU). Nvidia T4 GPU
• DeepStream Version - 5.1 and 6.0
• JetPack Version (valid for Jetson only) –
• TensorRT Version. 8.0.1-1+cuda11.3
• NVIDIA GPU Driver Version (valid for GPU only) 460.32.03
• Issue Type( questions, new requirements, bugs) - QUESTION
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

In Gst-nvinfer — DeepStream 6.1.1 Release documentation there is a description of symetric-padding option in inference-engine configuration:

Indicates whether to pad image symmetrically while scaling input. DeepStream pads the images asymmetrically by default.

This suggests that there is a way to pad images asymmetrically in deepstream - I imagine bottom-right padding is the default. Unfortunately running this pipeline:

gst-launch-1.0 nvstreammux name=mux batch-size=2 width=608 height=608 ! queue ! nvinfer config-file-path=/mounted/data/models/yolo/yolo_gie_config_fp32_bs_1_608x608.txt ! queue ! nvstreamdemux name=demux \
    uridecodebin uri=file:///mounted/data/videos/horizontal.mov ! nvvideoconvert ! queue ! mux.sink_0 \
    uridecodebin uri=file:///mounted/data/videos/vertical.mov ! nvvideoconvert ! queue ! mux.sink_1 \
    demux.src_0 ! queue ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=/mounted/data/images/deepstream-streammux-set/src_0/horizontal.mp4 \
    demux.src_1 ! queue ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=/mounted/data/images/deepstream-streammux-set/src_1/vertical.mp4

With config:

[property]
gpu-id=0
num-detected-classes=2
net-scale-factor=0.00392156862745098
batch-size=2
model-engine-file=/mounted/data/models/yolo/mmdet/yolo_fp32_bs_2_608x608_trt.engine
labelfile-path=/mounted/data/models/yolo/labels.txt
gie-unique-id=1
operate-on-gie-id=1
interval=0
### Integer 0: FP32 1: INT8 2: FP16
network-mode=0
offsets=0.0;0.0;0.0
output-blob-names=num_detections;boxes;scores;classes
parse-bbox-func-name=NvDsInferParseMmdet
custom-lib-path=/root/space/amirstan_plugin/build/lib/libamirstan_plugin.so
### Integer 0: RGB 1: BGR 2: GRAY
model-color-format=0
### Integer 1=Primary 2=Secondary
process-mode=1
### Integer 0: Detector 1: Classifier 2: Segmentation 3: Instance Segmentation
network-type=0
maintain-aspect-ratio=1
symmetric-padding=0  - does not work for some reason
infer-dims=3;608;608
### Integer 0: OpenCV groupRectangles() 1: DBSCAN 2: Non Maximum Suppression 3: DBSCAN + NMS Hybrid 4: No clustering
cluster-mode=4
scaling-filter=1
### Integer 0: Platform default – GPU (dGPU), VIC (Jetson) 1: GPU 2: VIC (Jetson only)
scaling-compute-hw=1
### Integer 0:NCHW 1:NHWC
network-input-order=0
### Dumb parameters which shouldn't matter but they do
workspace-size=500
classifier-threshold=0.1

[class-attrs-all]
### threshold=0.1  - deprecated
pre-cluster-threshold=0.1
post-cluster-threshold=0.1
nms-iou-threshold=1.0
## disbles clustering whatsoever
group-threshold=0
## 0 disables the clustering functionality
minBoxes=0
detected-min-w=16
detected-min-h=16
detected-max-w=0
detected-max-h=0
topk=100
roi-top-offset=0
roi-bottom-offset=0

Unfortunately, symmetric-padding flag does not work. The output is:
Unknown or legacy key specified 'symmetric-padding' for group [property]
and the output video is squashed instead of padded. when enable-padding flag is enabled on streammux element the padding is done symetrically. Is there a way to do it asymmetrically in deepstream ?

• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

3

nvinfer is source code release. Can you have a check the source code?

4

Hi, I have checked out the source code - it seems that container nvcr.io/nvidia/deepstream:6.0-ea-21.06-devel does not support this flag but the proper deepstream-6.0 container - nvcr.io/nvidia/deepstream:6.0-devel does. I have tried to switch this flag on and off in infer_config.txt but it does not seem to have any effect.

The question persists - how to enable asymmetric padding in deepstream ?

5

Can you add some log to check why it does not any effect?

6

How should I do that in the case of deepstream - should I export the variable GST_DEBUG=3 ?

7

Suppose you have the nvinfer source code. Can you add some log in the source code?

8

I have read the source code of nvinfer plugin. Seems that input-tensor-meta variable set to TRUE (by default) blocks any changes to images in buffer. When trying to set input-tensor-meta to FALSE or 0 deepstream prints out:

Unknown or legacy key specified 'input-tensor-meta' for group [property]

Which is not in line with documentation of nvinfer element - Gst-nvinfer — DeepStream 6.1.1 Release documentation

Can you tell me how to enable asymmetric padding in deepstream instead of reading source-code?

This is my pipeline inside of nvcr.io/nvidia/deepstream:6.0-devel container which you can easily replicate on your side (all needed resources are already inside of this container) :

 gst-launch-1.0 nvstreammux name=mux batch-size=1 width=608 height=342 ! queue ! nvinfer config-file-path=/mounted/my_config.txt input-tensor-meta=0 batch-size=1 ! queue ! nvstreamdemux name=demux \
    uridecodebin uri=file:///opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_1080p_h264.mp4 ! nvvideoconvert ! queue ! mux.sink_0 \
    demux.src_0 ! queue ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=/mounted/data/images/deepstream-streammux-set/src_0/horizontal.mp4

my_config.txt :

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
model-file=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Primary_Detector/resnet10.caffemodel
proto-file=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Primary_Detector/resnet10.prototxt
model-engine-file=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_int8.engine
labelfile-path=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Primary_Detector/labels.txt
int8-calib-file=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Primary_Detector/cal_trt.bin
force-implicit-batch-dim=1
batch-size=1
network-mode=1
num-detected-classes=4
interval=0
infer-dims=3;608;608
symmetric-padding=1
gie-unique-id=1
output-blob-names=conv2d_bbox;conv2d_cov/Sigmoid
cluster-mode=2
input-tensor-meta=0

[class-attrs-all]
pre-cluster-threshold=0.2
topk=20
nms-iou-threshold=0.5

Output from the pipeline:

root@9329ce241db4:/mounted#  gst-launch-1.0 nvstreammux name=mux batch-size=1 width=608 height=342 ! queue ! nvinfer config-file-path=/mounted/weird_config.txt input-tensor-meta=0 batch-size=1 ! queue ! nvstreamdemux name=demux \
>     uridecodebin uri=file:///opt/nvidia/deepstream/deepstream-6.0/samples/streams/sample_1080p_h264.mp4 ! nvvideoconvert ! queue ! mux.sink_0 \
>     demux.src_0 ! queue ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=/mounted/data/images/deepstream-streammux-set/src_0/horizontal.mp4

(gst-plugin-scanner:14): GStreamer-WARNING **: 12:26:22.224: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_inferserver.so': libtritonserver.so: cannot open shared object file: No such file or directory

(gst-plugin-scanner:14): GStreamer-WARNING **: 12:26:22.233: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_udp.so': librivermax.so.0: cannot open shared object file: No such file or directory
Unknown or legacy key specified 'input-tensor-meta' for group [property]
Setting pipeline to PAUSED ...
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:1484 Deserialize engine failed because file path: /opt/nvidia/deepstream/deepstream-6.0/samples/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_int8.engine open error
0:00:02.662365186    13 0x55eff8ba8160 WARN                 nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<nvinfer0> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1889> [UID = 1]: deserialize engine from file :/opt/nvidia/deepstream/deepstream-6.0/samples/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_int8.engine failed
0:00:02.662424770    13 0x55eff8ba8160 WARN                 nvinfer gstnvinfer.cpp:635:gst_nvinfer_logger:<nvinfer0> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1996> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/deepstream/deepstream-6.0/samples/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_int8.engine failed, try rebuild
0:00:02.662464525    13 0x55eff8ba8160 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<nvinfer0> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1914> [UID = 1]: Trying to create engine from model files
WARNING: [TRT]: Detected invalid timing cache, setup a local cache instead
0:00:14.212304120    13 0x55eff8ba8160 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<nvinfer0> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1947> [UID = 1]: serialize cuda engine to file: /opt/nvidia/deepstream/deepstream-6.0/samples/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_int8.engine successfully
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:610 [Implicit Engine Info]: layers num: 3
0   INPUT  kFLOAT input_1         3x608x608
1   OUTPUT kFLOAT conv2d_bbox     16x38x38
2   OUTPUT kFLOAT conv2d_cov/Sigmoid 4x38x38

0:00:14.216990067    13 0x55eff8ba8160 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<nvinfer0> [UID 1]: Load new model:/mounted/weird_config.txt sucessfully
Pipeline is PREROLLING ...
WARNING: from element /GstPipeline:pipeline0/GstNvStreamMux:mux: Rounding muxer output height to the next multiple of 4: 344
Additional debug info:
gstnvstreammux.c(2803): gst_nvstreammux_change_state (): /GstPipeline:pipeline0/GstNvStreamMux:mux
Redistribute latency...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:02.471190176
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

I am doing a really simple pipeline and trying to enable simple thing in accordance with documentation - I don’t understand why is it so hard to enable such a simple feature?

9

I don’t see nvdspreprocess in your pipeline, please don’t use “input-tensor-meta”.

How can you know “symmetric-padding” don’t take effect?

10

I know that symmetric-padding does not take effect because as you can see in the pipeline I am saving the frames that enter and leave nvinfer element as video-file - I have tried various configurations - you can try them yourself and verify my claims.

Can you explain how to use nvdspreprocess element in this pipeline to enable asymmetric padding?

11

Suppose “symmetric-padding” is for scale video to model size and then inference on it. The output of nvinfer shouldn’t effect by “symmetric-padding”.

12

ok, so in that scenario streammux element should only be responsible for batching - as you can see I have two video sources - one vertical and one horizontal - output resolution of streammux must be specified (so it must be either vertical or horizontal) - how to overcome this issue ?

13

Yes. streammux will scale to same resolution. You can enable padding to keep ratio.

14

Yes, and this padding is always symmetric - can you provide me a simple pipeline with asymmetric padding enabled? Just a simple example will do

15

streammux only support symmetric padding currently. One possible solution is process video based on your requirement before feed to streammux.

17

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.