Gstdsexample pipeline order matter?

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) AGX Xavier
• DeepStream Version 5.0
• JetPack Version (valid for Jetson only) 4.4

I have successfully created a simple full frame filter (GaussianBlur) by modifying gstdsexample.cpp as shown in this post and am able to run the following pipeline:

gst-launch-1.0 --gst-debug-level=0 filesrc location= ~/data/ar.h264 ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvinfer config-file-path= /opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.txt ! nvvideoconvert ! dsexample full-frame=1 processing-width=1920 processing-height=1080 ! nvdsosd ! nvegltransform ! nveglglessink

Then I want to run the GaussianBlur filter before the nvinfer, so I move the dsexample element in front of nvinfer as below:

gst-launch-1.0 --gst-debug-level=0 filesrc location= ~/data/ar.h264 ! h264parse ! nvv4l2decoder ! nvvideoconvert ! dsexample full-frame=1 processing-width=1920 processing-height=1080 ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvinfer config-file-path= /opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.txt ! nvvideoconvert ! nvdsosd ! nvegltransform ! nveglglessink

when run above pipeline, I encounter the following error message:

Setting pipeline to PAUSED …

Using winsys: x11
Opening in BLOCKING MODE
0:00:03.291576256 12973 0x55be6e1580 INFO nvinfer gstnvinfer.cpp:602:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1577> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-5.0/samples/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_int8.engine
INFO: [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT input_1 3x368x640
1 OUTPUT kFLOAT conv2d_bbox 16x23x40
2 OUTPUT kFLOAT conv2d_cov/Sigmoid 4x23x40

0:00:03.291798728 12973 0x55be6e1580 INFO nvinfer gstnvinfer.cpp:602:gst_nvinfer_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1681> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-5.0/samples/models/Primary_Detector/resnet10.caffemodel_b1_gpu0_int8.engine
0:00:03.294633527 12973 0x55be6e1580 INFO nvinfer gstnvinfer_impl.cpp:311:notifyLoadModelStatus: [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.txt sucessfully
Pipeline is PREROLLING …
Got context from element ‘eglglessink0’: gst.egl.EGLDisplay=context, display=(GstEGLDisplay)NULL;
NvMMLiteOpen : Block : BlockType = 261
NVMEDIA: Reading vendor.tegra.display-size : status: 6
NvMMLiteBlockCreate : Block : BlockType = 261
terminate called after throwing an instance of ‘cv::Exception’
what(): OpenCV(4.3.0) /home/paul/Documents/cuda_python/install/opencv_contrib-4.3.0/modules/cudafilters/src/cuda/row_filter.hpp:172: error: (-217:Gpu API call) an illegal memory access was encountered in function ‘caller’

./front.agx: line 1: 12973 Aborted gst-launch-1.0 --gst-debug-level=0 filesrc location= ~/data/ar.h264 ! h264parse ! nvv4l2decoder ! nvvideoconvert ! dsexample full-frame=1 processing-width=1920 processing-height=1080 ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! nvinfer config-file-path= /opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.txt ! nvvideoconvert ! nvdsosd ! nvegltransform ! nveglglessink

Then I comment out
filter->apply (d_mat, d_mat);
in gstdsexample.cpp and “make” and “sudo make install”, then I am able to run the above pipeline without error (of course this “do-nothing” dsexample deviates from my original intention to do filtering before nvinfer)

Question: why the same filter (opencv cuda) can run after nvinfer but cannot run before nvinfer? does the order matter? or am I missing something if I place the dsexample element infront of nvinfer? Please advise. Thank you for your help. (Be aware that Here the dsexample already removes all meta data processing shown in the original example and is stripped down to a simple full frame in-place filtering as mentioned in this post )

Opencv cuda may need specific cuda memory. Can you try to add “nvbuf-memory-type=4” property with nvvideoconvert before dsexample to make it use compatible memory?

Thank you Fiona. I tried to add “nvbuf-memory-type=4”, it still gives the same error message. However, I rearrange my pipeline order to move my dsexample element to after nvstreammux and before nvinfer, then it works:

gst-launch-1.0 --gst-debug-level=0 filesrc location= ~/data/ar.h264 ! h264parse ! nvv4l2decoder ! nvvideoconvert ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 ! dsexample full-frame=1 processing-width=1920 processing-height=1080 ! nvinfer config-file-path= /opt/nvidia/deepstream/deepstream-5.0/sources/apps/sample_apps/deepstream-test1/dstest1_pgie_config.txt ! nvvideoconvert ! nvdsosd ! nvegltransform ! nveglglessink

I think this is something to do with that nvstreammux breaks down the batched Gst buffer into a single frame Gst buffer which might be required by the dsexample plugin to consume. So the answer is yes and no: Yes, the order matters: both dsexample and nvinfer plugins need to be after nvstreammux. Then, No, order doesn’t matter whether dsexample or nvinfer is in front as long as both are after nvstreammux.