Feed SGIE output back into pipeline

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

• Hardware Platform: GPU
• DeepStream Version: 5.0.0
• TensorRT Version: 7.0.0.11
• NVIDIA GPU Driver Version (valid for GPU only): 460.32.03

I want to setup a pipeline including PGIE and SGIE. PGIE is a detector which outputs bboxes which I parse and everything gets displayed correctly. I’m now trying to add a SGIE which is another detector which I want to detect features on each of the parsed bounding boxes. Now I would like to
a) create bounding boxes out of SGIEs output and have them drawn in OSD
as well as
b) eventually add a 2nd SGIE which operates on PGIE but the data fed into SGIE2 is manipulated according to SGIE1s output

I’m using the stock deepstream-app from the SDK and so far just edited the respective config files like this:

[primary-gie]
enable=1
gpu-id=0
batch-size=1
gie-unique-id=1
interval=0
config-file=config_infer_primary_ssd_custom.txt
nvbuf-memory-type=0

[secondary-gie0]
enable=1
gpu-id=0
batch-size=1
interval=0
gie-unique-id=2
operate-on-gie-id=1
config-file=config_infer_secondary_landmark.txt
nvbuf-memory-type=0

where config_infer_secondary_landmark.txt is this

[property]
gpu-id=0
net-scale-factor=1
onnx-file=/opt/models/lmd192.onnx
batch-size=1
# Integer 1=Primary 2=Secondary
process-mode=2
gie-unique-id=2
operate-on-gie-id=1
#nw mode 0: FP32, 1: INT8, 2: FP16
network-mode=0
#nw type Integer 0: Detector 1: Classifier 2: Segmentation 3: Instance Segmentation 100=Other
network-type=0
# Integer 0: OpenCV groupRectangles() 1: DBSCAN 2: Non Maximum Suppression 3: DBSCAN + NMS Hybrid 4:No clustering
cluster-mode=2
# Binding dimensions to set on the image input layer.
# infer-dims=3;192;192
infer-dims=1;192;192
parse-bbox-func-name=NvDsInferParseLandmarks
custom-lib-path=nvdsinfer_landmark_parser/libnvdsinfer_landmark_parser.so
# Color format required by the model Integer 0: RGB 1: BGR 2: GRAY
model-color-format=2
num-detected-classes=2
# Gst-nvinfer attaches raw tensor output as Gst Buffer metadata.
#output-tensor-meta=1

There are some questions:

  • Why is outputLayersInfo in NvDsInferParseLandmarks always empty?
  • Is network-type=0 right or should it be 100? What is this 100?
  • Do I need the num-detected-classes=2
  • infer-dims=1;192;192 resizes the crops from PGIE to this format before being fed into SGIE right?
  • NvDsInferParseLandmarks is called for each individual bbox found in PGIE?
3

Does adding one tee which branch the data into different branches meet your requirement, pgie, sgie0, sgie1 not in sequence order, see tee

  • Why is outputLayersInfo in NvDsInferParseLandmarks always empty?

→ Did your inference run success?

  • Is network-type=0 right or should it be 100 ? What is this 100?

→ should be 100 for your case.

  • Do I need the num-detected-classes=2

→ how many output you have in your model

  • infer-dims=1;192;192 resizes the crops from PGIE to this format before being fed into SGIE right?

→ before inference, it will do resize and conversion based on network information

  • NvDsInferParseLandmarks is called for each individual bbox found in PGIE?

→ Yes.

6

how many output you have in your model

it’s just one output

Does adding one tee which branch the data into different branches meet your requirement, pgie, sgie0, sgie1 not in sequence order, see tee

This is what deepstream-app apparently does. There is a tee after PGIE where one branch goes into SGIE which straight leads into a fakesink. Is it possible to feed SGIEs output back into the pipeline?

7

I changed my pipeline to follow deepstream-test2 example and now things look like supposed (… → PGIE → SGIE → … → OSD → …). However I don’t see any output from SGIE. Where would I expect that? I installed a probe on SGIEs src pad and see the data from PGIE there but no idea where to look for SGIEs data. Any hint? When enabling

output-tensor-meta=1

in SGIE config my pipeline is not reaching the playing state. Calling gst_element_get_state() after gst_element_set_state(…, PLAYING) is not returning, while it works when output-tensor-meta=0. Any idea?

And final question: When using probes on SGIE src pad and on OSD sink pad it seems like only SGIE src probe callback gets called but not OSD sinks cb. Is that supposed to be like that and in case why?

8

back into which element?

and from your description,

b) eventually add a 2nd SGIE which operates on PGIE but the data fed into SGIE2 is manipulated according to SGIE1s output

from the first part, data fed into 2nd SGIE is from PGIE, but the second part, it’s from SGIE1? isn’t it conflict?

9

if your pipeline run success, you will get the OSD pad after the probe on OSD sink pad, please check GST log where got wrong. https://gstreamer.freedesktop.org/documentation/tutorials/basic/debugging-tools.html?gi-language=c