No sgie metadata for some pgie detections using pyds

[pwoolvett]

Hi. I’m running a deepstream pipeline containing pgie detector + sgie classifier. Given detection metadata extracted from some frame, I’m not always able to extract its corresponding classification metadata.

• PIPELINE: The pipeline I’m using is the following (parsed with Gst.parse_launch)

  multifilesrc
    name=src
    location=cars/%012d.jpg
    caps="image/jpeg"
  ! jpegdec
    name=decoder
  ! nvvideoconvert
  ! video/x-raw(memory:NVMM), format=NV12, width=1280, height=720
  ! m.sink_0
    nvstreammux
    name=m
    batch-size=1
    width=1280
    height=720
  ! nvinfer
    config-file-path=models/Primary_Detector/config_infer_primary.txt
  ! nvinfer
    config-file-path=models/Secondary_CarColor/config_infer_secondary_carcolor.txt
  ! fakesink name=monitor
  

• SGIE CONFIGURATION FILE:

  [property]
  gie-unique-id=2
  operate-on-gie-id=1
  network-type=1
  process-mode=2
  classifier-async-mode=0
  
  net-scale-factor=1
  model-file=../../models/Secondary_CarColor/resnet18.caffemodel
  proto-file=../../models/Secondary_CarColor/resnet18.prototxt
  model-engine-file=../../models/Secondary_CarColor/resnet18.caffemodel_b32_gpu0_int8.engine
  int8-calib-file=../../models/Secondary_CarColor/cal_trt.bin
  mean-file=../../models/Secondary_CarColor/mean.ppm
  labelfile-path=../../models/Secondary_CarColor/labels.txt
  model-color-format=1
  network-mode=0
  output-blob-names=predictions/Softmax
  
  force-implicit-batch-dim=1
  batch-size=32
  
  classifier-threshold=0
  input-object-min-width=0
  input-object-min-height=0
  input-object-max-width=0
  input-object-max-height=0
  

• DATA: Data generated from cars video with this cmd

  ffmpeg -i /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264 -start_number 0 -vframes 1088 cars/%012d.jpg
  

• BUFFER_PROBE_CALLBACK: Metadata extracted just like in the sample callback up to this point, attached to the monitor element. Then:

  ...
  obj_meta=pyds.NvDsObjectMeta.cast(l_obj.data)
  
  report_obj_meta_counter(obj_meta)
  classifier_meta_objects = obj_meta.classifier_meta_list  # THIS IS NONE SOMETIMES, but generally a `pyds.GList`
  while classifier_meta_objects is not None:
      try:
          classifier_metadata = pyds.NvDsClassifierMeta.cast(classifier_meta_objects.data)
      except StopIteration:
          break
  
      label_info_list = obj_meta.label_info_list  # I think we've got to iterate again here for the multilabel case...
      while label_info_list is not None:
          try:
              label_info = pyds.NvDsLabelInfo.cast(label_info_list.data)
          except StopIteration:
              break
          report_label_info_counter(label_info)
          try:
              label_info_list = label_info_list.next
          except StopIteration:
              break
      try:
          classifier_meta_objects = classifier_meta_objects.next
      except StopIteration:
          break
  ...
  
  

• NOTES

1. Using the snippet above, the number of classifications (15249 as reported by report_label_info_counter) is slightly lower than the number of detections (16496 as reported by report_obj_meta_counter). There are cases where obj_meta.classifier_meta_list is None, instead of a pyds.GList.

2. The same number of frames (1088) are being processed if I change the decoder from jpegdec to nvjpegdec. If I turn on raw-output-tensors, both the pgie and sgie output 1088, consistent with the number of frames. However, I loose 11 detections (from 16496 to 16485. On the other hand, the number of classifications goes up from 15249 to 15252. These numbers are consistent across >10 runs each.

3. I can manually enforce the SGIE to “skip” detections, eg by increasing the input-object-min-width to 200. In that case, when also enabling raw-output-tensors, the number of sgie output tensors decreases (to 365), which makes me doubt its a sgie config file issue…

4. Other

   • Using dla does not affect the numbers.
   • network-mode does not affect the numbers
   • Using jpegdec vs nvjpegdec does, but slightly (maybe a sync or a flush thing?), and does not solve the problem.

• QUESTIONS

1. What happened to those obj_meta which do not have corresponding classifier_meta_list?
2. Is this a pyds or nvinfer issue?

---

• Hardware Platform (Jetson / GPU)

JETSON_TYPE=AGX Xavier [16GB]
JETSON_CHIP_ID=25
JETSON_SOC=tegra194
JETSON_MACHINE=NVIDIA Jetson AGX Xavier [16GB]
JETSON_CODENAME=galen
JETSON_BOARD=P2822-0000
JETSON_MODULE=P2888-0001

• DeepStream Version Version: 5.0 (GCID: 23607587)
• JetPack Version (valid for Jetson only) 4.4
• TensorRT Version : 7.1.3.0

[bcao]

Hey, could you share a repro with us.
Also, have you tried to use deepstream-app to run your pipeline?

[pwoolvett]

Hi, thanks for answering…

@bcao

Hey, could you share a repro with us.

Here’s a MWE for Jetson Xavier

@bcao

Also, have you tried to use deepstream-app to run your pipeline?

I have not. Can I use it from python? I’m not comfortable with C++, but if it ends up being a pyds issue, I’m not closed to considering other alternatives. I’ll take a look in the meantime.

[pbcorrea1]

Any updates on this? I’m experiencing a similar issue.

[pwoolvett]

Nothing so far. I’ve also tried extracting label metadata directly from the batch. Same number of “lost” detections:

...
batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
...
plist = batch_meta.label_info_meta_pool.full_list
while plist is not Note:
  try:
    label_info = pyds.NvDsLabelInfo.cast(plist.data)
  except StopIteration:
    break
  report_label2_meta_counter(label_info)
  try:
    plist=plist.next
  except StopIteration:
    break
...

[mohammad1]

Hello there, any updates regarding this issue. I am facing similar thing using C++ deepstream.

I am using a primary model to detect people and a custom secondary model (classifier). However, not all detected objects have metadata attached from the secondary although setting classification-threshold to 0.1 and even 0

[bcao]

Hey @mohammad1 , so it’s not related to python, seems caused by meta data management, would you mind to share your repro with us, I would like to use the native c/c++ sample to debug.

[mohammad1]

Thanks @bcao for following up with this issue. The code is available at https://github.com/zaka-ai/ds_example2_classification_problem

It is the same deepstream-test2 example (vehicle + color + type + make) with small modifications. I am printing the output of the classification of each object. As shown in the image attached, classification exits for some objects but not all and for some classification models but not all.

[mohammad1]

These are some relevant issues but the solutions are either unsatisfactory or didn’t work

Secondary classifiers labels are missing from output

Please provide complete information as applicable to your setup. • Hardware Platform (Jetson / GPU) - T4 • DeepStream Version - 5 • TensorRT Version - 7.0 • NVIDIA GPU Driver Version (valid for GPU only) - 440.82 I am using a .c file to create deepstream-app configurations I am getting correct output up till the tracker but not the classification part! Like a have pipeline as FaceDetector -> tracker -> classifier and I am getting output for 4 faces as face 4, face 3, face 2, face 1. Also …

deepstream4.0 secondary classification problem

@AastaLLL While this question is old, I am facing some similar problems. I have a pipeline with 1 detector and 3 classifiers, running on Deepstream Version 5. My pipeline builds on the pipeline from DeepStream Test 2 Python Application. I am not able to run more than 2 classifiers at the same time on the detected objects. All the classifiers work on the same class of object detected on first primary inference engine. I am also using a tracker. Primary Engine Detection |---------------- Classif…

how do secondary-gie detection

Hi, The output of classifier (secondary-gie) is the text of label. It is shown when the bounding box is large enough. Here is a sample for your reference: # Copyright (c) 2019 NVIDIA Corporation. All rights reserved. # # NVIDIA Corporation and its licensors retain all intellectual property # and proprietary rights in and to this software, related documentation # and any modifications thereto. Any use, reproduction, disclosure or # distribution of this software and related documentation wit…

[mohammad1]

Hello @bcao , any updates regarding this issue?

[bcao]

Hey, we are looking into it, will reply you ASAP.

[bcao]

I checked the issue and the reason should be the object width or height < 16.
You can check the gstnvinfer.cpp ->should_infer_object for more details.

[pwoolvett]

Thanks @bcao !

• Every single detection without a corresponding classification has either of the dimmensions lower than 16.

• Increasing MIN_INPUT_OBJECT_WIDTH and MIN_INPUT_OBJECT_HEIGHT( around line 60), increases the difference between detections vs classifications.

I cannot lower the values (segfault). A comment in the code even says

/* Should not infer on objects smaller than MIN_INPUT_OBJECT_WIDTH x MIN_INPUT_OBJECT_HEIGHT
   * since it will cause hardware scaling issues. */

Is there a way around this?

It would be good tho have in the docs somewhere that a value lower than 16 in the pgie-config-file is ignored :)

[bcao]

@pwoolvett

I cannot lower the values (segfault).

Could you set scaling-compute-hw=1 and then lower those values and try again?

[pwoolvett]

SOLUTION:

1. lower the MIN_INPUT_OBJECT_WIDTH and MIN_INPUT_OBJECT_HEIGHT macro values in gstnvinfer.cpp
2. scaling-compute-hw=1 in sgie

---

1. revert to original values
2. make clean && make install
3. sgie → scaling-compute-hw={0,1,2}

original results, no changes (16496 vs 15249)

1. change macros to (8,8)
2. cd /opt/nvidia/deepstream/deepstream-5.0/sources/gst-plugins/gst-nvinfer && make clean && make install
3. sgie → scaling-compute-hw=2 → segfault most of the times:

   /dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform.cpp:3115: => VIC Configuration failed image scale factor exceeds 16, use GPU for Transformation
   0:00:03.629223122  5083      0xfa42e80 WARN                 nvinfer gstnvinfer.cpp:1268:convert_batch_and_push_to_input_thread:<nvinfer1> error: NvBufSurfTransform failed with error -2 while converting buffer
   0:00:03.629371225  5083      0xfa42e80 WARN                 nvinfer gstnvinfer.cpp:1975:gst_nvinfer_output_loop:<nvinfer0> error: Internal data stream error.
   0:00:03.629393434  5083      0xfa42e80 WARN                 nvinfer gstnvinfer.cpp:1975:gst_nvinfer_output_loop:<nvinfer0> error: streaming stopped, reason error (-5)
   0:00:03.630007863  5083      0xfa42e80 WARN                 nvinfer gstnvinfer.cpp:1268:convert_batch_and_push_to_input_thread:<nvinfer1> error: NvBufSurfTransform failed with error -3 while converting buffer
   Segmentation fault (core dumped)
   

which, accordfing to nvbufsurftransform.h, mean “NvBufSurfTransformError_Invalid_Params” and “NvBufSurfTransformError_Execution_Error”

• when it did not crash, it got 795 (same value for both detector and classifier!). I think this might have happened regardless of the macro change If I had tried enough times…

This got me thinking:

• The docs say this Integer 0: Platform default – GPU (dGPU), VIC (Jetson) 1: GPU 2: VIC (Jetson only)
• The log asked to use GPU
• sgie → scaling-compute-hw=1 => 16496 detections and 16496 classifications!!!

Thanks again for the insights @bcao , I’ll run more tests to ensure this is reproducible with different values, but this workaround completely fulfills my needs. If I am able to reproduce this after more experimentation, I’ll come back and mark as solved :)

[pwoolvett]

I’ve tried changing the macros all the way down to 0, and scaling-compute-hw=1 allows correct execution.

• NOTE: the smallest detections I got from the sample data had minimum bboxes in the 8-16 range, so while I can confirm this works for smaller bboxes than the original setting (lower than 16), I have not tested everything works /won’t crash in case the detection produces bboxes with dimensions lower than 8.

[bcao]

Great work, had corrected my commets, yeah, scaling-compute-hw should be 1, since the known hardware scaling issues is from VIC.

But keep in mind we also didn’t test such case

@pwoolvett

I’ve tried changing the macros all the way down to 0, and scaling-compute-hw=1 allows correct execution.