No sgie metadata for some pgie detections using pyds

[pwoolvett]

Hi. I’m running a deepstream pipeline containing pgie detector + sgie classifier. Given detection metadata extracted from some frame, I’m not always able to extract its corresponding classification metadata.

• PIPELINE: The pipeline I’m using is the following (parsed with Gst.parse_launch)

  multifilesrc
    name=src
    location=cars/%012d.jpg
    caps="image/jpeg"
  ! jpegdec
    name=decoder
  ! nvvideoconvert
  ! video/x-raw(memory:NVMM), format=NV12, width=1280, height=720
  ! m.sink_0
    nvstreammux
    name=m
    batch-size=1
    width=1280
    height=720
  ! nvinfer
    config-file-path=models/Primary_Detector/config_infer_primary.txt
  ! nvinfer
    config-file-path=models/Secondary_CarColor/config_infer_secondary_carcolor.txt
  ! fakesink name=monitor
  

• SGIE CONFIGURATION FILE:

  [property]
  gie-unique-id=2
  operate-on-gie-id=1
  network-type=1
  process-mode=2
  classifier-async-mode=0
  
  net-scale-factor=1
  model-file=../../models/Secondary_CarColor/resnet18.caffemodel
  proto-file=../../models/Secondary_CarColor/resnet18.prototxt
  model-engine-file=../../models/Secondary_CarColor/resnet18.caffemodel_b32_gpu0_int8.engine
  int8-calib-file=../../models/Secondary_CarColor/cal_trt.bin
  mean-file=../../models/Secondary_CarColor/mean.ppm
  labelfile-path=../../models/Secondary_CarColor/labels.txt
  model-color-format=1
  network-mode=0
  output-blob-names=predictions/Softmax
  
  force-implicit-batch-dim=1
  batch-size=32
  
  classifier-threshold=0
  input-object-min-width=0
  input-object-min-height=0
  input-object-max-width=0
  input-object-max-height=0
  

• DATA: Data generated from cars video with this cmd

  ffmpeg -i /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264 -start_number 0 -vframes 1088 cars/%012d.jpg
  

• BUFFER_PROBE_CALLBACK: Metadata extracted just like in the sample callback up to this point, attached to the monitor element. Then:

  ...
  obj_meta=pyds.NvDsObjectMeta.cast(l_obj.data)
  
  report_obj_meta_counter(obj_meta)
  classifier_meta_objects = obj_meta.classifier_meta_list  # THIS IS NONE SOMETIMES, but generally a `pyds.GList`
  while classifier_meta_objects is not None:
      try:
          classifier_metadata = pyds.NvDsClassifierMeta.cast(classifier_meta_objects.data)
      except StopIteration:
          break
  
      label_info_list = obj_meta.label_info_list  # I think we've got to iterate again here for the multilabel case...
      while label_info_list is not None:
          try:
              label_info = pyds.NvDsLabelInfo.cast(label_info_list.data)
          except StopIteration:
              break
          report_label_info_counter(label_info)
          try:
              label_info_list = label_info_list.next
          except StopIteration:
              break
      try:
          classifier_meta_objects = classifier_meta_objects.next
      except StopIteration:
          break
  ...
  
  

• NOTES

1. Using the snippet above, the number of classifications (15249 as reported by report_label_info_counter) is slightly lower than the number of detections (16496 as reported by report_obj_meta_counter). There are cases where obj_meta.classifier_meta_list is None, instead of a pyds.GList.

2. The same number of frames (1088) are being processed if I change the decoder from jpegdec to nvjpegdec. If I turn on raw-output-tensors, both the pgie and sgie output 1088, consistent with the number of frames. However, I loose 11 detections (from 16496 to 16485. On the other hand, the number of classifications goes up from 15249 to 15252. These numbers are consistent across >10 runs each.

3. I can manually enforce the SGIE to “skip” detections, eg by increasing the input-object-min-width to 200. In that case, when also enabling raw-output-tensors, the number of sgie output tensors decreases (to 365), which makes me doubt its a sgie config file issue…

4. Other

   • Using dla does not affect the numbers.
   • network-mode does not affect the numbers
   • Using jpegdec vs nvjpegdec does, but slightly (maybe a sync or a flush thing?), and does not solve the problem.

• QUESTIONS

1. What happened to those obj_meta which do not have corresponding classifier_meta_list?
2. Is this a pyds or nvinfer issue?

---

• Hardware Platform (Jetson / GPU)

JETSON_TYPE=AGX Xavier [16GB]
JETSON_CHIP_ID=25
JETSON_SOC=tegra194
JETSON_MACHINE=NVIDIA Jetson AGX Xavier [16GB]
JETSON_CODENAME=galen
JETSON_BOARD=P2822-0000
JETSON_MODULE=P2888-0001

• DeepStream Version Version: 5.0 (GCID: 23607587)
• JetPack Version (valid for Jetson only) 4.4
• TensorRT Version : 7.1.3.0

[bcao]

Hey, could you share a repro with us.
Also, have you tried to use deepstream-app to run your pipeline?

[pwoolvett]

Hi, thanks for answering…

@bcao

Hey, could you share a repro with us.

Here’s a MWE for Jetson Xavier

@bcao

Also, have you tried to use deepstream-app to run your pipeline?

I have not. Can I use it from python? I’m not comfortable with C++, but if it ends up being a pyds issue, I’m not closed to considering other alternatives. I’ll take a look in the meantime.

[pbcorrea1]

Any updates on this? I’m experiencing a similar issue.

[pwoolvett]

Nothing so far. I’ve also tried extracting label metadata directly from the batch. Same number of “lost” detections:

...
batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
...
plist = batch_meta.label_info_meta_pool.full_list
while plist is not Note:
  try:
    label_info = pyds.NvDsLabelInfo.cast(plist.data)
  except StopIteration:
    break
  report_label2_meta_counter(label_info)
  try:
    plist=plist.next
  except StopIteration:
    break
...

[mohammad1]

Hello there, any updates regarding this issue. I am facing similar thing using C++ deepstream.

I am using a primary model to detect people and a custom secondary model (classifier). However, not all detected objects have metadata attached from the secondary although setting classification-threshold to 0.1 and even 0

[bcao]

Hey @mohammad1 , so it’s not related to python, seems caused by meta data management, would you mind to share your repro with us, I would like to use the native c/c++ sample to debug.

[mohammad1]

Thanks @bcao for following up with this issue. The code is available at https://github.com/zaka-ai/ds_example2_classification_problem

It is the same deepstream-test2 example (vehicle + color + type + make) with small modifications. I am printing the output of the classification of each object. As shown in the image attached, classification exits for some objects but not all and for some classification models but not all.

[mohammad1]

These are some relevant issues but the solutions are either unsatisfactory or didn’t work

Secondary classifiers labels are missing from output

Please provide complete information as applicable to your setup. • Hardware Platform (Jetson / GPU) - T4 • DeepStream Version - 5 • TensorRT Version - 7.0 • NVIDIA GPU Driver Version (valid for GPU only) - 440.82 I am using a .c file to create deepstream-app configurations I am getting correct output up till the tracker but not the classification part! Like a have pipeline as FaceDetector -> tracker -> classifier and I am getting output for 4 faces as face 4, face 3, face 2, face 1. Also …

deepstream4.0 secondary classification problem

@AastaLLL While this question is old, I am facing some similar problems. I have a pipeline with 1 detector and 3 classifiers, running on Deepstream Version 5. My pipeline builds on the pipeline from DeepStream Test 2 Python Application. I am not able to run more than 2 classifiers at the same time on the detected objects. All the classifiers work on the same class of object detected on first primary inference engine. I am also using a tracker. Primary Engine Detection |---------------- Classif…

how do secondary-gie detection

Hi, The output of classifier (secondary-gie) is the text of label. It is shown when the bounding box is large enough. Here is a sample for your reference: # Copyright (c) 2019 NVIDIA Corporation. All rights reserved. # # NVIDIA Corporation and its licensors retain all intellectual property # and proprietary rights in and to this software, related documentation # and any modifications thereto. Any use, reproduction, disclosure or # distribution of this software and related documentation wit…