No sgie metadata for some pgie detections using pyds

Hi. I’m running a deepstream pipeline containing pgie detector + sgie classifier. Given detection metadata extracted from some frame, I’m not always able to extract its corresponding classification metadata.

  • PIPELINE: The pipeline I’m using is the following (parsed with Gst.parse_launch)

    multifilesrc
      name=src
      location=cars/%012d.jpg
      caps="image/jpeg"
    ! jpegdec
      name=decoder
    ! nvvideoconvert
    ! video/x-raw(memory:NVMM), format=NV12, width=1280, height=720
    ! m.sink_0
      nvstreammux
      name=m
      batch-size=1
      width=1280
      height=720
    ! nvinfer
      config-file-path=models/Primary_Detector/config_infer_primary.txt
    ! nvinfer
      config-file-path=models/Secondary_CarColor/config_infer_secondary_carcolor.txt
    ! fakesink name=monitor
    
  • SGIE CONFIGURATION FILE:

    [property]
    gie-unique-id=2
    operate-on-gie-id=1
    network-type=1
    process-mode=2
    classifier-async-mode=0
    
    net-scale-factor=1
    model-file=../../models/Secondary_CarColor/resnet18.caffemodel
    proto-file=../../models/Secondary_CarColor/resnet18.prototxt
    model-engine-file=../../models/Secondary_CarColor/resnet18.caffemodel_b32_gpu0_int8.engine
    int8-calib-file=../../models/Secondary_CarColor/cal_trt.bin
    mean-file=../../models/Secondary_CarColor/mean.ppm
    labelfile-path=../../models/Secondary_CarColor/labels.txt
    model-color-format=1
    network-mode=0
    output-blob-names=predictions/Softmax
    
    force-implicit-batch-dim=1
    batch-size=32
    
    classifier-threshold=0
    input-object-min-width=0
    input-object-min-height=0
    input-object-max-width=0
    input-object-max-height=0
    
  • DATA: Data generated from cars video with this cmd

    ffmpeg -i /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264 -start_number 0 -vframes 1088 cars/%012d.jpg
    
  • BUFFER_PROBE_CALLBACK: Metadata extracted just like in the sample callback up to this point, attached to the monitor element. Then:

    ...
    obj_meta=pyds.NvDsObjectMeta.cast(l_obj.data)
    
    report_obj_meta_counter(obj_meta)
    classifier_meta_objects = obj_meta.classifier_meta_list  # THIS IS NONE SOMETIMES, but generally a `pyds.GList`
    while classifier_meta_objects is not None:
        try:
            classifier_metadata = pyds.NvDsClassifierMeta.cast(classifier_meta_objects.data)
        except StopIteration:
            break
    
        label_info_list = obj_meta.label_info_list  # I think we've got to iterate again here for the multilabel case...
        while label_info_list is not None:
            try:
                label_info = pyds.NvDsLabelInfo.cast(label_info_list.data)
            except StopIteration:
                break
            report_label_info_counter(label_info)
            try:
                label_info_list = label_info_list.next
            except StopIteration:
                break
        try:
            classifier_meta_objects = classifier_meta_objects.next
        except StopIteration:
            break
    ...
    
    

• NOTES

  1. Using the snippet above, the number of classifications (15249 as reported by report_label_info_counter) is slightly lower than the number of detections (16496 as reported by report_obj_meta_counter). There are cases where obj_meta.classifier_meta_list is None, instead of a pyds.GList.

  2. The same number of frames (1088) are being processed if I change the decoder from jpegdec to nvjpegdec. If I turn on raw-output-tensors, both the pgie and sgie output 1088, consistent with the number of frames. However, I loose 11 detections (from 16496 to 16485. On the other hand, the number of classifications goes up from 15249 to 15252. These numbers are consistent across >10 runs each.

  3. I can manually enforce the SGIE to “skip” detections, eg by increasing the input-object-min-width to 200. In that case, when also enabling raw-output-tensors, the number of sgie output tensors decreases (to 365), which makes me doubt its a sgie config file issue…

  4. Other

    • Using dla does not affect the numbers.
    • network-mode does not affect the numbers
    • Using jpegdec vs nvjpegdec does, but slightly (maybe a sync or a flush thing?), and does not solve the problem.

• QUESTIONS

  1. What happened to those obj_meta which do not have corresponding classifier_meta_list?
  2. Is this a pyds or nvinfer issue?

• Hardware Platform (Jetson / GPU)

JETSON_TYPE=AGX Xavier [16GB]
JETSON_CHIP_ID=25
JETSON_SOC=tegra194
JETSON_MACHINE=NVIDIA Jetson AGX Xavier [16GB]
JETSON_CODENAME=galen
JETSON_BOARD=P2822-0000
JETSON_MODULE=P2888-0001

• DeepStream Version Version: 5.0 (GCID: 23607587)
• JetPack Version (valid for Jetson only) 4.4
• TensorRT Version : 7.1.3.0

2 Likes

Hey, could you share a repro with us.
Also, have you tried to use deepstream-app to run your pipeline?

Hi, thanks for answering…

Here’s a MWE for Jetson Xavier

I have not. Can I use it from python? I’m not comfortable with C++, but if it ends up being a pyds issue, I’m not closed to considering other alternatives. I’ll take a look in the meantime.

Any updates on this? I’m experiencing a similar issue.

Nothing so far. I’ve also tried extracting label metadata directly from the batch. Same number of “lost” detections:

...
batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
...
plist = batch_meta.label_info_meta_pool.full_list
while plist is not Note:
  try:
    label_info = pyds.NvDsLabelInfo.cast(plist.data)
  except StopIteration:
    break
  report_label2_meta_counter(label_info)
  try:
    plist=plist.next
  except StopIteration:
    break
...

Hello there, any updates regarding this issue. I am facing similar thing using C++ deepstream.

I am using a primary model to detect people and a custom secondary model (classifier). However, not all detected objects have metadata attached from the secondary although setting classification-threshold to 0.1 and even 0

Hey @mohammad1 , so it’s not related to python, seems caused by meta data management, would you mind to share your repro with us, I would like to use the native c/c++ sample to debug.

Thanks @bcao for following up with this issue. The code is available at https://github.com/zaka-ai/ds_example2_classification_problem

It is the same deepstream-test2 example (vehicle + color + type + make) with small modifications. I am printing the output of the classification of each object. As shown in the image attached, classification exits for some objects but not all and for some classification models but not all.

image

These are some relevant issues but the solutions are either unsatisfactory or didn’t work