Hi. I’m running a deepstream pipeline containing pgie detector + sgie classifier. Given detection metadata extracted from some frame, I’m not always able to extract its corresponding classification metadata.
-
PIPELINE: The pipeline I’m using is the following (parsed with
Gst.parse_launch
)multifilesrc name=src location=cars/%012d.jpg caps="image/jpeg" ! jpegdec name=decoder ! nvvideoconvert ! video/x-raw(memory:NVMM), format=NV12, width=1280, height=720 ! m.sink_0 nvstreammux name=m batch-size=1 width=1280 height=720 ! nvinfer config-file-path=models/Primary_Detector/config_infer_primary.txt ! nvinfer config-file-path=models/Secondary_CarColor/config_infer_secondary_carcolor.txt ! fakesink name=monitor
-
SGIE CONFIGURATION FILE:
[property] gie-unique-id=2 operate-on-gie-id=1 network-type=1 process-mode=2 classifier-async-mode=0 net-scale-factor=1 model-file=../../models/Secondary_CarColor/resnet18.caffemodel proto-file=../../models/Secondary_CarColor/resnet18.prototxt model-engine-file=../../models/Secondary_CarColor/resnet18.caffemodel_b32_gpu0_int8.engine int8-calib-file=../../models/Secondary_CarColor/cal_trt.bin mean-file=../../models/Secondary_CarColor/mean.ppm labelfile-path=../../models/Secondary_CarColor/labels.txt model-color-format=1 network-mode=0 output-blob-names=predictions/Softmax force-implicit-batch-dim=1 batch-size=32 classifier-threshold=0 input-object-min-width=0 input-object-min-height=0 input-object-max-width=0 input-object-max-height=0
-
DATA: Data generated from cars video with this cmd
ffmpeg -i /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264 -start_number 0 -vframes 1088 cars/%012d.jpg
-
BUFFER_PROBE_CALLBACK: Metadata extracted just like in the sample callback up to this point, attached to the
monitor
element. Then:... obj_meta=pyds.NvDsObjectMeta.cast(l_obj.data) report_obj_meta_counter(obj_meta) classifier_meta_objects = obj_meta.classifier_meta_list # THIS IS NONE SOMETIMES, but generally a `pyds.GList` while classifier_meta_objects is not None: try: classifier_metadata = pyds.NvDsClassifierMeta.cast(classifier_meta_objects.data) except StopIteration: break label_info_list = obj_meta.label_info_list # I think we've got to iterate again here for the multilabel case... while label_info_list is not None: try: label_info = pyds.NvDsLabelInfo.cast(label_info_list.data) except StopIteration: break report_label_info_counter(label_info) try: label_info_list = label_info_list.next except StopIteration: break try: classifier_meta_objects = classifier_meta_objects.next except StopIteration: break ...
• NOTES
-
Using the snippet above, the number of classifications (
15249
as reported byreport_label_info_counter
) is slightly lower than the number of detections (16496
as reported byreport_obj_meta_counter
). There are cases whereobj_meta.classifier_meta_list
isNone
, instead of apyds.GList
. -
The same number of frames (
1088
) are being processed if I change the decoder fromjpegdec
tonvjpegdec
. If I turn onraw-output-tensors
, both the pgie and sgie output1088
, consistent with the number of frames. However, I loose 11 detections (from16496
to16485
. On the other hand, the number of classifications goes up from15249
to15252
. These numbers are consistent across >10 runs each. -
I can manually enforce the SGIE to “skip” detections, eg by increasing the
input-object-min-width
to200
. In that case, when also enablingraw-output-tensors
, the number of sgie output tensors decreases (to365
), which makes me doubt its a sgie config file issue… -
Other
- Using
dla
does not affect the numbers. -
network-mode
does not affect the numbers - Using
jpegdec
vsnvjpegdec
does, but slightly (maybe a sync or a flush thing?), and does not solve the problem.
- Using
• QUESTIONS
- What happened to those
obj_meta
which do not have correspondingclassifier_meta_list
? - Is this a
pyds
ornvinfer
issue?
• Hardware Platform (Jetson / GPU)
JETSON_TYPE=AGX Xavier [16GB]
JETSON_CHIP_ID=25
JETSON_SOC=tegra194
JETSON_MACHINE=NVIDIA Jetson AGX Xavier [16GB]
JETSON_CODENAME=galen
JETSON_BOARD=P2822-0000
JETSON_MODULE=P2888-0001
• DeepStream Version Version: 5.0 (GCID: 23607587)
• JetPack Version (valid for Jetson only) 4.4
• TensorRT Version : 7.1.3.0