Secondary models not working when using custom nvinferserver model

EDIT : I found the issue and a solution. But I am not sure why the solution is correct. The question now is: what is the equivalent of pgie.set_property("gie-unique-id", 1) for nvinferserver? It seems that this only works with nvinfer as nvinferserver does not have this property. Setting unique_id: 1 in infer_config of the triton model does not seem to have any effect.

I have a Python Deepstream 6.0 pipelines that involves an object detection model, and two classification models (no tracker for now).

Simply put, after the video decoding, there are the following elements:

... -> streammux -> pgie -> sgie1 -> sgie2 -> streamdemux -> nvvidconvs -> nvods -> ...

I am using the secondary models included in Deepstream at /opt/nvidia/deepstream/deepstream-6.0/samples/models/Secondary_CarColor/resnet18.caffemodel and /opt/nvidia/deepstream/deepstream-6.0/samples/models/Secondary_VehicleTypes/resnet18.caffemodel.

My goal is to use a primary custom object detection model running on triton server. I have been able to do this, but when with this model, it seems that the secondary models are not working as I am not able to retrieve their metadata. The metadata for the primary model are fine, as I am able to output a live RTSP video stream with correct bounding boxes.

However, the pipeline works perfectly when I use the object detection model provided with deepstream at /opt/nvidia/deepstream/deepstream-6.0/samples/models/Primary_Detector/resnet10.caffemodel with nvinfer element. Using a probe on the sink pad of the nvods element I am able to retrieve all metadata about the object, including the secondary models attribute. But as I wrote above, when using the custom triton model, it seems that the secondary models are not working as there are not secondary metadata while the pgie.

Issue summary

  • Secondary models are working with standard deepstream pgie that uses nvinfer
  • Secondary models are not working with my custom pgie that uses nvinferserver

Could you help me understand why is that?

Things I checked

From the secondary models settings:

gie-unique-id=2
operate-on-gie-id=1
operate-on-class-ids=0;1;2

Therefore, in my pgie model config I included:

infer_config {
  unique_id: 1     # is this correct? does this correspond to the operate-on-gie-id in the secondary models?
  ...

Also, I am assuming that classes operate-on-class-ids0;1;2 will refer to the first 3 classes of the labels.txt file of my custom triton server model. Is this correct?

As said above, I made sure that the my custom object detection model included the correct bounding boxes (I output a live RTSP with bounding boxes).

I also printed the metadata of the detection for the custom model to confirm that my probe added them properly after the postprocessing from the primary model. Looking at the metadata below:

  • class_id is 2 so the model should run on it
  • height and width are greater than min_height and min_width for the secondary model
  • Is the unique_component_id=0 relevant? EDIT: this seems to be the issue! If I set in the secondary model operate-on-gie-id=0 the secondary models work. However, why is that? I thought that unique_id: 1 in the infer_config of the triton model would set this value to 1.
{'class_id': 2,
 'classifier_meta_list': [],
 'confidence': 0.884190559387207,
 'misc_obj_info': array([0, 0, 0, 0], dtype=int32),
 'obj_label': 'car',
 'obj_user_meta_list': {},
 'object_id': 18446744073709551615,
 'parent': None,
 'rect_params': {'bg_color': <pyds.NvOSD_ColorParams object at 0x7f2e4d4f87b0>,
                 'border_color': <pyds.NvOSD_ColorParams object at 0x7f2e4d4f8c30>,
                 'border_width': 3,
                 'color_id': 0,
                 'has_bg_color': 0,
                 'has_color_info': 0,
                 'height': 179.0,
                 'left': 444.0,
                 'reserved': 0,
                 'top': 630.0,
                 'width': 263.0},
 'reserved': array([0, 0, 0, 0], dtype=int32),
 'text_params': <pyds.NvOSD_TextParams object at 0x7f2e4d4f89b0>,
 'unique_component_id': 0}

I compared this with the metadata from the deepstream pgie and they look alike apart from the lack of the secondary models metadata above (note: the metadata do not refer to the same object so the bounding boxes are different). Yeah, the class-id is different but the secondary models should be set to run on class ids 0;1;2:

{'class_id': 0,
 'classifier_meta_list': [{'result_label': 'sedan',
                           'result_prob': 0.8264346122741699},
                          {'result_label': 'grey',
                           'result_prob': 0.9371008276939392}],
 'confidence': 0.8414442539215088,
 'misc_obj_info': array([0, 0, 0, 0], dtype=int32),
 'obj_label': 'Car',
 'obj_user_meta_list': {},
 'object_id': 18446744073709551615,
 'parent': None,
 'rect_params': {'bg_color': <pyds.NvOSD_ColorParams object at 0x7f4ce8495af0>,
                 'border_color': <pyds.NvOSD_ColorParams object at 0x7f4ce849aeb0>,
                 'border_width': 3,
                 'color_id': 0,
                 'has_bg_color': 0,
                 'has_color_info': 0,
                 'height': 188.39158630371094,
                 'left': 797.4658813476562,
                 'reserved': 0,
                 'top': 645.3853759765625,
                 'width': 215.25466918945312},
 'reserved': array([0, 0, 0, 0], dtype=int32),
 'text_params': <pyds.NvOSD_TextParams object at 0x7f4ce84a2230>,
 'unique_component_id': 1}

Note
I am not including a tracker in this question, so try to focus on a single problem. But I noticed that if I add a tracker between the primary and secondary models, when using the custom nvinferserver model all the objects seem to disappear after the tracker. However, with the object detection model included in deepstream, the tracker works fine. The tracker is not the primary issue of this question. I am including this details as I am note sure if this can help you to understand the issue.

Appendix

[property]
gpu-id=0
net-scale-factor=1
model-file=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Secondary_CarColor/resnet18.caffemodel
proto-file=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Secondary_CarColor/resnet18.prototxt
model-engine-file=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Secondary_CarColor/resnet18.caffemodel_b16_gpu0_int8.engine
mean-file=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Secondary_CarColor/mean.ppm
labelfile-path=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Secondary_CarColor/labels.txt
int8-calib-file=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Secondary_CarColor/cal_trt.bin
force-implicit-batch-dim=1
batch-size=16
# 0=FP32 and 1=INT8 mode
network-mode=1
input-object-min-width=64
input-object-min-height=64
process-mode=2
model-color-format=1
gpu-id=0
gie-unique-id=2
operate-on-gie-id=1
operate-on-class-ids=0;1;2
is-classifier=1
output-blob-names=predictions/Softmax
classifier-async-mode=0
classifier-threshold=0.51
process-mode=2
#scaling-filter=0
#scaling-compute-hw=0
[property]
gpu-id=0
net-scale-factor=1
model-file=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Secondary_VehicleTypes/resnet18.caffemodel
proto-file=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Secondary_VehicleTypes/resnet18.prototxt
model-engine-file=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Secondary_VehicleTypes/resnet18.caffemodel_b16_gpu0_int8.engine
mean-file=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Secondary_VehicleTypes/mean.ppm
labelfile-path=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Secondary_VehicleTypes/labels.txt
int8-calib-file=/opt/nvidia/deepstream/deepstream-6.0/samples/models/Secondary_VehicleTypes/cal_trt.bin
force-implicit-batch-dim=1
batch-size=16
# 0=FP32 and 1=INT8 mode
network-mode=1
input-object-min-width=64
input-object-min-height=64
model-color-format=1
process-mode=2
gpu-id=0
gie-unique-id=4
operate-on-gie-id=1
operate-on-class-ids=0;1;2
is-classifier=1
output-blob-names=predictions/Softmax
classifier-async-mode=0
classifier-threshold=0.51
process-mode=2
#scaling-filter=0
#scaling-compute-hw=0

The solution is to explicitly set obj_meta.unique_component_id = 1 before calling pyds.nvds_add_obj_meta_to_frame(frame_meta, obj_meta, None) when creating the metadata object.

Good, thanks for the update!

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.