Where is the classified meta data

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

According to the file gstnvinfer_meta_utils.cpp, the classification plugin attatches the meta as following,

frame.frame_meta->base_meta.batch_meta->object_meta->rect_params
text_params
object_id = UNTRACKED_OBJECT_ID
class_id = -1
classifier_meta->unique_component_id
label_info->label_id = object_info.attributes[i].attributeIndex
result_class_id = object_info.attributes[i].attributeValue
result_prob = object_info.attributes[i].attributeConfidence
result_label = object_info.attributes[i].attr.attributeLabel
label_info->label_id = object_info.attributes[i].attributeIndex
result_class_id = object_info.attributes[i].attributeValue
result_prob = object_info.attributes[i].attributeConfidence
result_label = object_info.attributes[i].attr.attributeLabel

It seems defferent with the flow of the program osd_sink_pad_buffer_probe. According to osd_sink_pad_buffer_probe, the flow is batch_meta->frame_meta->object_meta.

ps. Where is the classified meta data? It is attatched to frame_meta, or object_meta? How to get the classified id of every frame?

3

There is NvDsUserMetaList *obj_user_meta_list in struct NvDsObjectMeta.

https://docs.nvidia.com/metropolis/deepstream/sdk-api/Meta/_NvDsObjectMeta.html

Classifier meta data is for every object but not for the whole frame. You can get NvDsClassifierMeta from obj_user_meta_list in each NvDsObjectMeta

4

I am classifying every frame, so the nvinfer plugin work on full frame. The classifying model is the primary model. So I think Classifier meta should be attached to frame_meta. Is it still in object_meta? How to read the classifying result?

5

So there is only one object in the frame.

The classifier result will be attached to the only object’s meta data.

6

Thanks a lot.

7

Are you sure that there is only one object in a frame. Why do some frames has 2 objects and some frames has no objects?
I test this by the following program:

GstBuffer *buf = (GstBuffer *) info->data;
  NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta (buf);
  NvDsMetaList *l_frame = NULL;
  NvDsMetaList *l_obj = NULL;
  NvDsClassifierMetaList *l_classifier = NULL;
  NvDsObjectMeta *obj_meta = NULL;
  NvDsLabelInfo *classifier_meta = NULL;
  NvDsDisplayMeta *display_meta = NULL;

  g_print("num_frames_in_batch: %d\n", batch_meta->num_frames_in_batch);
  for (l_frame = batch_meta->frame_meta_list; l_frame != NULL; l_frame = l_frame->next) {
      NvDsFrameMeta *frame_meta = (NvDsFrameMeta *) (l_frame->data);
      g_print("the %d frame\n", frame_meta->frame_num);
      int offset = 0;
      for (l_obj = frame_meta->obj_meta_list; l_obj != NULL; l_obj = l_obj->next) {
          NvDsObjectMeta *obj_meta = (NvDsObjectMeta *) (l_obj->data);
          g_print("the %d object\n", obj_meta->object_id);
          for (l_classifier = obj_meta->classifier_meta_list; l_classifier != NULL; l_classifier = l_classifier->next) {
            NvDsLabelInfo *labelinfo = (NvDsLabelInfo *) l_classifier->data;
            int val_id = labelinfo->result_class_id;
            g_print("belong to the %d type. \n", val_id);
          }
      }
}

And the output is following

…
num_frames_in_batch: 1
the 0 frame
the -1 object
*belong to the 0 type. *
the -1 object
*belong to the 6933 type. *
num_frames_in_batch: 1
the 1 frame
the -1 object
*belong to the 0 type. *
the -1 object
*belong to the 6933 type. *
num_frames_in_batch: 1
the 2 frame
the -1 object
*belong to the 0 type. *
the -1 object
*belong to the 6933 type. *
num_frames_in_batch: 1
the 3 frame
the -1 object
*belong to the 0 type. *
the -1 object
*belong to the 6933 type. *
num_frames_in_batch: 1
the 4 frame
num_frames_in_batch: 1
the 5 frame
the -1 object
*belong to the 0 type. *
the -1 object
*belong to the 0 type. *
num_frames_in_batch: 1
the 6 frame
the -1 object
*belong to the 0 type. *
the -1 object
*belong to the 0 type. *
num_frames_in_batch: 1
the 7 frame
num_frames_in_batch: 1
the 8 frame
num_frames_in_batch: 1
the 9 frame
num_frames_in_batch: 1
the 10 frame
num_frames_in_batch: 1
the 11 frame
num_frames_in_batch: 1
the 12 frame
num_frames_in_batch: 1
the 13 frame
num_frames_in_batch: 1
the 14 frame
num_frames_in_batch: 1
the 15 frame
num_frames_in_batch: 1
the 16 frame
num_frames_in_batch: 1
the 17 frame
num_frames_in_batch: 1
…

It seems that some frames have 2 objects, and some frame have no object without the line “belong to the *** type”. I mistake something?
Any help is appreciated.

8

I don’t know anything about your model. The results come from your model, please check with your model.

9

There is only a classified mode which works on full frame. In fact, I am testing this by the Secondary_CarColor model which is provided by deepstream. originally it is usred as secondary model. Now it are used as primary model and is the only model.

Even the object metadata are output by the model, why there is no any object metadata in the 13 frame, 14 frame, 15frame, 16 frame. It should has at least 1 object metadata.

Thanks a lot.

10

What is the nvinfer config file? This model is designed and trained by car images(single car), I’m not sure what will happen if the input image is not a single car.

If there is no output from the model, there is no output meta data. Why do you think the model must have 1 object metadata？

11

This is the nvinfer config file. I donot care what is output. I just want to get the classifier result, and send it to nvmsgbroker.

[property]
gpu-id=0
net-scale-factor=1
model-file=…/…/…/…/samples/models/Secondary_CarColor/resnet18.caffemodel
proto-file=…/…/…/…/samples/models/Secondary_CarColor/resnet18.prototxt
model-engine-file=…/…/…/…/samples/models/Secondary_CarColor/resnet18.caffemodel_b16_gpu0_int8.engine
mean-file=…/…/…/…/samples/models/Secondary_CarColor/mean.ppm
labelfile-path=…/…/…/…/samples/models/Secondary_CarColor/labels.txt
int8-calib-file=…/…/…/…/samples/models/Secondary_CarColor/cal_trt.bin
force-implicit-batch-dim=1
batch-size=16
# 0=FP32 and 1=INT8 mode
network-mode=0
process-mode=1
model-color-format=0
gie-unique-id=1
is-classifier=1
output-blob-names=predictions/Softmax
classifier-async-mode=0
classifier-threshold=0.51
#scaling-filter=0
#scaling-compute-hw=0

Since the nvinfer works on the full frame, every frame should has a classified result. According to your opinion, it is attached to a object metadata, and the object metadata is attached to the frame metadata. So I think it is the value val_id = labelinfo->result_class_id and print this value and frame No.

for (l_obj = frame_meta->obj_meta_list; l_obj != NULL; l_obj = l_obj->next) {
      NvDsObjectMeta *obj_meta = (NvDsObjectMeta *) (l_obj->data);
      g_print(“the %d object\n”, obj_meta->object_id);
      for (l_classifier = obj_meta->classifier_meta_list; l_classifier != NULL; l_classifier = l_classifier->next) {
               NvDsLabelInfo *labelinfo = (NvDsLabelInfo *) l_classifier->data;
               int val_id = labelinfo->result_class_id;
               g_print(“belong to the %d type. \n”, val_id);
      }
}

Referenced the results, some frames has 2 objects with same object_id -1, and some has no any object.
Any mistake in this thought?

12

The output depends on the model output. It is possible there is no output from the model or there is wrong output from the model.