Access Detection Confidence output by nvinfer (DeepStream 3.0)

Accelerated Computing Intelligent Video Analytics DeepStream SDK
#1

Detections from nvinfer are output to the Gstreamer metadata bus as NvDsFrameMeta structures. Unfortunately, they do not contain the confidence scores output by the model.

The gstdsexample shows that these confidences can be stored in the attr_info field. However, without the nvinfer source code, we can’t modify it do add this information. Is there any way to use callbacks/pad callbacks to add confidence information for detections to the metadata output by nvinfer? Or is there any other way I am missing to access the detection confidence scores outside of the nvinfer plugin?

#2

Can you get the detection confidence from NvDsFrameMeta->NvDsObjectParams->NvDsAttrInfo->NvDsAttr->attr_prob ?

#3

Hi daniel.suess,

Have you tried the suggestions?
Any result can be shared?

Thanks

#4
#5

If I remember correctly, all those attributes were set to 0. Unfortunately, we could not figure out any way to get to the confidence values and decided to postpone this work until after we’ve updated to DS4

#6

metadata of DS4.0 has a lot of improvement. You can get tensorRT/tensor output meta.

#7

I tried also.But the confidence value I got is 0…
And by the way.I set --print_prediction_info=true inside the yolov3.txt for deepstream-yolo-app.cpp.And then when I ran the program they printed the labels,confidences,bbox coordinates(xmin,ymin,xmax,ymax).But But But I use these code in deepstream-yolo-app.cpp in order to get such detection information.They cannot matched each other.Could you tell me the difference between them and which method is then right way to get bbox detection information?Very thanks…

while ((gst_meta = gst_buffer_iterate_meta (buf, &state))) {
    if (gst_meta_api_type_has_tag (gst_meta->info->api, _nvdsmeta_quark)) {

      nvdsmeta = (NvDsMeta *) gst_meta;

      /* We are interested only in intercepting Meta of type
       * "NVDS_META_FRAME_INFO" as they are from our infer elements. */
      if (nvdsmeta->meta_type == NVDS_META_FRAME_INFO) {
        frame_meta = (NvDsFrameMeta *) nvdsmeta->meta_data;
        if (frame_meta == NULL) {
          g_print ("NvDS Meta contained NULL meta \n");
          frame_number++;
          return GST_PAD_PROBE_OK;
        }

        if(output_kitti)
        {
          g_snprintf(bbox_file, sizeof(bbox_file) - 1, "%s/%06d.txt",
            output_kitti, frame_number);
            bbox_params_dump_file = fopen(bbox_file, "w");
        }

        num_rects = frame_meta->num_rects;

        /* This means we have num_rects in frame_meta->obj_params.
         * Now lets iterate through them and count the number of cars,
         * trucks, persons and bicycles in each frame */

        for (rect_index = 0; rect_index < num_rects; rect_index++) {
          obj_meta = (NvDsObjectParams *) & frame_meta->obj_params[rect_index];
          rect_params = &(obj_meta->rect_params);
          text_params = &(obj_meta->text_params);
          if (!g_strcmp0 (obj_meta->attr_info[YOLO_UNIQUE_ID].attr_label,
                  "car"))
            {
              // g_print("CAR topX=%d, topY=%d, width=%d, height=%d, score=%.2f\n", 
              // rect_params->left, rect_params->top, 
              // (rect_params->left + rect_params->width),
              // (rect_params->top + rect_params->height),
              // obj_meta->attr_info[YOLO_UNIQUE_ID].attrs->attr_prob);
              // g_print("txtTopLeft=%d, txtTopRight=%d\n", 
              // text_params->x_offset, text_params->y_offset);
              // g_print("attr_label=%s\n", 
              // obj_meta->attr_info[YOLO_UNIQUE_ID].attr_label);
              // g_print("CAR topX=%d, topY=%d, width=%d, height=%d\n", 
              // obj_meta->line_params[0].x1, obj_meta->line_params[0].y1,
              // obj_meta->line_params[0].x2, obj_meta->line_params[0].y2);
              if(bbox_params_dump_file)
              {
                int left = (int)(rect_params->left);
                int top = (int)(rect_params->top);
                int right = left + (int)(rect_params->width);
                int bottom = top + (int)(rect_params->height);
                int class_index = obj_meta->class_id;
                char *text = "car";
                fprintf(bbox_params_dump_file,
                  "%s  %d.00 %d.00 %d.00 %d.00 \n",
                  text, left, top, right, bottom);
                std::cout << text << " left=" << left << " top=" << top << 
                  " right=" << right << " bottom=" << bottom << std::endl;
              }
              car_count++;
            }
          else if (!g_strcmp0 (obj_meta->attr_info[YOLO_UNIQUE_ID].attr_label,
                  "person"))
            {
              // g_print("PERSON topX=%d, topY=%d, width=%d, height=%d, score=%.2f\n", 
              // rect_params->left, rect_params->top, 
              // (rect_params->left + rect_params->width),
              // (rect_params->top + rect_params->height),
              //  obj_meta->attr_info[YOLO_UNIQUE_ID].attrs->attr_prob);
              // g_print("CAR topX=%d, topY=%d, width=%d, height=%d\n", 
              // obj_meta->line_params[0].x1, obj_meta->line_params[0].y1,
              // obj_meta->line_params[0].x2, obj_meta->line_params[0].y2);
              if(bbox_params_dump_file)
              {
                int left = (int)(rect_params->left);
                int top = (int)(rect_params->top);
                int right = left + (int)(rect_params->width);
                int bottom = top + (int)(rect_params->height);
                int class_index = obj_meta->class_id;
                char *text = "person";
                fprintf(bbox_params_dump_file,
                  "%s  %d.00 %d.00 %d.00 %d.00 \n",
                  text, left, top, right, bottom);
                std::cout << text << " left=" << left << " top=" << top << 
                  " right=" << right << " bottom=" << bottom << std::endl;
              }
              person_count++;
            } 
          else if (!g_strcmp0 (obj_meta->attr_info[YOLO_UNIQUE_ID].attr_label,
                  "bicycle"))
            {
              // g_print("BICYCLE topX=%d, topY=%d, width=%d, height=%d, score=%.2f\n", 
              // rect_params->left, rect_params->top, 
              // (rect_params->left + rect_params->width),
              // (rect_params->top + rect_params->height),
              //  obj_meta->attr_info[YOLO_UNIQUE_ID].attrs->attr_prob);
              // g_print("CAR topX=%d, topY=%d, width=%d, height=%d\n", 
              // obj_meta->line_params[0].x1, obj_meta->line_params[0].y1,
              // obj_meta->line_params[0].x2, obj_meta->line_params[0].y2);
              bicycle_count++;
            }
          else if (!g_strcmp0 (obj_meta->attr_info[YOLO_UNIQUE_ID].attr_label,
                  "truck"))
           {
            //  g_print("TRUCK topX=%d, topY=%d, width=%d, height=%d, score=%.2f\n", 
            //   rect_params->left, rect_params->top, 
            //   (rect_params->left + rect_params->width),
            //   (rect_params->top + rect_params->height),
            //    obj_meta->attr_info[YOLO_UNIQUE_ID].attrs->attr_prob);
            // g_print("CAR topX=%d, topY=%d, width=%d, height=%d\n", 
            //   obj_meta->line_params[0].x1, obj_meta->line_params[0].y1,
            //   obj_meta->line_params[0].x2, obj_meta->line_params[0].y2);
              truck_count++;
           }
        }
        if(bbox_params_dump_file)
        {
          fclose(bbox_params_dump_file);
          bbox_params_dump_file = NULL;
        }

OUTPUT

Frame Number = 395 Number of objects = 4 Car Count = 2 Person Count = 1 Bicycle Count = 0 Truck Count = 0 
 label:0(person) confidence:0.964754 xmin:600.152 ymin:24.6138 xmax:617.771 ymax:71.5274
 label:2(car) confidence:0.995311 xmin:209.451 ymin:79.6301 xmax:503.994 ymax:288.228
 label:2(car) confidence:0.954675 xmin:429.626 ymin:-3.30851 xmax:523.656 ymax:82.3553
 label:58(pottedplant) confidence:0.736185 xmin:525.58 ymin:9.90438 xmax:593.096 ymax:56.2763
person left=750 top=30 right=771 bottom=87
car left=261 top=98 right=628 bottom=358
car left=536 top=1073741820 right=653 bottom=1073741926
Frame Number = 396 Number of objects = 4 Car Count = 2 Person Count = 1 Bicycle Count = 0 Truck Count = 0 
 label:0(person) confidence:0.963777 xmin:600.222 ymin:24.4306 xmax:617.637 ymax:71.4721
 label:2(car) confidence:0.995401 xmin:209.441 ymin:79.5833 xmax:503.964 ymax:288.249
 label:2(car) confidence:0.951221 xmin:429.66 ymin:-3.22625 xmax:523.695 ymax:82.3932
 label:58(pottedplant) confidence:0.739988 xmin:525.957 ymin:9.56697 xmax:592.888 ymax:56.1502
person left=750 top=30 right=771 bottom=88
car left=261 top=98 right=628 bottom=358
car left=536 top=1073741820 right=653 bottom=1073741926
Frame Number = 397 Number of objects = 4 Car Count = 2 Person Count = 1 Bicycle Count = 0 Truck Count = 0 
 label:0(person) confidence:0.963482 xmin:600.22 ymin:24.4311 xmax:617.636 ymax:71.4774
 label:2(car) confidence:0.995374 xmin:209.508 ymin:79.5012 xmax:503.877 ymax:288.232
 label:2(car) confidence:0.950079 xmin:429.695 ymin:-3.22583 xmax:523.647 ymax:82.463
 label:58(pottedplant) confidence:0.739842 xmin:525.956 ymin:9.56345 xmax:592.89 ymax:56.1504
person left=750 top=30 right=771 bottom=88
car left=261 top=98 right=628 bottom=358
car left=536 top=1073741820 right=652 bottom=1073741926
Frame Number = 398 Number of objects = 4 Car Count = 2 Person Count = 1 Bicycle Count = 0 Truck Count = 0 
 label:0(person) confidence:0.957847 xmin:600.241 ymin:23.9763 xmax:617.665 ymax:71.6299
 label:2(car) confidence:0.995379 xmin:209.448 ymin:79.5313 xmax:503.999 ymax:288.276
 label:2(car) confidence:0.949252 xmin:429.645 ymin:-3.14279 xmax:523.605 ymax:82.5501
 label:58(pottedplant) confidence:0.718636 xmin:526.122 ymin:9.55772 xmax:592.91 ymax:56.1201
person left=750 top=28 right=771 bottom=86
car left=261 top=98 right=628 bottom=358
car left=536 top=1073741820 right=652 bottom=1073741926
Frame Number = 399 Number of objects = 4 Car Count = 2 Person Count = 1 Bicycle Count = 0 Truck Count = 0 
 label:0(person) confidence:0.957746 xmin:600.241 ymin:23.9768 xmax:617.666 ymax:71.6298
 label:2(car) confidence:0.995387 xmin:209.443 ymin:79.5256 xmax:504.035 ymax:288.279
 label:2(car) confidence:0.949492 xmin:429.653 ymin:-3.15577 xmax:523.615 ymax:82.5767
 label:58(pottedplant) confidence:0.719251 xmin:526.123 ymin:9.55535 xmax:592.912 ymax:56.1231
person left=750 top=28 right=771 bottom=86
car left=261 top=98 right=628 bottom=358
car left=536 top=1073741820 right=652 bottom=1073741926
Frame Number = 400 Number of objects = 4 Car Count = 2 Person Count = 1 Bicycle Count = 0 Truck Count = 0 
 label:0(person) confidence:0.964895 xmin:600.153 ymin:23.703 xmax:617.701 ymax:71.7675
 label:2(car) confidence:0.995395 xmin:209.451 ymin:79.516 xmax:504.028 ymax:288.284
 label:2(car) confidence:0.951143 xmin:429.705 ymin:-3.29359 xmax:523.446 ymax:82.69
 label:58(pottedplant) confidence:0.742259 xmin:525.985 ymin:9.71285 xmax:592.927 ymax:56.4793
person left=750 top=28 right=771 bottom=88
car left=261 top=98 right=628 bottom=358
car left=536 top=1073741820 right=652 bottom=1073741926
Frame Number = 401 Number of objects = 4 Car Count = 2 Person Count = 1 Bicycle Count = 0 Truck Count = 0 
 label:0(person) confidence:0.964916 xmin:600.153 ymin:23.7028 xmax:617.7 ymax:71.7674
 label:2(car) confidence:0.995404 xmin:209.48 ymin:79.4941 xmax:503.985 ymax:288.291
 label:2(car) confidence:0.950972 xmin:429.705 ymin:-3.29434 xmax:523.44 ymax:82.6814
 label:58(pottedplant) confidence:0.742007 xmin:525.987 ymin:9.71174 xmax:592.926 ymax:56.4829
person left=750 top=28 right=771 bottom=88
car left=261 top=98 right=628 bottom=358
car left=536 top=1073741820 right=652 bottom=1073741926

Why???

#8

deepstream-yolo-app is based on deepstreaem 3.0. Don’t use it based on 4.0
You can file a new topic if still have problem.