Nvinfer multi-threading

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) Jetson
• DeepStream Version 5.1
• JetPack Version (valid for Jetson only) 4.5.1
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Hi,

I have one primary model and 8 other models running on deep-stream, the fps very low. Currently i am working on the optimization to increase the speed.
the primary model has two output (A and B )
five of the secondary models work on A output from primary
and the rest work on B output.

Currently they are working on sequential, how to guarantee the secondary models are working on parallel?
does tee or queue help us on this?

Yes, please have a try with tee.

https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_troubleshooting.html#the-deepstream-application-is-running-slowly

Could you please share an example of using tee?

You can search GStreamer tee example on Google. Can you share your use case? So we can have a check if tee can resolve your issue.


Secondary models are not dependent on each other ? can we run them in parallel?

Have you tried setting operate-on-class-ids parameter?
I think that should be able to do what you’re trying. But only difference is that it is not parallel like your diagram.

Say for secondary_model5, secondary_model6, secondary_model7 if you set operate-on-class-ids=2 and secondary_model1, secondary_model2, secondary_model3, secondary_model4 have operate-on-class-ids=1 then I think you should have the desired behaviour.

Yes this is what is implemented currently, but in sequential not in parallel. I want the same result on parallel

Can you share more on call Frame Parser Function?

FrameParser is tiler_src_pad_buffer_probe function for metadata parsing


static GstPadProbeReturn tiler_src_pad_buffer_probe(GstPad *pad,
                                                    GstPadProbeInfo *info,
                                                    gpointer u_data)
{

  NvDsFrameMeta *frame_meta = NULL;
  GstBuffer *buf = (GstBuffer *)info->data;
  guint num_rects = 0;
  NvDsObjectMeta *obj_meta = NULL;
  NvDsObjectMeta *obj_meta_767 = NULL;

  NvDsMetaList *l_frame = NULL;
  NvDsMetaList *l_obj = NULL;
  NvDsClassifierMetaList *l_classifier = NULL;
  NvDsClassifierMeta *class_meta = NULL;
  NvDsLabelInfoList *l_label = NULL;
  NvDsLabelInfo *label_info = NULL;

  NvDsBatchMeta *batch_meta = gst_buffer_get_nvds_batch_meta(buf);

  std::vector<vehicle> last_vehicle;

  std::vector<vehicle>  vehicles_list;
  std::string image_name = "";

  std::vector<bbox_ws> ws;
  std::vector<bbox_vh> vh;
  int number_of_plates=0;
  int number_of_vehicles=0;
  unsigned int vh_count = 0;
  for (l_frame = batch_meta->frame_meta_list; l_frame != NULL;
       l_frame = l_frame->next)
  {
    std::vector<Plate> plates;
 
    char img_number[7];
    sprintf(img_number, "%04d", counter);
    std::string img_number_str(img_number);
    image_name = "img_" + img_number_str + ".jpg";
    image_name = get_image_name(image_name);
    unsigned int numOfws = 0;
    unsigned int numOfvh = 0;

    frame_meta = (NvDsFrameMeta *)(l_frame->data);
    NvOSD_RectParams rect_params;

    for (l_obj = frame_meta->obj_meta_list; l_obj != nullptr;
         l_obj = l_obj->next)
    {
      obj_meta = (NvDsObjectMeta *)(l_obj->data);
      NvOSD_RectParams rect_params = obj_meta->rect_params;

      float detection_confidence = obj_meta->confidence;
      int plate_type = 0;

      float vh_ymin = 0.0;
      std::string plate_number = "";
      float plate_status_conf = 0.0;
      float vehicle_type_conf = 0.0;
      float vh_xmin = 0.0;
      float vh_width = 0.0;
      float vh_height = 0.0;
      float vh_ymax = 0.0;
      float vh_xmax = 0.0;

      //for plate detection

      float pl_xmin = 0.0;
      float pl_width = 0.0;
      float pl_height = 0.0;
      float pl_ymax = 0.0;
      float pl_xmax = 0.0;
      float pl_ymin = 0.0;

      float plate_recognition_confidence = 0.0;
      std::string plate_status = "";

      std::string vehicle_type = "";
      // plate detection
      if (obj_meta->unique_component_id == 4)
      {
        pl_ymin = obj_meta->rect_params.top;
        pl_xmin = obj_meta->rect_params.left;
        pl_width = obj_meta->rect_params.width;
        pl_height = obj_meta->rect_params.height;
        pl_ymax = pl_ymin + pl_height;
        pl_xmax = pl_xmin + pl_width;
        for (l_classifier = obj_meta->classifier_meta_list;
             l_classifier != NULL; l_classifier = l_classifier->next)
        {
          // plate recognition with type 
          class_meta = (NvDsClassifierMeta *)(l_classifier->data);
          if (class_meta->unique_component_id == 5)
          {
            l_label = class_meta->label_info_list;
            label_info = (NvDsLabelInfo *)(l_label->data);
            std::string plate_label = label_info->result_label;
              plate_number = plate_label.substr(0, 7);
              plate_type = stoi(plate_label.substr(8, 1));
              plate_recognition_confidence = label_info->result_prob;
          } // end class 5
          // plate status 
          if (class_meta->unique_component_id == 7)
          {

            l_label = class_meta->label_info_list;
            label_info = (NvDsLabelInfo *)(l_label->data);
            plate_status = label_info->result_label;
            plate_status_conf = label_info->result_prob;
          }
        }
                number_of_plates=number_of_plates+1;

        // vehicle detection
        NvDsClassifierMeta *class_meta_vehicle_type = NULL;
        if (obj_meta->parent != NULL)
        {
          vh_ymin = obj_meta->parent->rect_params.top;
          vh_xmin = obj_meta->parent->rect_params.left;
          vh_width = obj_meta->parent->rect_params.width;
          vh_height = obj_meta->parent->rect_params.height;
          vh_ymax = vh_ymin + vh_height;
          vh_xmax = vh_xmin + vh_width;
          for (l_classifier = obj_meta->parent->classifier_meta_list;
               l_classifier != NULL; l_classifier = l_classifier->next)
          {

            class_meta = (NvDsClassifierMeta *)(l_classifier->data);
            // vehicle type
            if (class_meta->unique_component_id == 6)
            {
              l_label = class_meta->label_info_list;
              label_info = (NvDsLabelInfo *)(l_label->data);
              vehicle_type = label_info->result_label;
              vehicle_type_conf = label_info->result_prob;
            }

            Plate *plate_ob = new Plate(
                obj_meta->object_id, plate_number, detection_confidence,
                plate_recognition_confidence, 0.0, plate_type, pl_xmin, pl_ymin, pl_xmax, pl_ymax);
            bbox_vh *ob = new bbox_vh(vh_xmin, vh_xmax, vh_ymin, vh_ymax,
                                      vehicle_type, vehicle_type_conf,
                                      plate_status, plate_status_conf);
            ob->plate = plate_ob;
            numOfvh++;

            vh.push_back(*ob);
          }
          // end plate detection and vehicle
        }
      }
      if (obj_meta->unique_component_id == 1)
      {

        float seatbelt_conf = 0.0;
        float mobile_conf = 0.0;
        float image_q_conf = 0.0;
        // vh
        if (obj_meta->class_id == 0)
        {
          number_of_vehicles=number_of_vehicles+1;
        }
        // ws
        if (obj_meta->class_id == 1)
        {


          float ws_xmin = 0.0;
          float ws_width = 0.0;
          float ws_height = 0.0;
          float ws_ymax = 0.0;
          float ws_xmax = 0.0;
          float ws_ymin = 0.0;

          ws_ymin = obj_meta->rect_params.top;
          ws_xmin = obj_meta->rect_params.left;
          ws_width = obj_meta->rect_params.width;
          ws_height = obj_meta->rect_params.height;
          ws_ymax = ws_ymin + ws_height;
          ws_xmax = ws_xmin + ws_width;

          for (l_classifier = obj_meta->classifier_meta_list;
               l_classifier != NULL; l_classifier = l_classifier->next)
          {  // mobile violation 
            class_meta = (NvDsClassifierMeta *)(l_classifier->data);
            if (class_meta->unique_component_id == 2)
            {
              for (l_label = class_meta->label_info_list; l_label != NULL;
                   l_label = l_label->next)
              {
                label_info = (NvDsLabelInfo *)(l_label->data);
                mobile_conf = label_info->result_prob;
              }
            }// seat belt violation 
            else if (class_meta->unique_component_id == 3)
            {
              l_label = class_meta->label_info_list;
              label_info = (NvDsLabelInfo *)(l_label->data);
              seatbelt_conf = label_info->result_prob;
            } // image quality 
            else if (class_meta->unique_component_id == 8)
            {
              l_label = class_meta->label_info_list;
              label_info = (NvDsLabelInfo *)(l_label->data);
              image_q_conf = label_info->result_prob;
            }
          }

          bbox_ws *ob = new bbox_ws(ws_xmin, ws_xmax, ws_ymin, ws_ymax,
                                    seatbelt_conf, mobile_conf, image_q_conf);
          ws.push_back(*ob);
          numOfws++;
        }
      }

    } // end object loob

  } /// end frameloop

  for (bbox_vh &vhob : vh)
  {

    for (bbox_ws &wsob : ws)
    {
      if ((vhob.get_xmin() < wsob.get_xmin()) and
          (vhob.get_ymin() < wsob.get_ymin()))
      {

        if ((wsob.get_xmax() < vhob.get_xmax()) and
            (wsob.get_ymax() < vhob.get_ymax()))
        {
          vehicle *ob = new vehicle(
              vhob.get_xmin(), vhob.get_xmax(), vhob.get_ymin(),
              vhob.get_ymax(), wsob.get_xmin(), wsob.get_xmax(),
              wsob.get_ymin(), wsob.get_ymax(), wsob.get_seat_belt(),
              wsob.get_mobile(), vhob.get_vehicle_type(),
              vhob.get_vehicle_type_conf(), vhob.get_plate_status(),
              vhob.get_plate_status_conf(), wsob.get_image_quality());
          
          ob->bbx_vh->plate = vhob.plate;

          vehicles_list.push_back(*ob);

          break;
        }
      }
    }

    vh_count++;
  }

Do you have any requirement to sync between the parallel pipeline? Or the parallel pipeline will link to fakesink.

parallel pipeline will link to fakesink.

Got it, seems your design is fine.

For me is very slow. the question is can I make it faster by run the models in parallel?? and how ? because now it is sequential.

Below is for performance tuning:

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.