NvBufSurface access stalling pipeline

• Hardware Platform GPU
• DeepStream 6.2

Hello, I’m creating a C/C++ deepstream pipeline where my nvinfer can be dynamically removed/added at runtime, I placed the nvinfer(segmentation model) in a bin which gets removed/added like so:

queue → |||| BIN START ||||| → nvinfer → nvvideoconvert → capsfilter → nvvideoconvert (probe_here) → capsfilter → |||| BIN END ||||| → tiler

I also a probe on the nvvideoconvert element src pad to draw the mask. The pipeline works perectly fine.

These are the function responsible for creating, adding, and removing the bin:

Creating:

void App::CreateBin() {
  GstElement *infer, 
             *filter,
             *conv,
             *conv_out,
             *filter_out;

  GstCaps *caps, *caps_out;
  
  ...

  inference_bin_ = gst_bin_new("infer_bin");
  infer          = gst_element_factory_make("nvinfer", "inference");

  filter = gst_element_factory_make("capsfilter", "post-infer-filter");
  conv   = gst_element_factory_make("nvvideoconvert", "post-infer-conv");
  caps   = gst_caps_from_string("video/x-raw(memory:NVMM),\
                                  format=(string)NV12");

  g_object_set(G_OBJECT(filter), "caps", caps, NULL);

  conv_out   = gst_element_factory_make("nvvideoconvert", "out-conv");
  filter_out = gst_element_factory_make("capsfilter", "out-filter");
  caps_out   = gst_caps_from_string("video/x-raw(memory:NVMM),\
                                  format=(string)RGBA");

  g_object_set(G_OBJECT(filter_out), "caps", caps_out, NULL);
  g_object_set(G_OBJECT(conv_out), "nvbuf-memory-type", 3, NULL);

  queue      = gst_element_factory_make("queue", "queue-segmentation");

  auto create_probe = [&](GstElement* element) {
    GstPad *src_pad = gst_element_get_static_pad(element, "src");
    gst_pad_add_probe(src_pad , GST_PAD_PROBE_TYPE_BUFFER,
        c_src_probe, this, nullptr);
    gst_object_unref(src_pad );
  };

  if (state == State::On) {
    gst_bin_add_many(GST_BIN(inference_bin_), infer, conv, filter, conv_out, filter_out, NULL);
    gst_element_link_many(infer, conv, filter, conv_out, filter_out,NULL);

    create_probe (conv_out);

    AddSinkGhost(GST_BIN(inference_bin_), infer);
    AddSourceGhost(GST_BIN(inference_bin_), filter_out);
  }
  else {
    std::cout << "Switched OFF" << std::endl;
    inference_bin_ = nullptr;
  }
}

Adding:

void 
App::AddBin() {
  CreateInferBin();
  if (inference_bin_ == nullptr) {
    // nothing
  } else {
    gulong queue_src_pad_id;
    GstPad *queue_src,
           *tiler_sink,
           *infer_sink,
           *infer_src;

    gst_bin_add(GST_BIN(pipeline_), inference_bin_);
    queue_src  = gst_element_get_static_pad(infer_queue_, "src");
    tiler_sink = gst_element_get_static_pad(tiler_, "sink");
    infer_sink = gst_element_get_static_pad(inference_bin_, "sink");
    infer_src  = gst_element_get_static_pad(inference_bin_, "src");

    queue_src_pad_id = gst_pad_add_probe(queue_src,  
	  (GstPadProbeType) (GST_PAD_PROBE_TYPE_BLOCK |	
            GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM), 
	  NULL, NULL, NULL);

    Pause();
    gst_pad_unlink(queue_src, tiler_sink);
    gst_element_link_many(infer_queue_, inference_bin_, tiler_, NULL);
    Play();
    gst_pad_remove_probe(queue_src, queue_src_pad_id);

    // Unref
    ...
  }
}

Removing:

void
App::RemoveBin() {
  GstPad *queue_src,
         *tiler_sink,
         *infer_sink,
         *infer_src;
         
  if (inference_bin_ == nullptr) return;
  
  queue_src  = gst_element_get_static_pad(infer_queue_, "src");
  tiler_sink = gst_element_get_static_pad(tiler_, "sink");
	infer_sink = gst_element_get_static_pad(inference_bin_, "sink");
	infer_src  = gst_element_get_static_pad(inference_bin_, "src");

  Pause();
  gst_pad_unlink(queue_src, infer_sink);
  gst_pad_unlink(infer_src, tiler_sink);
  gst_pad_link(queue_src, tiler_sink);

  gst_element_set_state(inference_bin_, GST_STATE_NULL);
  gst_bin_remove(GST_BIN(pipeline_), inference_bin_);
  Play();

  // Unref
  ...
}

This is the probe:

GstPadProbeReturn
DGpuPipeline::SrcProbe(GstPad *pad, GstPadProbeInfo *info) {
  ...

  output_buffer = GST_PAD_PROBE_INFO_BUFFER(info);
  output_buffer = gst_buffer_make_writable(output_buffer);

  if (!gst_buffer_map(output_buffer, &output_map, GST_MAP_WRITE)) {
    std::cout << "buffer failed to map" << std::endl;
    gst_buffer_unmap(output_buffer, &output_map);
    return GST_PAD_PROBE_OK;
  }

  dest_surface = (NvBufSurface*)output_map.data;
  gst_buffer_unmap(output_buffer, &output_map);

  batch_meta = gst_buffer_get_nvds_batch_meta(output_buffer);
  dest_surface->numFilled = batch_meta->num_frames_in_batch;
  if (NvBufSurfaceMap(dest_surface, -1, -1, NVBUF_MAP_WRITE) != 0)
    return GST_PAD_PROBE_OK;

  for (int i = 0; i < batch_meta->num_frames_in_batch; i++) {
    ...
        buffer = (unsigned char*)malloc(4 * frame_width * frame_height);
        for (unsigned int h = 0; h < frame_height; h++) {
          memcpy(buffer + h * frame_width * 4, 
            (char*)dest_surface->surfaceList[i].mappedAddr.addr[0] + h * 
            dest_surface->surfaceList[i].planeParams.pitch[0],
            frame_width * 4);
        }


        ApplyResults(frame_width, frame_height, seg_width, seg_height,
         buffer, seg_meta->class_map);

        for (unsigned int h = 0; h < frame_height; h++) {
            memcpy((char*)dest_surface->surfaceList[i].mappedAddr.addr[0] + h * 
              dest_surface->surfaceList[i].planeParams.pitch[0],
              buffer + h * frame_width * 4, frame_width * 4); 
          }
          free(buffer);
    ...
  }

  NvBufSurfaceUnMap (dest_surface, -1, -1);
  return GST_PAD_PROBE_OK;
}

However, when I remove the nvinfer and just place the probe on a nvvideoconvert element, the pipeline works when the bin is added, but when it gets removed it stalls until I kill the application and restart it.

void App::CreateBin() {
  ...

  if (state == State::On) {
    gst_bin_add_many(GST_BIN(inference_bin_), conv, filter, conv_out, filter_out, NULL);
    gst_element_link_many(conv, filter, conv_out, filter_out,NULL);

    create_probe (conv_out);

    AddSinkGhost(GST_BIN(inference_bin_), conv);
    AddSourceGhost(GST_BIN(inference_bin_), filter_out);
  }
  else {
    std::cout << "Switched OFF" << std::endl;
    inference_bin_ = nullptr;
  }
}

In the log, there are no errors, the pipeline goes to paused and never returns to the playing state. The problem goes away when I remove the probe.

why do you need to remove/add nvinfer at runtime? could you share your scenario? if don’t want to do inference, you can change nvinfer’s interval dynamically.

I’m not just disabling inference and enabling it, in my scenario, I’m also allowing the user to pick and change models at runtime

about “change models at runtime”, please refer to on-the-fly-model-update

if adding a empty probe, is the app ok? can you simplify the probe function to check which code will cause the problem?

Can you explain how this dynamic modification is done?

nviner plugin is opensource. it has a property “interval”, which means “Specifies the number of consecutive batches to be skipped for inference”.
you can modify this value to turn on/off the inference. here s a sample.

g_object_set (G_OBJECT (pgie),  "interval", G_MAXINT, NULL); //turn off
g_object_set (G_OBJECT (pgie),  "interval", 0, NULL); //turn on
1 Like

Do I need to pause the pipeline and set the interval or can I do this while the pipeline is in playing state?

There is no update from you for a period, assuming this is not an issue any more. Hence we are closing this topic. If need further support, please open a new one. Thanks.
sorry for the late reply. no, you don’t need to pause the pipeline after setting interval. please refer to skip_batch in gst_nvinfer_process_full_frame.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.