Tracker hanging on cleanup

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

**• Hardware Platform ** GPU
• DeepStream Version 6.4
• TensorRT Version Using official docker image
• NVIDIA GPU Driver Version (valid for GPU only) 555.58.02
• Issue Type( questions, new requirements, bugs) bug
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Hello,
I am facing a bug using deepstream’s nvtracker along with appsink.

On stopping the pipeline setting the state to NULL, the nvtracker thread will hang in the function TrackerMiscData::deInit probably on the m_Cond.wait(lock) statement.

This happens on multiple different pipelines, but all have this tracker in common.
appsource → queue → nvinfer → queue → nvtracker ->queue → nvanalytics → queue → nvtiler → converter → nvosd → converter → appsink

Now if I remplace the appsink by fakesink, filesink. or nveglglessink, it works fine. Same if I remove the tracker from the pipeline.

  g_object_set(G_OBJECT(_sink), "sync", false, "async", false, "wait-on-eos",
               true, "drop", true, "max-buffers", 60, nullptr);
  g_object_set(_sink, "emit-signals", TRUE, nullptr);
  g_object_set(G_OBJECT(_sink), "enable-last-sample", false, nullptr);

those are the appsink parameters (already tried to mix and match…)

GstFlowReturn VideoShmemWriter::toShmem(GstElement *sink,
                                        const ShmemWriter *shmemWriter) {
  GstSample *sample;

  sample = gst_app_sink_pull_sample(GST_APP_SINK(sink));

  if (gst_app_sink_is_eos(GST_APP_SINK(sink))) {
    spdlog::info("EOS received in Appsink");
  }

  if (sample) {
    GstBuffer *buffer = gst_sample_get_buffer(sample);
    GstMapInfo info;
    gst_buffer_map(buffer, &info, GST_MAP_READ);
    unsigned char *buf = info.data;

    if (shmemWriter->_frame && shmemWriter->_resource) {
      spdlog::debug("Writing video frame to {}", shmemWriter->_shmem);
      shmemWriter->_frame->SetSamplesPtr(buf, true, false);
      shmemWriter->_resource->WriteVideoFrame(shmemWriter->_shmem,
                                              shmemWriter->_frame);
    }
    gst_buffer_unmap(buffer, &info);
    //gst_buffer_unref(buffer);
    gst_sample_unref(sample);

    return GST_FLOW_OK;
  }

  gst_sample_unref(sample);
  spdlog::error("No new sample to pull");
  return GST_FLOW_ERROR;
}

The code above is the appsink’s callback.
I did try to unref the buffer, hopping to free the memory from the NvDS MetaPool, but I got some complains from gstreamer with the following error assertion 'GST_MINI_OBJECT_REFCOUNT_VALUE (mini_object) > 0' failed

I assume I am either missing something in my appsink or there is an issue in how the cleaning is implemented for the tracker, anyway I am a bit lost here. If there is a fix please let me know.

Updating the deesptream version is not an option.

Thanks

3

Can you have a try to move the code which in function gst_nv_tracker_stop () to the begin of function gst_nv_tracker_finalize ()?

4 
static gboolean gst_nv_tracker_stop (GstBaseTransform * btrans)
{
  GstNvTracker *nvtracker = GST_NVTRACKER (btrans);


  if (nvtracker->output_loop) {
    g_thread_join(nvtracker->output_loop);
  }
  nvtracker->output_loop = NULL;

  delete nvtracker->trackerIface;
  nvtracker->trackerIface = NULL;

  return TRUE;
}

static void gst_nv_tracker_finalize (GObject * object)
{
  GstNvTracker *nvtracker = GST_NVTRACKER (object);

  /** De-init the low-level threads and plugin */
  nvtracker->trackerIface->deInit();

  /** Terminate the output loop
   * Note: This needs to be done AFTER plugin deInit() process
   * so that all the pending buffers can be returned properly
   */
  nvtracker->running = FALSE;
  if (nvtracker->trackerConfig.trackerLibFile != NULL) {
      g_free(nvtracker->trackerConfig.trackerLibFile);
      nvtracker->trackerConfig.trackerLibFile = NULL;
  }

  if (nvtracker->trackerConfig.trackerConfigFile != NULL) {
      g_free(nvtracker->trackerConfig.trackerConfigFile);
      nvtracker->trackerConfig.trackerConfigFile = NULL;
  }

  if (nvtracker->trackerConfig.trackerConfigFileList != NULL) {
      g_free(nvtracker->trackerConfig.trackerConfigFileList);
      nvtracker->trackerConfig.trackerConfigFileList = NULL;
  }
  for(auto it=nvtracker->trackerConfig.trackerConfigFilePerSubBatch.begin();
      it!=nvtracker->trackerConfig.trackerConfigFilePerSubBatch.end(); it++)
  {
    it->clear();
  }
  nvtracker->trackerConfig.trackerConfigFilePerSubBatch.clear();

  if (nvtracker->trackerConfig.gstName != NULL) {
      g_free(nvtracker->trackerConfig.gstName);
      nvtracker->trackerConfig.gstName = NULL;
  }

  for(auto it=nvtracker->trackerConfig.subBatchesConfig.begin(); it!=nvtracker->trackerConfig.subBatchesConfig.end(); it++)
  {
    it->clear();
  }
  nvtracker->trackerConfig.subBatchesConfig.clear();

  g_cond_clear(&nvtracker->eventCondition);
  g_mutex_clear(&nvtracker->eventLock);
  G_OBJECT_CLASS (parent_class)->finalize (object);
}

Like this ?
If yes, it hangs at the g_thread_join

5

gst_nv_tracker_stop () will be like below after you move all the code to the begin of gst_nv_tracker_finalize ().

static gboolean gst_nv_tracker_stop (GstBaseTransform * btrans)
{
  return TRUE;
}
6

That worked

Is this a known issue fixed in the next releases ?

I can apply the fix and recompile the plugin when I build my docker image, but is this going to be fixed (if it’s viewed as a bug) in the base library ?

Thanks for the help

7

Yes, the fix will be included in the next release.