BUG REPORT: Wrong reference on Stream EOS message of NvDsStreamMux

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) Tesla T4
• DeepStream Version 5.0
• JetPack Version (valid for Jetson only)
• TensorRT Version Same as deepstream devel docker
• NVIDIA GPU Driver Version (valid for GPU only) 10.2

My application set up is briefly descripted in CUDA Memory Leak Problem While Runtime Souce Addition/Deletion

There will be lots of sources sending EOS to streammux, and the bus callback will handle the stream EOS messages generated by streammux.

/**
 * callback function to receive messages from components
 * in the pipeline.
 */
static gboolean
bus_callback (GstBus * bus, GstMessage * message, gpointer data)
{
  AppCtx *appCtx = (AppCtx *) data;

  switch (GST_MESSAGE_TYPE (message)) {
    ... ... ...
    case GST_MESSAGE_ELEMENT: { // process stream eos messages.
      // the stream EOS message is posted by streammux
      if (gst_nvmessage_is_stream_eos (message)) {
        if (install_sink_stream_eos_probe) {
          break;
        }
        guint stream_id;
        if (!gst_nvmessage_parse_stream_eos (message, &stream_id)) {
          NVGSTDS_ERR_MSG_V ("Fail to parse stream eos message.\n");
        }

        g_mutex_lock (&appCtx->source_lock);
        NVGSTDS_INFO_MSG_V ("Received Stream EOS from source id: %4d. " \
            "source uri: '%s'", stream_id, appCtx->config.multi_source_config[stream_id].uri);
        // g_print ("stream EOS comes from %s.\n", GST_OBJECT_NAME (message->src));
        // g_print ("stream EOS message type = %d.\n", GST_MESSAGE_TYPE (message));
        // g_print ("stream EOS message type name = %s.\n", GST_MESSAGE_TYPE_NAME (message));
        AppCtx *appCtx = (AppCtx *) data;
        // check if the stream is already deleted
        if (appCtx->config.multi_source_config[stream_id].enable) {        
          if (!delete_source_by_id (appCtx, stream_id)) {
            NVGSTDS_ERR_MSG_V ("Fail to delete source with id %d.", stream_id);
          }
        } else {
          NVGSTDS_ERR_MSG_V ("Get empty source with id: %d.", stream_id);
        }
        g_mutex_unlock (&appCtx->source_lock);
      }
      break;
    }
    default:
      break;
  }
  return TRUE;
}

I use the GST-TRACER to trace the gst objects’ ref count, the result shows that lots of stream-eos messages are alive with refcount=1 after program exit. According to gstreamer document, when callback function is called, the ownership of message will be taken by caller, so bus will unref it properly. I believe there’s somewhere in the closed source streammux part that streammux refed but havn’t unref the streamm-eos messages.

... ... ...
3:53:03.009774066 e[336m10097e[00m 0x56525f38d580 e[37mTRACE  e[00m e[00;34m          GST_TRACER :0::e[00m object-alive, type-name=(string)GstMessage, address=(gpointer)0x5652a46b3240, description=(string)element message: 0x5652a46b3240, time 99:99:99.999999999, seq-num 1510061, element 'src_bin_muxer', stream-eos, stream-id=(uint)0;, ref-count=(uint)1, trace=(string);
3:53:03.009782820 e[336m10097e[00m 0x56525f38d580 e[37mTRACE  e[00m e[00;34m          GST_TRACER :0::e[00m object-alive, type-name=(string)GstMessage, address=(gpointer)0x5652a35b3380, description=(string)element message: 0x5652a35b3380, time 99:99:99.999999999, seq-num 3153888, element 'src_bin_muxer', stream-eos, stream-id=(uint)1;, ref-count=(uint)1, trace=(string);
3:53:03.009791309 e[336m10097e[00m 0x56525f38d580 e[37mTRACE  e[00m e[00;34m          GST_TRACER :0::e[00m object-alive, type-name=(string)GstMessage, address=(gpointer)0x5652a2c73420, description=(string)element message: 0x5652a2c73420, time 99:99:99.999999999, seq-num 3077100, element 'src_bin_muxer', stream-eos, stream-id=(uint)0;, ref-count=(uint)1, trace=(string);
... ... ...

Hi @horacehxw,
Thank you so much for reporting this!
This is fixed in our upcoming release very soon.