During NvStreamMux and NvStreamDemux dynamic sink addition we get problems

Information

• Hardware Platform (Jetson / GPU) : Jetson TX2 - 4GB Ram
• DeepStream Version : DS-4.01 - 4.03
• JetPack Version (valid for Jetson only) : JetPack 4.1 - 4.3


We have below pipeline to which dynamically add/remove one or more input arm and correspondent one or more output arm

interpipeSrc-1  -> queue -> NvVideoConvert ->  \														        / AppSinkBin-1 
interpipeSrc-2  -> queue -> NvVideoConvert ->  /	-> NvStreamMux -> NvInfer -> NvTracker -> NvStreamDemux -> \ AppSinkBin-2
. . . . . . .    . . . .   . . . . . . . .     /                                                               \  . . . .
interpipeSrc-N  -> queue -> NvVideoConvert -> /														         \ AppSinkBin-N

AppSinkBin contains these simple components
AppSinkBin-1 = queue -> AppSink-1

It happens that
-) when removing one o more channels all work correctly, so for example when removing channel 2 it is removed correctly and the other channels keep on working correctly and producing correct tracking metadata.
-) when adding channels, the new channel work if no other channels are already active
but if adding a new channel having 1 or more other channel active then “after adding” new channel we get below crash
ATTENTION: it seems that crash happen not during new component creation so but when new channel frames are starting arriving.

	(gdb)
	(gdb) bt
	#0  0x0000007f5b14f718 in ?? () from /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurftransform.so.1.0.0
	#1  0x0000007f5b153ba4 in NvBufSurfTransform () from /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurftransform.so.1.0.0
	#2  0x0000007f783539d0 in ?? () from /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libnvdsgst_multistream.so
	#3  0x0000007f7e5c7b44 in ?? () from ./libgstreamer-1.0.so.0
	#4  0x000000000992faa0 in ?? ()
	Backtrace stopped: previous frame inner to this frame (corrupt stack?)
	(gdb) quit
	A debugging session is active.

ATTENTION: if for output component instead of AppSinkBin-X we use only one single TiledBin -> FileSink … it seems that addition works without crash.

So it could be that problem could be in NvStreamDemux after dynamic addition/linking of a component to pad of NvDeMuxer, because in crash there is no presence of AppSink components.
Or could it be there is some problem between NvStreamDemux and AppSinkBin ?
Than it is not clear why it involved also “nvbufsurftransform”

I saw that there are other guys having very similar problems like the one described here
How to add source and sink with nvstreammux and nvstreamdemux dynamically

and so I followed Nvidia support suggestion for nvDemuxer pad pre-allocation, because it seems Demuxer doesn’t support dynamic allocation, this helped a lot during channel deletion … but during channel addition we always get a crash.

Do you have some suggestions ? So what is the correct way to dynamically allocate pad in NvDemuxer and inject new frames ?

Here below there some parts of code showing how we add new dynamic input and output components.
There is not code for the static central part for NvMuxer + NvInfer + NvTracker + NvDemux … becuase it is very similar to what you use in your deepstream-app.

Alternatively another proposal is: if you can send me a debug version of library components (so library compiled with symbols ) of following 2 objects
libnvdsgst_multistream.so
libnvbufsurftransform.so.1.0.0
I can add them to our environment and we could have exact methods names producing crash

Now we are using both JetPack/deepsream 4.1 and 4.3, crash is always the same on both releases

thanks for support
Maurizio


// During creation of static NvStreamMux + NvInfer + NvTracker + NvStreamDemux there is the suggested preallocation of Demux pad
// ----------------------------------------------------------------------------------------------------------------------

	for ( gint i= 0; i< MAX_ANALYTICS_CH; ++i ) // i++ ) 
	{		
		GstPad *demux_src_pad;   
		gchar pad_name[16];
		g_snprintf(pad_name, 16, "src_%02d", i);	

		{
			demux_src_pad = gst_element_get_request_pad(_coreSet->demuxer, pad_name);
			if (demux_src_pad == NULL) 
				g_printerr ("\n ERROR: Failed to get the request pad [%s]\n", pad_name);
			else
				g_print ("\nOK: Succes to get the request pad [%s]", pad_name);
				
			gst_object_unref (demux_src_pad);
		}
	}	

// During creation of dynamic channel component there is linking of above created output DeMux pads with AppSinkBin
// ----------------------------------------------------------------------------------------------------------------------

GstElement *
createInOutComp_forChan ( CorePipeElements *_coreSet, gint pipeNum, gchar *interPipeName )
{
// ===============================================================================
// Input components Adding Interpipe src and nvvideoconvert before NvMux + NvInfer + NvTracker + NvDeMux
// ===============================================================================
g_snprintf (elem_name, sizeof (elem_name), “interPipeSrcIn-%d”, pipeNum);
_coreSet->interPipeSrcIn[pipeNum] = gst_element_factory_make(“interpipesrc”,elem_name);
g_assert(_coreSet->interPipeSrcIn[pipeNum]);
// --------------------------------------------------------------
g_snprintf (elem_name, sizeof (elem_name), “%s”, (const char *)_coreSet->chName[pipeNum]->str );
g_object_set ( _coreSet->interPipeSrcIn[pipeNum], “listen-to”, elem_name, NULL ) ;
// ---------------------------------------------------------------
g_object_set ( _coreSet->interPipeSrcIn[pipeNum], “format”, 3, NULL);
g_object_set ( _coreSet->interPipeSrcIn[pipeNum], “enable-sync”, 1, NULL);
g_object_set ( _coreSet->interPipeSrcIn[pipeNum], “do-timestamp”, 1, NULL);
g_object_set ( _coreSet->interPipeSrcIn[pipeNum], “is-live”, 1, NULL);
g_object_set ( _coreSet->interPipeSrcIn[pipeNum], “accept-events”, TRUE, NULL);
g_object_set ( _coreSet->interPipeSrcIn[pipeNum], “allow-renegotiation”, TRUE, NULL);
// ----------------------------------------------------------------------------
g_snprintf (elem_name, sizeof (elem_name), “interQueueCv-%d”, pipeNum);
_coreSet->interQueue[pipeNum] = gst_element_factory_make(“queue”,elem_name);
g_assert(_coreSet->interQueue[pipeNum]);
//// ----------------------------------------------------------------------------
g_snprintf (elem_name, sizeof (elem_name), “nvVideoConvert-My-%d”, pipeNum);
_coreSet->nvVideoConvertMux[pipeNum] = gst_element_factory_make(“nvvideoconvert”,elem_name);
g_assert(_coreSet->nvVideoConvertMux[pipeNum]);
// =================================================================================
gst_bin_add_many(GST_BIN(_coreSet->CoreInferencePipe), _coreSet->interPipeSrcIn[pipeNum],
_coreSet->interQueue[pipeNum], _coreSet->nvVideoConvertMux[pipeNum], NULL);

if (!gst_element_link_many(_coreSet->interPipeSrcIn[pipeNum], _coreSet->interQueue[pipeNum], _coreSet->nvVideoConvertMux[pipeNum], NULL))
{
	g_print("\n ERROR: cannot link  _coreSet->interPipeSrcIn[%d] ... interQueue[%d] to _coreSet->nvvidconv[%d] ....\n", pipeNum, pipeNum, pipeNum);
	_coreSet->CoreInferencePipe = NULL;
	goto done;
}

// this function link interPipeSrcIn to input mux pad number 0, 1, 2, 3, ... 
// --------------------------------------------------------------------------------
if (!link_element_to_streammux_sink_pad(_coreSet->nvstreammux, _coreSet->nvVideoConvertMux[pipeNum], pipeNum )) 
{
	g_print("\n ERROR: cannot link  _coreSet->nvVideoConvertMux[pipeNum] to nvstreammux  ....\n");
	gst_object_unref(_coreSet->CoreInferencePipe);
	_coreSet->CoreInferencePipe = NULL;
	goto done;
}
		
g_print("\n OK ... linked  _coreSet->nvVideoConvertMux[%d] to nvstreammux  ....", pipeNum);

// ==============================================================================
// Output components after NvMux + NvInfer + NvTracker + NvDeMux 
// ==============================================================================
{	
	gchar pad_name[16];
	GstPad *demux_src_pad;
	if (!create_app_sink_bin_CCM(&_coreSet->appSinkBins[pipeNum], pipeNum))
	{
		NVGSTDS_ERR_MSG_V("create_app_sink_bin %d failed\n", pipeNum);
		goto done;
	}
	gst_bin_add(GST_BIN(_coreSet->CoreInferencePipe), _coreSet->appSinkBins[pipeNum].bin);
// -----------------------------------------------------------------------------------------------
	GstAppSinkCallbacks callbacks;
	callbacks.new_preroll = new_preroll_comp; 
	callbacks.new_sample = new_sample_comp;
	callbacks.eos = eos_comp;
	gst_app_sink_set_callbacks(GST_APP_SINK(_coreSet->appSinkBins[pipeNum].appsink), &callbacks, GUINT_TO_POINTER(pipeNum), NULL);
	// -----------------------------------------------------------------------------------------------
	g_snprintf(pad_name, 16, "src_%02d", pipeNum);	
	g_print ("\n linking demuxer with  pad_name=[%s]  ", pad_name );				
	demux_src_pad = gst_element_get_static_pad (_coreSet->demuxer, pad_name);
			
	// linking demux_src_pad  with appSinkBin sink_pad
	---------------------------------------------------
	GstPad *sink_pad = gst_element_get_static_pad (_coreSet->appSinkBins[pipeNum].bin, "sink");
	// link the pads
	if (gst_pad_link (demux_src_pad, sink_pad) != GST_PAD_LINK_OK) {
	     g_printerr ("Failed to link sink_element to demux!\n");
	     // handle errors in any way you see fit (maybe exit)
	}
	else 
	   g_print ("\n linked demuxer with  appSink=[%s]  <<<<<<<<<<<<< ", pad_name );
			
	gst_object_unref(demux_src_pad);
}

When I ran into trouble with this type of thing, I got a lot of help from GST_DEBUG. You may already be using that, but I thought I’d mention it just in case.
I often ran at debug level 2, 3, or 4, but I saw so much irrelevant output from one of the Nvidia buffer elements (named v4l2bufferpool or similar) that I had to mute the debug output from it, so I ended up with something that looked like this:
$> GST_DEBUG=2,v4l2*:1 ./my-deepstream-app parameters

Here’s the documentation for GST_DEBUG, describing the levels and the syntax:
https://gstreamer.freedesktop.org/documentation/tutorials/basic/debugging-tools.html

Again, sorry if this is obvious stuff. And sorry I can’t help more. Hope this helps.

Hey Customer,
This pipeline should work, could you replace appsink element with fakesink and see if the issue persists. And if yes could you share me the code and makefile so we can repro it locally.
Also I think you need to move to DS 5.0 since we will not support fix on 4.0.2