Issues using the nvdemux from custom gstreamer element

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) Jetson TX2
• DeepStream Version 5.0
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs) bugs/ or questions
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Hello,

We did create a custom gstreamer element which is derrived from the nvarguscamerasrc element.
Some features are added and we created an own stream mux in it which is used by our application.
Currently we want to input our muxed stream with four video streams in it to de nvdemux.
But when we try this, we get a SIGSEGV crash in the nvdemux element.
We ran it both as pipeline command as from .c program (see attachment).

Things we already verified were:
-Checked that the batch and frame meta are filled in correctly by our custom element
-Locking the batch meta when modifying it

Would it be possible to get the source code of the nvdemux or debug symbols for it?
So that we can get the issue and fix it.

Best regards,
Floris van Drunen

Blockquote
“===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module3
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
Caught SIGSEGV
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
GST_ARGUS: Creating output stream
#0 0x0000007f9406de28 in __GI___poll (fds=0x5584b58440, nfds=547945427848, timeout=) at …/sysdeps/unix/sysv/linux/poll.c:41
#1 0x0000007f9417af58 in () at /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0
#2 0x0000005584a65200 in ()
Spinning. Please run ‘gdb gst-launch-1.0 9498’ to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.”
"

Blockquote
#include <gst/gst.h>
#include <glib.h>
#include <stdio.h>
#include “gstnvdsmeta.h”
/* The muxer output resolution must be set if the input streams will be of
different resolution. The muxer will scale all the input frames to this
resolution. /
#define MUXER_OUTPUT_WIDTH 1920
#define MUXER_OUTPUT_HEIGHT 1080
/
Muxer batch formation timeout, for e.g. 40 millisec. Should ideally be set
based on the fastest source’s framerate. */
#define MUXER_BATCH_TIMEOUT_USEC 40000
int
main (int argc, char *argv)
{
GMainLoop *loop = NULL;
GstElement *pipeline = NULL, *source = NULL, *nvstreamdemux = NULL, sink = NULL;
GstPad srcpad_0=NULL, sinkpad_0=NULL;
/
Standard GStreamer initialization /
gst_init (&argc, &argv);
loop = g_main_loop_new (NULL, FALSE);
/
Create gstreamer elements /
/
Create Pipeline element that will form a connection of other elements /
pipeline = gst_pipeline_new (“arguscamerasrc-pipeline”);
/
Source element for reading from the file /
source = gst_element_factory_make (“arguscamerasrc”, “camera-source”);
g_object_set (G_OBJECT (source), “num-sensors”, 1, NULL);
/
Demux /
nvstreamdemux = gst_element_factory_make (“nvstreamdemux”, “demux”);
sink = gst_element_factory_make (“fakesink”, “fakesink”);
if (!source || !nvstreamdemux || !sink) {
g_printerr (“One element could not be created. Exiting.\n”);
return -1;
}
/
Set up the pipeline /
/
we add all elements into the pipeline /
gst_bin_add_many (GST_BIN (pipeline), source, nvstreamdemux, sink, NULL);
if (!gst_element_link_many (source, nvstreamdemux, NULL)) {
g_printerr (“Elements could not be linked: 1. Exiting.\n”);
return -1;
}
/
Link the Demuxer and sink via request pads
/
srcpad_0 = gst_element_get_request_pad(nvstreamdemux, “src_0”);
sinkpad_0 = gst_element_get_static_pad(sink, “sink”);
if (gst_pad_link(srcpad_0, sinkpad_0) != GST_PAD_LINK_OK) {
g_printerr(“Failed to link: srcpad_0 to sinkpad_0!\n”);
return -1;
}
gst_object_unref (srcpad_0);
gst_object_unref (sinkpad_0);
/
Set the pipeline to “playing” state /
g_print (“Now playing: %s\n”, argv[1]);
gst_element_set_state (pipeline, GST_STATE_PLAYING);
/
Wait till pipeline encounters an error or EOS /
g_print (“Running…\n”);
g_main_loop_run (loop);
/
Out of the main loop, clean up nicely */
g_print (“Returned, stopping playback\n”);
gst_element_set_state (pipeline, GST_STATE_NULL);
g_print (“Deleting pipeline\n”);
gst_object_unref (GST_OBJECT (pipeline));
g_main_loop_unref (loop);
return 0;
}

1 Like

We have made a bit more progress on this crash by testing with cuda-gdb. That gives us the following stack trace:

GST_DEBUG=8 /usr/local/cuda-10.2/bin/cuda-gdb --args gst-launch-1.0 arguscamerasrc ! nvstreamdemux name=demux demux.src_1 ! fakesink
0x0000007fb359b5a4 in gst_plugin_nvdsgst_multistream_get_desc () from /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_multistream.so
(cuda-gdb) bt
#0  0x0000007fb359b5a4 in gst_plugin_nvdsgst_multistream_get_desc () from /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_multistream.so
#1  0x0000007fb7f08b44 in gst_flow_get_name () from /usr/lib/aarch64-linux-gnu/libgstreamer-1.0.so.0

With GST_DEBUG=8, I found the following differences:

Working version with nvarguscamerasrc:

0:00:02.990288372 10470   0x55898abed0 DEBUG               GST_PADS gstpad.c:3597:probe_hook_marshal:<demux:sink> probe returned OK
0:00:02.990308788 10470   0x55898abed0 TRACE        GST_REFCOUNTING gstobject.c:237:gst_object_ref:<demux> 0x55898ae080 ref 4->5
0:00:02.990339830 10470   0x55898abed0 DEBUG         GST_SCHEDULING gstpad.c:4320:gst_pad_chain_data_unchecked:<demux:sink> calling chainfunction &gst_nvstreamdemux_sink_chain_cuda_batch with buffer buffer: 0x7f6c00f770, pts 0:00:01.899999981, dts 99:99:99.999999999, dur 99:99:99.999999999, size 64, offset none, offset_end none, flags 0x0
0:00:02.990404121 10470   0x55898abed0 DEBUG               GST_CAPS gstpad.c:2705:gst_pad_has_current_caps:<demux:src_0> check current pad caps video/x-raw(memory:NVMM), format=(string)NV12, framerate=(fraction)30/1, width=(int)1920, height=(int)1080, batch-size=(int)1, num-surfaces-per-frame=(int)1
0:00:02.990427866 10470   0x55898abed0 DEBUG             GST_BUFFER gstbuffer.c:1148:gst_buffer_get_memory_range: idx 0, length 1
0:00:02.990447579 10470   0x55898abed0 LOG               GST_BUFFER gstbuffer.c:213:_get_merged_memory: buffer 0x7f6c00f770, idx 0, length 1
0:00:02.990474300 10470   0x55898abed0 TRACE        GST_REFCOUNTING gstminiobject.c:355:gst_mini_object_ref: 0x7f6c00d040 ref 30->31
0:00:02.990507709 10470   0x55898abed0 LOG               GST_BUFFER gstbuffer.c:1721:gst_buffer_map_range: buffer 0x7f6c00f770, idx 0, length -1, flags 0001
0:00:02.990536991 10470   0x55898abed0 LOG               GST_BUFFER gstbuffer.c:213:_get_merged_memory: buffer 0x7f6c00f770, idx 0, length 1
0:00:02.990560896 10470   0x55898abed0 TRACE        GST_REFCOUNTING gstminiobject.c:355:gst_mini_object_ref: 0x7f6c00d040 ref 31->32
0:00:02.990576672 10470   0x55898abed0 TRACE            GST_LOCKING gstminiobject.c:179:gst_mini_object_lock: lock 0x7f6c00d040: state 00010000, access_mode 1
0:00:02.990619650 10470   0x55898abed0 TRACE            GST_LOCKING gstminiobject.c:239:gst_mini_object_unlock: unlock 0x7f6c00d040: state 00010101, access_mode 1
0:00:02.990637731 10470   0x55898abed0 TRACE        GST_REFCOUNTING gstminiobject.c:441:gst_mini_object_unref: 0x7f6c00d040 unref 32->31
0:00:02.990651556 10470   0x55898abed0 TRACE        GST_REFCOUNTING gstminiobject.c:355:gst_mini_object_ref: 0x7f6c00f330 ref 1->2

Custom version:

0:00:01.099164295 12884   0x55982b38f0 DEBUG               GST_PADS gstpad.c:3597:probe_hook_marshal:<demux:sink> probe returned OK
0:00:01.099177127 12884   0x55982b38f0 TRACE        GST_REFCOUNTING gstobject.c:237:gst_object_ref:<demux> 0x55982ae020 ref 4->5
0:00:01.099194120 12884   0x55982b38f0 DEBUG         GST_SCHEDULING gstpad.c:4320:gst_pad_chain_data_unchecked:<demux:sink> calling chainfunction &gst_nvstreamdemux_sink_chain_cuda_batch with buffer buffer: 0x7f886040c0, pts 0:00:00.000000000, dts 0:00:00.000000000, dur 99:99:99.999999999, size 64, offset none, offset_end none, flags 0x0
0:00:01.099267468 12884   0x55982b38f0 DEBUG               GST_CAPS gstpad.c:2705:gst_pad_has_current_caps:<demux:src_0> check current pad caps video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, batch-size=(int)1, num-surfaces-per-frame=(int)1, framerate=(fraction)30/1
0:00:01.099282381 12884   0x55982b38f0 DEBUG             GST_BUFFER gstbuffer.c:1148:gst_buffer_get_memory_range: idx 0, length 1
0:00:01.099293229 12884   0x55982b38f0 LOG               GST_BUFFER gstbuffer.c:213:_get_merged_memory: buffer 0x7f886040c0, idx 0, length 1
0:00:01.099302702 12884   0x55982b38f0 TRACE        GST_REFCOUNTING gstminiobject.c:355:gst_mini_object_ref: 0x55982b6320 ref 1->2
0:00:01.099315503 12884   0x55982b38f0 LOG               GST_BUFFER gstbuffer.c:1721:gst_buffer_map_range: buffer 0x7f886040c0, idx 0, length -1, flags 0001
0:00:01.099325455 12884   0x55982b38f0 LOG               GST_BUFFER gstbuffer.c:213:_get_merged_memory: buffer 0x7f886040c0, idx 0, length 1
0:00:01.099334032 12884   0x55982b38f0 TRACE        GST_REFCOUNTING gstminiobject.c:355:gst_mini_object_ref: 0x55982b6320 ref 2->3
0:00:01.099344272 12884   0x55982b38f0 TRACE            GST_LOCKING gstminiobject.c:179:gst_mini_object_lock: lock 0x55982b6320: state 00010000, access_mode 1
Segmentation fault

Looks like right after locking the miniobject, the nvstreamdemux is doing something that makes it crash.

As mentioned before, access to the source code or some pointers on what we are doing wrong would be much appreciated!