• Hardware Platform (Jetson / GPU): GPU
• DeepStream Version: Confirmed in 8.0, possibly in 9.1 as well.
• TensorRT Version: 10.9.0
• NVIDIA GPU Driver Version (valid for GPU only): 595.71.05
• Issue Type( questions, new requirements, bugs): Bug
• How to reproduce the issue ?
The bug is not-deterministic and intermittent across separate machines. That is, it becomes consistent on machines that produce it. The bug relates to the workarounds found in the FAQ of DS 9.1 specifically:
- the
pthread_setspecificcrash, blamed on GLib ≤2.72 (The bug is still occurring on the native glib 2.8.0 of Ubuntu 24.04 inside a container) - the libgomp/libGLdispatch
LD_PRELOADworkaround for plugins that fail to load.
The bug occurs when gst-inspect-1.0is first ran without the gstreamer plugin registry created. causing it to abort which results in some deepstream elements not fully discovered. How to reproduce? you need a machine/container that produces it to begin with.
I performed the following to obtain the trace.
echo 'set pagination off
set breakpoint pending on
python
import gdb
owner = {}
def caller():
f = gdb.newest_frame().older()
while f is not None and (gdb.solib_name(f.pc()) or "").endswith("libc.so.6"):
f = f.older()
if f is None:
return "?"
return "%s (%s)" % (f.name() or "??", gdb.solib_name(f.pc()) or "main-exe")
class CreateDone(gdb.FinishBreakpoint):
def init(self, kp, who):
super().init(gdb.newest_frame(), internal=True)
self.kp, self.who = kp, who
def stop(self):
k = int(gdb.parse_and_eval("(unsigned int)%d" % self.kp))
owner[k] = self.who
print("CREATE key=%d by %s" % (k, self.who))
return False
def out_of_scope(self):
pass
class Create(gdb.Breakpoint):
def stop(self):
CreateDone(int(gdb.parse_and_eval("$rdi")), caller())
return False
class Delete(gdb.Breakpoint):
def stop(self):
k = int(gdb.parse_and_eval("$rdi"))
who = caller()
own = owner.pop(k, "")
lib = lambda s: s.split("(")[-1]
tag = "" if lib(own) == lib(who) else " <<<<< FOREIGN DELETE (owner: %s)" % own
print("DELETE key=%d by %s%s" % (k, who, tag))
return False
Create("pthread_key_create")
Delete("pthread_key_delete")
end
run
bt 20' > /tmp/keys.gdb
Then
rm -f /tmp/reg-test.bin GST_REGISTRY_FORK=no GST_REGISTRY=/tmp/reg-test.bin \ gdb -q -batch -x /tmp/keys.gdb --args gst-inspect-1.0 nvvideoconvert 2>&1 | tee /tmp/keys.log grep -E 'CREATE|DELETE' /tmp/keys.log | tail -40
libnvbufsurf_transform_deinit() deletes pthread key 0, which it never created, crashing gst-plugin-scanner
When libnvbufsurftransform.so is unloaded during plugin scanning (dlclose → _dl_call_fini → libnvbufsurf_transform_deinit), it calls pthread_key_delete(0). Key 0 was created by GLib (g_private_replace), not by this library. GModule’s next g_private_replace() fails with EINVAL and aborts with GLib (gthread-posix.c): Unexpected error from C library during 'pthread_setspecific': Invalid argument. Aborting. Plugins after that point are missing from the registry (e.g. No such element or plugin 'nvvideoconvert').
Key trace (gdb, GST_REGISTRY_FORK=no, fresh registry):
CREATE key=0 by g_private_replace (libglib-2.0.so.0)
CREATE key=1 by g_private_get (libglib-2.0.so.0)
CREATE key=2 by g_private_get (libglib-2.0.so.0)
CREATE key=3 by ?? (libcuda.so.1)
DELETE key=0 by libnvbufsurf_transform_deinit (libnvbufsurftransform.so) <-- never created by this library
DELETE key=3 by ?? (libcuda.so.1)
GLib (gthread-posix.c): ... 'pthread_setspecific': Invalid argument. Aborting.
#6 g_private_replace () from libglib-2.0.so.0
#7 g_module_supported () from libgmodule-2.0.so.0
... gst_update_registry ()
The likely cause is a static pthread_key_t that stays 0 when the key is never created, deleted unconditionally in the destructor. Expected: only delete the key if it was created. Workaround: LD_PRELOAD libnvbufsurftransform.so so it is never unloaded.