DeepStream Coredump on Source Removal with Gst-nvmultiurisrcbin

Levi_Pereira · January 13, 2025, 7:45pm

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
GPU
Ubuntu (WSL)
• DeepStream Version
7.1
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
Driver Version: 566.36

• Issue Type( questions, new requirements, bugs)
I’ve identified a critical issue in DeepStream that causes a coredump when removing a source using hls source in gst-nvmultiurisrcbin (uridecodebin).
The crash occurs during the cleanup phase, specifically in the CUDA memory deallocation process. Importantly, this issue is not consistently reproducible - it happens sporadically during source removal operations.

Stack Trace Analysis: The crash originates from the nvv4l2decoder component attempting to clean up resources. The sequence flows through:

gst_element_change_state()
gst_v4l2_object_stop()
gst_buffer_pool_set_active()
gst_v4l2_allocator_stop()
v4l2_ioctl()
CuvidV4L2_Ioctl()
cuvidv4l2_dec_release_output_buffers()
NvBufSurfaceDestroyImpl()
ReleaseCudaBufferBatch()
cudaFreeHost() [crash]

Steps to Reproduce:

Set up a DeepStream pipeline using nvmultiurisrcbin
Add multiples video source
Remove the source through the API
Observe coredump during cleanup

Expected Behavior: Clean removal of source without memory deallocation issues.

Current Behavior: Pipeline crashes during CUDA memory cleanup with a coredump.

(gdb) bt full
#0  0x00007fff83fc48b6 in  () at /usr/lib/wsl/drivers/nv_dispi.inf_amd64_9425e4c3b1ac1c47/libcuda.so.1.1
#1  0x00007fff8402c3b5 in  () at /usr/lib/wsl/drivers/nv_dispi.inf_amd64_9425e4c3b1ac1c47/libcuda.so.1.1
#2  0x00007fff846aae3a in  () at /usr/lib/wsl/drivers/nv_dispi.inf_amd64_9425e4c3b1ac1c47/libcuda.so.1.1
#3  0x00007fff84127b8c in  () at /usr/lib/wsl/drivers/nv_dispi.inf_amd64_9425e4c3b1ac1c47/libcuda.so.1.1
#4  0x00007fff83e65871 in  () at /usr/lib/wsl/drivers/nv_dispi.inf_amd64_9425e4c3b1ac1c47/libcuda.so.1.1
#5  0x00007ffff444865d in  () at /usr/local/cuda-12/lib64/libcudart.so.12
#6  0x00007ffff441730f in  () at /usr/local/cuda-12/lib64/libcudart.so.12
#7  0x00007ffff4457db8 in cudaFreeHost () at /usr/local/cuda-12/lib64/libcudart.so.12
#8  0x00007ffff4aef37a in ReleaseCudaBufferBatch () at /opt/nvidia/deepstream/deepstream/lib/libnvbufsurface.so
#9  0x00007ffff4aeda85 in NvBufSurfaceDestroyImpl () at /opt/nvidia/deepstream/deepstream/lib/libnvbufsurface.so
#10 0x00007fffcae11810 in cuvidv4l2_dec_release_output_buffers(v4l2_decoder_context_rec*) () at ///opt/nvidia/deepstream/deepstream-7.1/lib/libcuvidv4l2.so
#11 0x00007fffcae097de in vidioc_dec_ioctl(int, unsigned long, void*) () at ///opt/nvidia/deepstream/deepstream-7.1/lib/libcuvidv4l2.so
#12 0x00007fffcae039dd in CuvidV4L2_Ioctl () at ///opt/nvidia/deepstream/deepstream-7.1/lib/libcuvidv4l2.so
#13 0x00007fffcaebf07c in v4l2_ioctl () at /lib/x86_64-linux-gnu/libv4l2.so.0
#14 0x00007fffcafd74d8 in gst_v4l2_allocator_stop () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libgstnvvideo4linux2.so
#15 0x00007fffcafdd688 in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libgstnvvideo4linux2.so
#16 0x00007ffff62ab5e2 in  () at /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#17 0x00007ffff62b3c78 in gst_buffer_pool_set_active () at /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#18 0x00007fffcafea959 in gst_v4l2_object_stop () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libgstnvvideo4linux2.so
#19 0x00007fffcaff17dc in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libgstnvvideo4linux2.so
#20 0x00007fff9c66287a in  () at /lib/x86_64-linux-gnu/libgstvideo-1.0.so.0
#21 0x00007ffff62cf7c9 in gst_element_change_state () at /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#22 0x00007ffff62cfed5 in  () at /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#23 0x00007ffff62a888c in  () at /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#24 0x00007fff740c7d72 in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstplayback.so
#25 0x00007ffff62cf7c9 in gst_element_change_state () at /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#26 0x00007ffff62cfed5 in  () at /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#27 0x00007ffff62a888c in  () at /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#28 0x00007fff740df389 in  () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstplayback.so
#29 0x00007ffff62cf7c9 in gst_element_change_state () at /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#30 0x00007ffff62cfed5 in  () at /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#31 0x00007ffff62a888c in  () at /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#32 0x00007fff80027346 in gst_ds_nvurisrc_bin_change_state(_GstElement*, GstStateChange) () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_nvurisrcbin.so
#33 0x00007ffff62cf7c9 in gst_element_change_state () at /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#34 0x00007ffff62cf80f in gst_element_change_state () at /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#35 0x00007ffff62cfed5 in  () at /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
#36 0x00007ffff4adb967 in s_nvmultiurisrcbincreator_remove_source_impl(void*, unsigned int, int) () at /opt/nvidia/deepstream/deepstream/lib/libnvdsgst_customhelper.so
#37 0x00007ffff4adaf1e in gst_nvmultiurisrcbincreator_remove_source () at /opt/nvidia/deepstream/deepstream/lib/libnvdsgst_customhelper.so
#38 0x00007fffec04cc6d in s_stream_api_impl(NvDsServerStreamInfo*, void*) () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_nvmultiurisrcbin.so
#39 0x00007fffec04f880 in rest_api_server_start(_GstDsNvMultiUriBin*)::{lambda(NvDsServerStreamInfo*, void*)#1}::operator()(NvDsServerStreamInfo*, void*) const ()
    at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_nvmultiurisrcbin.so
#40 0x00007fffec052d0c in void std::__invoke_impl<void, rest_api_server_start(_GstDsNvMultiUriBin*)::{lambda(NvDsServerStreamInfo*, void*)#1}&, NvDsServerStreamInfo*, void*>(std::__invoke_other, rest_api_server_start(_GstDsNvMultiUriBin*)::{lambda(NvDsServerStreamInfo*, void*)#1}&, NvDsServerStreamInfo*&&, void*&&) () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_nvmultiurisrcbin.so
#41 0x00007fffec051acb in std::enable_if<is_invocable_r_v<void, rest_api_server_start(_GstDsNvMultiUriBin*)::{lambda(NvDsServerStreamInfo*, void*)#1}&, NvDsServerStreamInfo*, void*>, void>::type std::__invoke_r<void, rest_api_server_start(_GstDsNvMultiUriBin*)::{lambda(NvDsServerStreamInfo*, void*)#1}&, NvDsServerStreamInfo*, void*>(rest_api_server_start(_GstDsNvMultiUriBin*)::{lambda(NvDsServerStreamInfo*, void*)#1}&, NvDsServerStreamInfo*&&, void*&&) () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_nvmultiurisrcbin.so
#42 0x00007fffec050d4f in std::_Function_handler<void (NvDsServerStreamInfo*, void*), rest_api_server_start(_GstDsNvMultiUriBin*)::{lambda(NvDsServerStreamInfo*, void*)#1}>::_M_invoke(std::_Any_data const&, NvDsServerStreamInfo*&&, void*&&) () at /usr/lib/x86_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst_nvmultiurisrcbin.so
#43 0x00007fffbea9c645 in std::function<void (NvDsServerStreamInfo*, void*)>::operator()(NvDsServerStreamInfo*, void*) const () at /opt/nvidia/deepstream/deepstream/lib/libnvds_rest_server.so
#44 0x00007fffbea8a277 in handleRemoveStream(Json::Value const&, Json::Value const&, Json::Value&, mg_connection*, std::function<void (NvDsServerStreamInfo*, void*)>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) () at /opt/nvidia/deepstream/deepstream/lib/libnvds_rest_server.so
#45 0x00007fffbea8b453 in nvds_rest_server_start(NvDsServerConfig*, NvDsServerCallbacks*)::{lambda(Json::Value const&, Json::Value const&, Json::Value&, mg_connection*)#3}::operator()(Json::Value const&, Json::Value const&, Json::Value&, mg_connection*) const () at /opt/nvidia/deepstream/deepstream/lib/libnvds_rest_server.so
#46 0x00007fffbea95fb5 in NvDsServerStatusCode std::__invoke_impl<NvDsServerStatusCode, nvds_rest_server_start(NvDsServerConfig*, NvDsServerCallbacks*)::{lambda(Json::Value const&, Json::Value const&, Json::Value&--Type <RET> for more, q to quit, c to continue without paging--
, mg_connection*)#3}&, Json::Value const&, Json::Value const&, Json::Value&, mg_connection*>(std::__invoke_other, nvds_rest_server_start(NvDsServerConfig*, NvDsServerCallbacks*)::{lambda(Json::Value const&, Json::Value const&, Json::Value&, mg_connection*)#3}&, Json::Value const&, Json::Value const&, Json::Value&, mg_connection*&&) () at /opt/nvidia/deepstream/deepstream/lib/libnvds_rest_server.so
#47 0x00007fffbea93256 in std::enable_if<is_invocable_r_v<NvDsServerStatusCode, nvds_rest_server_start(NvDsServerConfig*, NvDsServerCallbacks*)::{lambda(Json::Value const&, Json::Value const&, Json::Value&, mg_connection*)#3}&, Json::Value const&, Json::Value const&, Json::Value&, mg_connection*>, NvDsServerStatusCode>::type std::__invoke_r<NvDsServerStatusCode, nvds_rest_server_start(NvDsServerConfig*, NvDsServerCallbacks*)::{lambda(Json::Value const&, Json::Value const&, Json::Value&, mg_connection*)#3}&, Json::Value const&, Json::Value const&, Json::Value&, mg_connection*>(nvds_rest_server_start(NvDsServerConfig*, NvDsServerCallbacks*)::{lambda(Json::Value const&, Json::Value const&, Json::Value&, mg_connection*)#3}&, Json::Value const&, Json::Value const&, Json::Value&, mg_connection*&&) ()
    at /opt/nvidia/deepstream/deepstream/lib/libnvds_rest_server.so
#48 0x00007fffbea91165 in std::_Function_handler<NvDsServerStatusCode (Json::Value const&, Json::Value const&, Json::Value&, mg_connection*), nvds_rest_server_start(NvDsServerConfig*, NvDsServerCallbacks*)::{lambda(Json::Value const&, Json::Value const&, Json::Value&, mg_connection*)#3}>::_M_invoke(std::_Any_data const&, Json::Value const&, Json::Value const&, Json::Value&, mg_connection*&&) ()
    at /opt/nvidia/deepstream/deepstream/lib/libnvds_rest_server.so
#49 0x00007fffbea9bc84 in std::function<NvDsServerStatusCode (Json::Value const&, Json::Value const&, Json::Value&, mg_connection*)>::operator()(Json::Value const&, Json::Value const&, Json::Value&, mg_connection*) const () at /opt/nvidia/deepstream/deepstream/lib/libnvds_rest_server.so
#50 0x00007fffbea99683 in RequestHandler::handle(CivetServer*, mg_connection*) () at /opt/nvidia/deepstream/deepstream/lib/libnvds_rest_server.so
#51 0x00007fffbea99e2b in RequestHandler::handlePost(CivetServer*, mg_connection*) () at /opt/nvidia/deepstream/deepstream/lib/libnvds_rest_server.so
#52 0x00007fff9f88a88b in CivetServer::requestHandler(mg_connection*, void*) () at /opt/nvidia/deepstream/deepstream-7.1/lib/libcivetweb.so.1
#53 0x00007fff9f87ef5a in handle_request () at /opt/nvidia/deepstream/deepstream-7.1/lib/libcivetweb.so.1
#54 0x00007fff9f881027 in handle_request_stat_log () at /opt/nvidia/deepstream/deepstream-7.1/lib/libcivetweb.so.1
#55 0x00007fff9f8816a1 in process_new_connection () at /opt/nvidia/deepstream/deepstream-7.1/lib/libcivetweb.so.1
#56 0x00007fff9f882267 in worker_thread () at /opt/nvidia/deepstream/deepstream-7.1/lib/libcivetweb.so.1
#57 0x00007ffff7ce2ac3 in  () at /lib/x86_64-linux-gnu/libc.so.6
#58 0x00007ffff7d74850 in  () at /lib/x86_64-linux-gnu/libc.so.6

fanzh · January 14, 2025, 2:01am

did you install DeepStream according to this doc? what is the GPU model? can deepstream-test1 run well?

Levi_Pereira · January 15, 2025, 9:07pm

@fanzh

docker run:

docker run \
        -it \
        --privileged \
        --net=host \
        --ipc=host \
        --gpus all \
        -e DISPLAY=$DISPLAY \
        -e CUDA_CACHE_DISABLE=0 \
        --device /dev/snd \
        -v /tmp/.X11-unix/:/tmp/.X11-unix \
        nvcr.io/nvidia/deepstream:7.1-triton-multiarch

Yes, I installed DeepStream following the documentation. I’m using an RTX 2060 GPU, which might be the source of the issue since the documentation specifically mentions validation with RTX-3080, RTX-3090, and RTX-A6000 using GameReady Driver version 546.65.

I’ve validated that the problem only occurs in Linux WSL environment. I ran multiple tests successfully on both native Linux and Jetson platforms without any issues.

For anyone experiencing coredump issues related to cudaFreeHost() when using nvvideo4linux2 for decoding, this could be related to GPU compatibility.

Thank you

fanzh · January 16, 2025, 7:25am

Is this still an DeepStream issue to support? Thanks!

Levi_Pereira · January 16, 2025, 6:19pm

I don’t use DeepStream in WSL for production environments - the impact was specifically in my development environment where I encountered this issue. I’m unable to test using the RTX configuration recommended in the documentation.

I think this should be flagged as a potential concern. I plan to test this later with a clean installation of Windows/WSL/DeepStream, as Windows systems often undergo significant changes that could potentially cause issues. Given that Windows environments can vary significantly due to updates and modifications over time, it would be valuable to verify this behavior in a fresh setup.

Thank you

fanzh · January 17, 2025, 6:06am

As the doc shown, WSL 2 GPU acceleration will be available on Pascal and later GPU architecture. RTX 2060 Turing architecture is not in the list.

system · January 31, 2025, 6:06am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
"reason not-negotiated (-4)" python deepstream-app1 is stopped DeepStream SDK	2	618	December 26, 2022
Deepstream 6.4 issue DeepStream SDK	31	1207	January 4, 2024
DeepStream samples fail in fresh docker-container on centos 7.9 host system: Device is in streaming mode DeepStream SDK	15	551	October 27, 2022
Deepstream pipeline waits for input indefinitely DeepStream SDK deepstream61	19	707	June 16, 2022
Depstream:7.0-triton-arm-sbsa can not use nvvidconv DeepStream SDK	29	147	September 3, 2024
DeepStream SDK 5.0 with Nvidia GTX 960(4GB RAM) - deepstream-app throwing Bus error DeepStream SDK	9	740	October 12, 2021
Deepstream setup issues DeepStream SDK	6	2780	February 23, 2022
Cuda failure: status=801 Error(-1) in buffer allocation DeepStream SDK	36	1611	September 18, 2023
Requirements of a host to run a deepstream in a container DeepStream SDK	19	1979	October 12, 2021
New installation Multiple Failues DeepStream SDK	18	1131	June 28, 2022

DeepStream Coredump on Source Removal with Gst-nvmultiurisrcbin

Related topics