Deepstream gstreamer plugin blacklist in custom docker container

Hi all,

I need some help with debugging Gstreamer blacklisting Deepstream plugins in my docker container. I’m not running on JetPack, but rather on an OS called Balena OS. I believe the issue is not related to the OS, but rather the installation.

The OS is built on L4T 32.6.1, and I’m using a barebone TX2NX base image provided by Balena. The top layers of the image are shown below. We are using cuda 10.2, tensorrt 8.0.1 and deepstream 6.0.0.

# base image common to both build and run images
FROM balenalib/jetson-tx2-ubuntu:bionic-20220622 as base

# Don't prompt with any configuration questions
ENV DEBIAN_FRONTEND=noninteractive


# Update apt sources for 32.6
RUN echo "deb r32.6 main" >  /etc/apt/sources.list.d/nvidia.list  && \
    echo "deb r32.6  main" >>  /etc/apt/sources.list.d/nvidia.list

# Download and install BSP binaries for L4T 32.6.1
RUN apt-get update && apt-get install -y wget tar lbzip2 python3 libegl1 binutils && \
    wget && \
    tar xf jetson_linux_r32.6.1_aarch64.tbz2 && \
    cd Linux_for_Tegra && \
    sed -i 's/config.tbz2\"/config.tbz2\" --exclude=etc\/hosts --exclude=etc\/hostname/g' && \
    sed -i 's/install --owner=root --group=root \"${QEMU_BIN}\" \"${L4T_ROOTFS_DIR}\/usr\/bin\/\"/#install --owner=root --group=root \"${QEMU_BIN}\" \"${L4T_ROOTFS_DIR}\/usr\/bin\/\"/g' nv_tegra/ && \
    sed -i 's/LC_ALL=C chroot . mount -t proc none \/proc/ /g' nv_tegra/ && \
    sed -i 's/umount ${L4T_ROOTFS_DIR}\/proc/ /g' nv_tegra/ && \
    sed -i 's/chroot . \//  /g' nv_tegra/ && \
    ./ -r / --target-overlay && \
    rm -rf /jetson_linux_r32.6.1_aarch64.tbz2 && \
    rm -rf /Linux_for_Tegra && \
    echo "/usr/lib/aarch64-linux-gnu/tegra" > /etc/ && ldconfig

# Install required NVIDIA packages
RUN apt-get update && apt-get install --yes \
    cuda-cudart-10-2=10.2.300-1 \
    cuda-nvprof-10-2=10.2.300-1 \
    cuda-driver-dev-10-2=10.2.300-1 \
    cuda-cudart-dev-10-2=10.2.300-1 \
    cuda-nvcc-10-2=10.2.300-1 \
    cuda-nvrtc-10-2=10.2.300-1 \
    libcublas10= \
    libcudnn8= \
    libcurand-10-2= \
    libcurand-dev-10-2= \
    libnvinfer8=8.0.1-1+cuda10.2 \
    libnvinfer-plugin8=8.0.1-1+cuda10.2 \
    libnvonnxparsers8=8.0.1-1+cuda10.2 \
    libnvparsers8=8.0.1-1+cuda10.2 \
    python3-libnvinfer=8.0.1-1+cuda10.2 \
    && rm -rf /var/lib/apt/lists/*

# Install gstreamer
RUN apt-get update && apt-get install --yes \
    libgstreamer1.0-0=1.14.5-0ubuntu1~18.04.2 \
    gstreamer1.0-tools=1.14.5-0ubuntu1~18.04.2 \
    gstreamer1.0-plugins-good=1.14.5-0ubuntu1~18.04.3 \
    gstreamer1.0-plugins-bad=1.14.5-0ubuntu1~18.04.1 \
    gstreamer1.0-plugins-ugly=1.14.5-0ubuntu1~18.04.1 \
    gstreamer1.0-libav=1.14.5-0ubuntu1~18.04.1 \
    libgstreamer-plugins-base1.0-dev=1.14.5-0ubuntu1~18.04.3 \
    libgstrtspserver-1.0-0=1.14.5-0ubuntu1~18.04.1 \
    libgstrtspserver-1.0-dev=1.14.5-0ubuntu1~18.04.1 \
    && rm -rf /var/lib/apt/lists/*

# CUDA Varibles
ENV CUDA_ROOT="/usr/local/cuda-10.2"
ENV CPATH="$CUDA_ROOT/targets/aarch64-linux/include:${CPATH}"
ENV LD_LIBRARY_PATH="$CUDA_ROOT/targets/aarch64-linux/lib:${LD_LIBRARY_PATH}"

# Install important apt packages
RUN apt-get update && apt-get install -y --allow-unauthenticated --no-install-recommends \
    libusb-1.0-0-dev \
    openssh-client \
    python3-gi=3.26.1-2ubuntu1 \
    python3-numpy=1:1.13.3-2ubuntu1 \
    python3-pip \
    python3-venv=3.6.7-1~18.04 \
    python3-dev=3.6.7-1~18.04 \
    nfs-common \
    kmod \
    libssl-dev \
    net-tools \
    usbutils \
    python3-dbus \
    iputils-ping \
    cryptsetup \
    gdisk \
    libjson-glib-dev=1.4.2-3ubuntu0.18.04.1 \
    libssl1.1=1.1.1-1ubuntu2.1~18.04.23 \
    libjansson4=2.11-1 \
    libyaml-cpp-dev=0.5.2-4ubuntu1 \
    && rm -rf /var/lib/apt/lists/*

# Install ffmpeg version 4.3.2 and its relevant required libs
RUN apt-get update && \
    apt-get install -y software-properties-common && \
    add-apt-repository ppa:jonathonf/ffmpeg-4 && \
    apt-get install -y ffmpeg && \
    apt-get install -y --no-install-recommends \
    libavformat-dev \
    libavcodec-dev \
    libavdevice-dev \
    libavutil-dev \
    libavfilter-dev \
    libswscale-dev \
    libswresample-dev && \
    rm -rf /var/lib/apt/lists/* && \
    apt-get remove -y software-properties-common

# Install deepstream package (also remove installation of test app)
RUN wget && \
    tar -xvf deepstream_sdk_v6.0.0_jetson.tbz2 -C / && \
    sed -i '47,68d' /opt/nvidia/deepstream/deepstream-6.0/ && \
    /opt/nvidia/deepstream/deepstream-6.0/ && \
    ldconfig && \
    rm -rf /deepstream_sdk_v6.0.0_jetson.tbz2 && \
    rm -rf /opt/nvidia/deepstream/deepstream-6.0/bin && \
    rm -rf /opt/nvidia/deepstream/deepstream-6.0/doc && \
    rm -rf /opt/nvidia/deepstream/deepstream-6.0/samples

... rest of our application

The installation seems straightforward, and is very similar to the installation on Jetpack 4.6, apart from the BSP download and applying debs manually.

For some reason, gstreamer is blacklisting some of the deepstream plugins. I’ve manually check if the dependencies for the shared library exist by using ldd, and they all shows up fine. I also checked the output of ldconfig -p to ensure the gstreamer plugins exist, and they do. I’ve attach the output when doing GST_DEBUG=4 gst-inspect-1.0 -b after removing the cache file in /root/.cache/gstreamer-1.0.
inspect-output.txt (151.5 KB)

There’s a few stderr out regarding nvbuf_utils: Could not get EGL display connection, but the DISPLAY env var is already unset, and this device does not have any display output as well.

Any help pointing towards where to look into fixing this issue would be appreciated. Thanks!

Can you install “libssl1.0.0” instead of “libssl1.1”?

Can you find “/usr/lib/aarch64-linux-gnu/tegra/” in your docker?

What is the result of “ldd /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/”?

Can you install “libssl1.0.0” instead of “libssl1.1”?
I tried but still the same issue.

Can you find “/usr/lib/aarch64-linux-gnu/tegra/” in your docker?
Yep, all info shown below.

root@6caf369:/# find /usr/ -name "**"                                                                                                                                                      
root@6caf369:/# ldd /usr/lib/aarch64-linux-gnu/tegra/                                                                                                                                      (0x0000007f8ab2b000)                                                                                                                                                                       => /usr/lib/aarch64-linux-gnu/tegra/ (0x0000007f89388000)                                                                                                           => /usr/lib/aarch64-linux-gnu/tegra/ (0x0000007f89345000)                                                                                                                             => /usr/lib/aarch64-linux-gnu/ (0x0000007f89324000)                                                                                                                                 => /usr/lib/aarch64-linux-gnu/tegra/ (0x0000007f89306000)                                                                                                                   => /usr/lib/aarch64-linux-gnu/tegra/ (0x0000007f892e1000)                                                                                                               => /usr/lib/aarch64-linux-gnu/tegra/ (0x0000007f89263000)                                                                                                 => /usr/lib/aarch64-linux-gnu/tegra/ (0x0000007f88320000)                                                                                                                         => /usr/lib/aarch64-linux-gnu/ (0x0000007f8818c000)                                                                                                                           => /lib/aarch64-linux-gnu/ (0x0000007f88177000)                                                                                                                                       => /lib/aarch64-linux-gnu/ (0x0000007f8814b000)                                                                                                                             => /lib/aarch64-linux-gnu/ (0x0000007f88134000)                                                                                                                                       => /lib/aarch64-linux-gnu/ (0x0000007f87fdb000)                                                                                                                                         => /lib/aarch64-linux-gnu/ (0x0000007f87fb7000)                                                                                                                                 => /usr/lib/aarch64-linux-gnu/tegra/ (0x0000007f87f99000)                                                                                                                             => /usr/lib/aarch64-linux-gnu/ (0x0000007f87e6d000)                                                                                                                   
        /lib/ (0x0000007f8aaff000)                                                                                                                                                            => /usr/lib/aarch64-linux-gnu/tegra/ (0x0000007f87e5a000)                                                                                                   => /lib/aarch64-linux-gnu/ (0x0000007f87da1000)                                                                                                                                         => /usr/lib/aarch64-linux-gnu/tegra/ (0x0000007f87d5d000)                                                                                                                     => /usr/lib/aarch64-linux-gnu/tegra/ (0x0000007f87cee000)                                                                           
root@6caf369:/# ldconfig -p | grep libnvbufsurftransform                                                                                                                                                           (libc6,AArch64) => /usr/lib/aarch64-linux-gnu/tegra/                                                                                          (libc6,AArch64) => /opt/nvidia/deepstream/deepstream-6.0/lib/                                                                                             (libc6,AArch64) => /usr/lib/aarch64-linux-gnu/tegra/ 

What is the result of “ldd /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/”?
No “not-found” shared libs

root@6caf369:/# ldd /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/ (0x0000007fb602f000) => /usr/lib/aarch64-linux-gnu/ (0x0000007fb5eed000) => /usr/lib/aarch64-linux-gnu/ (0x0000007fb5dbd000) => /usr/lib/aarch64-linux-gnu/ (0x0000007fb5caf000) => /usr/lib/aarch64-linux-gnu/ (0x0000007fb5c52000) => /usr/lib/aarch64-linux-gnu/tegra/ (0x0000007fb5bd4000) => /usr/lib/aarch64-linux-gnu/tegra/ (0x0000007fb4491000) => /opt/nvidia/deepstream/deepstream-6.0/lib/ (0x0000007fb447f000) => /opt/nvidia/deepstream/deepstream-6.0/lib/ (0x0000007fb4468000) => /opt/nvidia/deepstream/deepstream-6.0/lib/ (0x0000007fb4453000) => /opt/nvidia/deepstream/deepstream-6.0/lib/ (0x0000007fb443c000) => /usr/lib/aarch64-linux-gnu/ (0x0000007fb42a8000) => /lib/aarch64-linux-gnu/ (0x0000007fb4293000) => /lib/aarch64-linux-gnu/ (0x0000007fb4267000) => /lib/aarch64-linux-gnu/ (0x0000007fb4250000) => /lib/aarch64-linux-gnu/ (0x0000007fb40f7000)
        /lib/ (0x0000007fb6003000) => /usr/lib/aarch64-linux-gnu/ (0x0000007fb40e3000) => /lib/aarch64-linux-gnu/ (0x0000007fb402a000) => /lib/aarch64-linux-gnu/ (0x0000007fb3fba000) => /usr/lib/aarch64-linux-gnu/ (0x0000007fb3fa2000) => /usr/lib/aarch64-linux-gnu/tegra/ (0x0000007fb3f5f000) => /usr/lib/aarch64-linux-gnu/ (0x0000007fb3f3e000) => /usr/lib/aarch64-linux-gnu/tegra/ (0x0000007fb3f20000) => /usr/lib/aarch64-linux-gnu/tegra/ (0x0000007fb3f0d000) => /usr/lib/aarch64-linux-gnu/tegra/ (0x0000007fb3eed000) => /usr/lib/aarch64-linux-gnu/tegra/ (0x0000007fb3ecf000) => /usr/lib/aarch64-linux-gnu/tegra/ (0x0000007fb3eaa000) => /usr/lib/aarch64-linux-gnu/tegra/ (0x0000007fb2f67000) => /lib/aarch64-linux-gnu/ (0x0000007fb2f43000) => /usr/lib/aarch64-linux-gnu/ (0x0000007fb2f15000) => /usr/lib/aarch64-linux-gnu/ (0x0000007fb2de9000) => /usr/lib/aarch64-linux-gnu/tegra/ (0x0000007fb2da5000) => /usr/lib/aarch64-linux-gnu/tegra/ (0x0000007fb2d36000)

Some updates,

I also try to use gdb to see whats’ causing the segfault as well, it seems like it’s missing something called pthread_mutex_lock.c?

root@6caf369:/# gdb /usr/bin/gst-inspect-1.0
GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
Find the GDB manual and other documentation resources online at:
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/gst-inspect-1.0...(no debugging symbols found)...done.
(gdb) set args /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/
(gdb) run
Starting program: /usr/bin/gst-inspect-1.0 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/".
nvbufsurftransform: Could not get EGL display connection

Program received signal SIGSEGV, Segmentation fault.
0x0000007fb7ac26e4 in __GI___pthread_mutex_lock (mutex=0x70404030103010e) at pthread_mutex_lock.c:67
67      pthread_mutex_lock.c: No such file or directory.

not sure if this helps too.

Seems the versions are not compatible.

Have you tried to install these packages directly in host and run?

No I have not, I’ll try and do that.

Any idea which package is running the wrong version?

Here’s the output of valgrind from running valgrind gst-inspect-1.0 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/

Segfaults and coredumps seems to be coming from libEGL_mesa. Still investigating more

root@6caf369:/# valgrind gst-inspect-1.0 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/                                                                                                 
==888== Memcheck, a memory error detector                                                                                                                                                                          
==888== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.                                                                                                                                            
==888== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info                                                                                                                                         
==888== Command: gst-inspect-1.0 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/deepstream/                                                                                                         
nvbufsurftransform: Could not get EGL display connection                                                                                                                                                           
nvbufsurftransform: Could not get EGL display connection                                                                                                                                                           
==888== Warning: set address range perms: large range [0x100000000, 0x1ef690000) (noaccess)                                                                                                                        
==888== Warning: set address range perms: large range [0xf00000000, 0xfef690000) (noaccess)                                                                                                                        
==888== Warning: set address range perms: large range [0x8ce9000, 0x28ce8000) (noaccess)                 
==888== Syscall param ioctl(generic) points to uninitialised byte(s)                                     
==888==    at 0x4BD860C: ioctl (ioctl.S:26)                                                              
==888==    by 0x8C5A2FB: ??? (in /usr/lib/aarch64-linux-gnu/tegra/                        
==888==    by 0x8C547DB: ??? (in /usr/lib/aarch64-linux-gnu/tegra/                        
==888==    by 0x7DF6D57: ??? (in /usr/lib/aarch64-linux-gnu/tegra/                        
==888==    by 0x7CBB143: ??? (in /usr/lib/aarch64-linux-gnu/tegra/                                      
==888==    by 0x7D46DAF: cuDeviceGetAttribute (in /usr/lib/aarch64-linux-gnu/tegra/                     
==888==    by 0x6267A1B: ??? (in /usr/lib/aarch64-linux-gnu/tegra/        
==888==  Address 0x1ffeffe2dc is on thread 1's stack                                                     
nvbufsurftransform: Could not get EGL display connection                                                 
==888== Warning: noted but unhandled ioctl 0x4e04 with no size/direction hints.                          
==888==    This could cause spurious value errors to appear.                                                           
==888==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.                                                
==888== Invalid read of size 8                                                                                                          
==888==    at 0x8D08310: ??? (in /usr/lib/aarch64-linux-gnu/                                                       
==888==  Address 0x5ef3888 is 8 bytes inside a block of size 48 free'd                                                                  
==888==    at 0x4846D58: free (in /usr/lib/valgrind/                                                                        
==888==  Block was alloc'd at                                                                                                                                 
==888==    at 0x4847B0C: calloc (in /usr/lib/valgrind/                                                                      
==888== Invalid write of size 8                                                                                                                               
==888==    at 0x8D07FFC: ??? (in /usr/lib/aarch64-linux-gnu/                                                                                                             
==888==  Address 0x5ef38a0 is 32 bytes inside a block of size 48 free'd                                                                                                                                                                                                                                                                                                                      
==888==    at 0x4846D58: free (in /usr/lib/valgrind/                                                                                                        
==888==  Block was alloc'd at                                                                  
==888==    at 0x4847B0C: calloc (in /usr/lib/valgrind/                                                                                                      
==888== Invalid read of size 8                                                                 
==888==    at 0x8D08040: ??? (in /usr/lib/aarch64-linux-gnu/                                                                                                             
==888==  Address 0x5ef3898 is 24 bytes inside a block of size 48 free'd                                                                                                                       
==888==    at 0x4846D58: free (in /usr/lib/valgrind/                                                                                                        
==888==  Block was alloc'd at                                                                  
==888==    at 0x4847B0C: calloc (in /usr/lib/valgrind/                                                                                                      
==888== Invalid write of size 8                                                                
==888==    at 0x8D08048: ??? (in /usr/lib/aarch64-linux-gnu/                                                                                                             
==888==  Address 0x5ef38a8 is 40 bytes inside a block of size 48 free'd                                                                                                                       
==888==    at 0x4846D58: free (in /usr/lib/valgrind/                                                                                                        
==888==  Block was alloc'd at                                                                  
==888==    at 0x4847B0C: calloc (in /usr/lib/valgrind/                                                                                                      
==888== Invalid free() / delete / delete[] / realloc()                                                                                                                                        
==888==    at 0x4846D58: free (in /usr/lib/valgrind/                                                                                                        
==888==  Address 0x5ef3880 is 0 bytes inside a block of size 48 free'd                                                                                                                        
==888==    at 0x4846D58: free (in /usr/lib/valgrind/                                                                                                        
==888==  Block was alloc'd at                                                                  
==888==    at 0x4847B0C: calloc (in /usr/lib/valgrind/                                                                                                      
==888== HEAP SUMMARY:                                                                          
==888==     in use at exit: 2,206,534 bytes in 23,835 blocks                                                                                                                                  
==888==   total heap usage: 89,539 allocs, 65,705 frees, 8,925,413 bytes allocated                                                                                                            
==888== LEAK SUMMARY:                                                                          
==888==    definitely lost: 18,474 bytes in 3 blocks                                                                                                                                          
==888==    indirectly lost: 0 bytes in 0 blocks                                                
==888==      possibly lost: 6,924 bytes in 81 blocks                                                                                                                                          
==888==    still reachable: 2,092,712 bytes in 23,459 blocks                                                                                                                                  
==888==                       of which reachable via heuristic:                                                                                                                               
==888==                         length64           : 80 bytes in 2 blocks                                                                                                                     
==888==                         newarray           : 1,552 bytes in 17 blocks                                                                                                                 
==888==         suppressed: 0 bytes in 0 blocks                                                
==888== Rerun with --leak-check=full to see details of leaked memory                                                                                                                          
==888== For counts of detected and suppressed errors, rerun with: -v                                                                                                                          
==888== Use --track-origins=yes to see where uninitialised values come from                                                                                                                   
==888== ERROR SUMMARY: 10 errors from 6 contexts (suppressed: 0 from 0)       

Either the driver is wrong or your “DISPLAY:0” is not Nvidia driver based.


I was able to solve the issue, the issue was not a missing library, but rather tegra-egl path is not in the ldconfig configs.

I updated the BSP installation part of the docker container to include echo "/usr/lib/aarch64-linux-gnu/tegra-egl" > /etc/ and that fixes the issue of gstreamer blacklisting the deepstream plugins.

However, whenever I do a gst-inspect-1.0 nvtracker to show its information, it would still crashes from a seg fault (similar to what I posted in my earlier post). This however doesn’t seem to cause any issue when running the sample apps.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.