GStreamer warning: Embedded video playback halted; module nvh264enc0 reported: Could not initialize supporting library

I’m working on a server where a python script analyzes an incoming video with AI, draws some boxes on it and returns the resulting video with gstreamer and opencv’s cv2.Videowriter() back to a rtsp-stream. The underlying docker-image was created 4 months ago and everything works like expected.

However, when I now rebuild the exact same docker image and run the same script again I get the following error message from Gstreamer:

[ WARN:0] global /var/local/git/opencv/modules/videoio/src/cap_gstreamer.cpp (1761) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module nvh264enc0 reported: Could not initialize supporting library.
[ WARN:0] global /var/local/git/opencv/modules/videoio/src/cap_gstreamer.cpp (1592) open OpenCV | GStreamer warning: GStreamer: cannot put pipeline to play

The pipeline looks like this:

"appsrc ! videoconvert ! nvh264enc ! h264parse ! rtph264pay name=pay0 config-interval=1 pt=96 ! udpsink host="+host+" port="+port+" sync=false "

The underlying relevant part of the dockerfile is basically installing CUDA, cudnn, Gstreamer and OPENCV, and as I said, the image from 4 months ago is still working, but not when I want to rebuild it:

FROM ubuntu:18.04
ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update -y
&& apt-get install -y
apt-utils
pkg-config
zlib1g-dev
libwebp-dev
libtbb2
libtbb-dev
libgtk2.0-dev
pkg-config
libavcodec-dev
libavformat-dev
libswscale-dev
libv4l-dev
cmake
&& apt-get install -y
autoconf
autotools-dev
build-essential
gcc
jq
wget
git
gtk-doc-tools
keyboard-configuration
software-properties-common
python3
python3-pip
python3-dev
&& pip3 install --upgrade pip
&& pip3 install numpy

RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
&& mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
&& apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
&& add-apt-repository “deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /”
&& apt-get update
&& wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb
&& apt install ./nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb
&& apt-get update
&& apt-get install -y cuda-11-4
&& apt-get install -y --allow-downgrades libcudnn8=8.2.2.26-1+cuda11.4
&& apt-get install -y libcudnn8-dev=8.2.2.26-1+cuda11.4

RUN apt-get update -y
&& apt-get install -y
libgstreamer1.0-0
gstreamer1.0-plugins-base
gstreamer1.0-plugins-good
gstreamer1.0-plugins-bad
gstreamer1.0-plugins-ugly
gstreamer1.0-libav
gstreamer1.0-doc
gstreamer1.0-tools
libgstreamer1.0-dev
libgstreamer-plugins-base1.0-dev
&& git clone git://github.com/GStreamer/gst-plugins-bad /var/local/git/gst-plugins-bad
&& cd /var/local/git/gst-plugins-bad
&& git checkout 1.14.0
COPY Video_Codec_SDK_9.1.23/include /var/local/git/Video_Codec_SDK

RUN cd /var/local/git/Video_Codec_SDK
&& cp /usr/local/cuda/include/cuda.h /var/local/git/gst-plugins-bad/sys/nvenc/cuda.h
&& cp nvEncodeAPI.h /var/local/git/gst-plugins-bad/sys/nvenc
&& cp cuviddec.h /var/local/git/gst-plugins-bad/sys/nvdec
&& cp nvcuvid.h /var/local/git/gst-plugins-bad/sys/nvdec
&& cd /var/local/git/gst-plugins-bad
&& NVENCODE_CFLAGS=“-Icnvenc” ./autogen.sh --with-cuda-prefix=“/usr/local/cuda” --disable-gtk-doc
&& cd /var/local/git/gst-plugins-bad/sys/nvenc
&& make
&& cd /var/local/git/gst-plugins-bad/sys/nvenc
&& cp .libs/libgstnvenc.so /usr/lib/x86_64-linux-gnu/gstreamer-1.0/
&& cd /var/local/git/gst-plugins-bad/sys/nvdec
&& make
&& cd /var/local/git/gst-plugins-bad/sys/nvdec
&& cp .libs/libgstnvdec.so /usr/lib/x86_64-linux-gnu/gstreamer-1.0/
ENV OPENCV_RELEASE_TAG=4.5.0

RUN git clone --depth 1 -b ${OPENCV_RELEASE_TAG} GitHub - opencv/opencv_contrib: Repository for OpenCV's extra modules /var/local/git/opencv_contrib
&& git clone --depth 1 -b ${OPENCV_RELEASE_TAG} GitHub - opencv/opencv: Open Source Computer Vision Library /var/local/git/opencv
&& cd /var/local/git/opencv
&& mkdir -p /var/local/git/opencv/build
&& cd /var/local/git/opencv/build $$
&& cmake -D CMAKE_INSTALL_PREFIX=/usr/local CMAKE_BUILD_TYPE=Release
-D INSTALL_PYTHON_EXAMPLES=ON
-D INSTALL_C_EXAMPLES=OFF
-D PYTHON_EXECUTABLE=$(which python3)
-D BUILD_opencv_python2=OFF
-D CMAKE_INSTALL_PREFIX=$(python3 -c “import sys; print(sys.prefix)”)
-D PYTHON3_EXECUTABLE=$(which python3)
-D PYTHON3_INCLUDE_DIR=$(python3 -c “from distutils.sysconfig import get_python_inc; print(get_python_inc())”)
-D PYTHON3_PACKAGES_PATH=$(python3 -c “from distutils.sysconfig import get_python_lib; print(get_python_lib())”)
-D WITH_GSTREAMER=ON
-D WITH_GSTREAMER_0_10=OFF
-D WITH_CUDA=ON
-D WITH_CUDNN=OFF
-D OPENCV_DNN_CUDA=OFF
-D BUILD_EXAMPLES=OFF
-D OPENCV_EXTRA_MODULES_PATH=/var/local/git/opencv_contrib/modules
-D WITH_TBB=ON
-D WITH_LIBV4L=OFF
-D WITH_FFMPEG=OFF
-D OPENCV_GENERATE_PKGCONFIG=ON …
&& cd /var/local/git/opencv/build
&& make -j8 install
&& cd /var/local/git/opencv/build
&& make install
&& ldconfig

The server is an ubuntu 18.04, has an intel CPU a nvidia Quadro P5000 GPU.
I am working on this problem for a few weeks now and just can’t find a solution for this. Every help is appreciated.

Pure speculation, but when you initially built the container, you likely got cuda 11.4.0 installed, when rebuilding now, you get 11.4.3 which raises the required driver version. Which nvidia driver version is the host running?

According to nvidia-smi it’s: 470.57.02
Cuda version according to nvcc --version: release 11.4, V11.4.100

The current cuda 11.4.3 depends on driver 470.82.x. Furthermore, if those commands were used to build the container, by installing full “cuda” you also installed the driver inside the container. With it comes the cuda driver, libcuda, which has to match the kernel driver. So
Host:
nvidia driver 470.57.02
libcuda 470.57.02
old container:
libcuda 470.57.02 ->matches, works
new container:
(don’t know which version it pulled)
libcuda >=470.82 ->incorrect version, doesn’t work.

1 Like

https://docs.docker.com/config/containers/resource_constraints/#gpu

I updated the host system nvidia driver to 495.29.05 and it worked! When I tried with 470.82 though, it didn’t work…
Nevermind I’m just happy that it works

Thank you very very much!!!