Reading a .mp4 file through Gstreamer and processing it's frames using opencv

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) = GPU RTX 2080Ti
• DeepStream Version = Deepstream-6.0
• JetPack Version (valid for Jetson only)
• NVIDIA GPU Driver Version (valid for GPU only) = 495.44
• Issue Type( questions, new requirements, bugs) = Question
**• How to reproduce the issue ? **
Hi, I’m a new to Gstreamer. I trying to read a .mp4 video through Gstreamer and want to process the frames using opencv.

I have pulled the deepstream-6.0 image and installed the python deepstream bindings, built opencv-python from source for python 3.8 and installed both the modules inside a virtual env.

The steps I followed for installing opencv-python from source.
git clone --recursive GitHub - opencv/opencv-python: Automated CI toolchain to produce precompiled opencv-python, opencv-python-headless, opencv-contrib-python and opencv-contrib-python-headless packages.
cd opencv-python
export CMAKE_ARGS="-DWITH_GSTREAMER=ON"
pip install --upgrade pip wheel
pip wheel . --verbose
pip install opencv_python*.whl

After the installation I was able to run the below code successfully
import cv2
cap = cv2.VideoCapture(‘autovideosrc ! videoconvert ! appsink’)
while True:
ret,frame = cap.read()
cv2.imshow(’’,frame)
if cv2.waitKey(1) & 0xFF == ord(‘q’):
break
cv2.destroyAllWindows()
cap.release()

with help of above reference code, I tried to create two different pipeline examples in gstreamer for reading .mp4 video and process it’s frames using opencv but getting same error for both the pipeline. I have attached the python code for your reference.

opencvpipeline.py (784 Bytes)

when I run the code I’m getting the following error:
[ WARN:0@0.308] global /opt/nvidia/deepstream/deepstream-6.0/sources/opencv-python/opencv/modules/videoio/src/cap_gstreamer.cpp (2402) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module qtdemux0 reported: Internal data stream error.
[ WARN:0@0.329] global /opt/nvidia/deepstream/deepstream-6.0/sources/opencv-python/opencv/modules/videoio/src/cap_gstreamer.cpp (1356) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0@0.329] global /opt/nvidia/deepstream/deepstream-6.0/sources/opencv-python/opencv/modules/videoio/src/cap_gstreamer.cpp (862) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created

Incase if both the pipelines are wrong can you suggest a pipeline for this issue? to read a .mp4 using gstreamer and process it’s frames using opencv in x86 machine

• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

I have attached my cv2.getBuildInformation for your reference

General configuration for OpenCV 4.5.5 =====================================
Version control: 4.5.5

Platform:
Timestamp: 2022-04-08T08:54:26Z
Host: Linux 5.11.0-41-generic x86_64
CMake: 3.22.3
CMake generator: Unix Makefiles
CMake build tool: /usr/bin/make
Configuration: Release

CPU/HW features:
Baseline: SSE SSE2 SSE3
requested: SSE3
Dispatched code generation: SSE4_1 SSE4_2 FP16 AVX AVX2 AVX512_SKX
requested: SSE4_1 SSE4_2 AVX FP16 AVX2 AVX512_SKX
SSE4_1 (16 files): + SSSE3 SSE4_1
SSE4_2 (1 files): + SSSE3 SSE4_1 POPCNT SSE4_2
FP16 (0 files): + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 AVX
AVX (4 files): + SSSE3 SSE4_1 POPCNT SSE4_2 AVX
AVX2 (31 files): + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2
AVX512_SKX (5 files): + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2 AVX_512F AVX512_COMMON AVX512_SKX

C/C++:
Built as dynamic libs?: NO
C++ standard: 11
C++ Compiler: /usr/bin/c++ (ver 7.5.0)
C++ flags (Release): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -msse -msse2 -msse3 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG -DNDEBUG
C++ flags (Debug): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Wsuggest-override -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -msse -msse2 -msse3 -fvisibility=hidden -fvisibility-inlines-hidden -g -O0 -DDEBUG -D_DEBUG
C Compiler: /usr/bin/cc
C flags (Release): -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -msse -msse2 -msse3 -fvisibility=hidden -O3 -DNDEBUG -DNDEBUG
C flags (Debug): -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections -msse -msse2 -msse3 -fvisibility=hidden -g -O0 -DDEBUG -D_DEBUG
Linker flags (Release): -Wl,–exclude-libs,libippicv.a -Wl,–exclude-libs,libippiw.a -Wl,–gc-sections -Wl,–as-needed
Linker flags (Debug): -Wl,–exclude-libs,libippicv.a -Wl,–exclude-libs,libippiw.a -Wl,–gc-sections -Wl,–as-needed
ccache: NO
Precompiled headers: NO
Extra dependencies: /usr/lib/x86_64-linux-gnu/libjpeg.so /usr/lib/x86_64-linux-gnu/libtiff.so /usr/lib/x86_64-linux-gnu/libz.so dl m pthread rt
3rdparty dependencies: libprotobuf ade ittnotify libwebp libpng libopenjp2 IlmImf quirc ippiw ippicv

OpenCV modules:
To be built: calib3d core dnn features2d flann gapi highgui imgcodecs imgproc ml objdetect photo python3 stitching video videoio
Disabled: world
Disabled by dependency: -
Unavailable: java python2 ts
Applications: -
Documentation: NO
Non-free algorithms: NO

GUI: GTK3
GTK+: YES (ver 3.22.30)
GThread : YES (ver 2.56.4)
GtkGlExt: NO
VTK support: NO

Media I/O:
ZLib: /usr/lib/x86_64-linux-gnu/libz.so (ver 1.2.11)
JPEG: /usr/lib/x86_64-linux-gnu/libjpeg.so (ver 80)
WEBP: build (ver encoder: 0x020f)
PNG: build (ver 1.6.37)
TIFF: /usr/lib/x86_64-linux-gnu/libtiff.so (ver 42 / 4.0.9)
JPEG 2000: build (ver 2.4.0)
OpenEXR: build (ver 2.3.0)
HDR: YES
SUNRASTER: YES
PXM: YES
PFM: YES

Video I/O:
DC1394: YES (2.2.5)
FFMPEG: YES
avcodec: YES (57.107.100)
avformat: YES (57.83.100)
avutil: YES (55.78.100)
swscale: YES (4.8.100)
avresample: YES (3.7.0)
GStreamer: YES (1.14.5)
v4l/v4l2: YES (linux/videodev2.h)

Parallel framework: pthreads

Trace: YES (with Intel ITT)

Other third-party libraries:
Intel IPP: 2020.0.0 Gold [2020.0.0]
at: /opt/nvidia/deepstream/deepstream-6.0/sources/opencv-python/_skbuild/linux-x86_64-3.8/cmake-build/3rdparty/ippicv/ippicv_lnx/icv
Intel IPP IW: sources (2020.0.0)
at: /opt/nvidia/deepstream/deepstream-6.0/sources/opencv-python/_skbuild/linux-x86_64-3.8/cmake-build/3rdparty/ippicv/ippicv_lnx/iw
VA: NO
Lapack: NO
Eigen: NO
Custom HAL: NO
Protobuf: build (3.19.1)

OpenCL: YES (no extra features)
Include path: /opt/nvidia/deepstream/deepstream-6.0/sources/opencv-python/opencv/3rdparty/include/opencl/1.2
Link libraries: Dynamic load

Python 3:
Interpreter: /root/Envs/py38/bin/python (ver 3.8)
Libraries: /usr/lib/x86_64-linux-gnu/libpython3.8.so (ver 3.8.0)
numpy: /tmp/pip-build-env-n4h88m9i/overlay/lib/python3.8/site-packages/numpy/core/include (ver 1.17.3)
install path: python/cv2/python-3

Python (for build): /root/Envs/py38/bin/python

Java:
ant: NO
JNI: NO
Java wrappers: NO
Java tests: NO

Install to: /opt/nvidia/deepstream/deepstream-6.0/sources/opencv-python/_skbuild/linux-x86_64-3.8/cmake-install

The key of DeepStream is the HW accelerated modules for IVA.

We have set the HW video decoder(decompress) the higheset priority, so the decoder works in decodebin is the HW decoder(Gst-nvvideo4linux2 — DeepStream 6.0.1 Release documentation), the common gstreamer plugin videoconvert can not handle the HW memory from the HW decoder.

Please refer to our python samples. GitHub - NVIDIA-AI-IOT/deepstream_python_apps: DeepStream SDK Python bindings and sample applications

Thank you for giving me clarity and your swift reply

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.