Python GStreamer HW decoding from RTSP not working

Hi all,
I’m trying to capture frames from RTSP stream using GStreamer in order to access HW decoding but it seems not working.

I compiled Opencv 4.0 using this script: https://github.com/AastaNV/JEP/blob/master/script/install_opencv4.0.0_Nano.sh
required cause original Opencv installed in jetson nano distro returns an error using GST pipeline as input

Now using this simple code:

import cv2
import gi
gi.require_version('Gst', '1.0')

if __name__ == '__main__':
    gst = "rtspsrc location=rtsp://admin:Password@192.168.0.200:554/ latency=200 ! queue ! rtph264depay ! h264parse ! queue ! h264parse ! omxh264dec ! videoconvert ! appsink"

    cap = cv2.VideoCapture(gst)
    if not cap.isOpened():
        print("capture failed")
        exit()

    while True:

        ret, frame = cap.read()
        if ret:
            frame = cv2.resize(frame, (1280,720)) # I know it could increase CPU usage but also removing it doesn't change
            cv2.imshow('frame', frame)
            if (cv2.waitKey(1) & 0xFF == ord('q')):
                break;

    cap.release()
    cv2.destroyAllWindows()

If I check CPU utilization with top command I get: 160-190%

but if I launch:

gst-launch-1.0 rtspsrc location=rtsp://admin:Password@192.168.0.200:554 latency=200 ! queue ! rtph264depay ! h264parse ! queue ! h264parse ! omxh264dec ! nveglglessink -e

I have a CPU utilization of: 34%

So seems that using gst-launch-1.0 HW decoding works but not using Python and opencv.

Python building result:

>>> import cv2
>>> print cv2.getBuildInformation()

General configuration for OpenCV 4.0.0 =====================================
  Version control:               unknown

  Extra modules:
    Location (extra):            /home/viper/opencv-4.0.0/opencv_contrib-4.0.0/modules
    Version control (extra):     unknown

  Platform:
    Timestamp:                   2019-09-13T09:09:00Z
    Host:                        Linux 4.9.140-tegra aarch64
    CMake:                       3.10.2
    CMake generator:             Unix Makefiles
    CMake build tool:            /usr/bin/make
    Configuration:               RELEASE

  CPU/HW features:
    Baseline:                    NEON FP16
      required:                  NEON
      disabled:                  VFPV3

  C/C++:
    Built as dynamic libs?:      YES
    C++ Compiler:                /usr/bin/c++  (ver 7.4.0)
    C++ flags (Release):         -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG
    C++ flags (Debug):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -g  -O0 -DDEBUG -D_DEBUG
    C Compiler:                  /usr/bin/cc
    C flags (Release):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-narrowing -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -O3 -DNDEBUG  -DNDEBUG
    C flags (Debug):             -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-narrowing -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -g  -O0 -DDEBUG -D_DEBUG
    Linker flags (Release):
    Linker flags (Debug):
    ccache:                      NO
    Precompiled headers:         YES
    Extra dependencies:          m pthread cudart_static -lpthread dl rt nppc nppial nppicc nppicom nppidei nppif nppig nppim nppist nppisu nppitc npps cublas cufft -L/usr/local/cuda/lib64 -L/usr/lib/aarch64-linux-gnu
    3rdparty dependencies:

  OpenCV modules:
    To be built:                 aruco bgsegm bioinspired calib3d ccalib core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev datasets dnn dnn_objdetect dpm face features2d flann freetype fuzzy gapi hdf hfs highgui img_hash imgcodecs imgproc java_bindings_generator line_descriptor ml objdetect optflow phase_unwrapping photo plot python2 python3 python_bindings_generator reg rgbd saliency shape stereo stitching structured_light superres surface_matching text tracking video videoio videostab xfeatures2d ximgproc xobjdetect xphoto
    Disabled:                    world
    Disabled by dependency:      -
    Unavailable:                 cnn_3dobj cvv java js matlab ovis sfm ts viz
    Applications:                apps
    Documentation:               NO
    Non-free algorithms:         NO

  GUI:
    GTK+:                        YES (ver 2.24.32)
      GThread :                  YES (ver 2.56.4)
      GtkGlExt:                  NO
    VTK support:                 NO

  Media I/O:
    ZLib:                        /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.11)
    JPEG:                        /usr/lib/aarch64-linux-gnu/libjpeg.so (ver 80)
    WEBP:                        build (ver encoder: 0x020e)
    PNG:                         /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.6.34)
    TIFF:                        build (ver 42 - 4.0.9)
    JPEG 2000:                   build (ver 1.900.1)
    OpenEXR:                     build (ver 1.7.1)
    HDR:                         YES
    SUNRASTER:                   YES
    PXM:                         YES
    PFM:                         YES

  Video I/O:
    DC1394:                      NO
    FFMPEG:                      YES
      avcodec:                   YES (ver 57.107.100)
      avformat:                  YES (ver 57.83.100)
      avutil:                    YES (ver 55.78.100)
      swscale:                   YES (ver 4.8.100)
      avresample:                YES (ver 3.7.0)
    GStreamer:
      base:                      YES (ver 1.14.5)
      video:                     YES (ver 1.14.5)
      app:                       YES (ver 1.14.5)
      riff:                      YES (ver 1.14.5)
      pbutils:                   YES (ver 1.14.5)
    v4l/v4l2:                    linux/videodev2.h

  Parallel framework:            pthreads

  Trace:                         YES (built-in)

  Other third-party libraries:
    Lapack:                      NO
    Eigen:                       YES (ver 3.3.4)
    Custom HAL:                  YES (carotene (ver 0.0.1))
    Protobuf:                    build (3.5.1)

  NVIDIA CUDA:                   YES (ver 10.0, CUFFT CUBLAS)
    NVIDIA GPU arch:             53
    NVIDIA PTX archs:

  OpenCL:                        YES (no extra features)
    Include path:                /home/viper/opencv-4.0.0/opencv-4.0.0/3rdparty/include/opencl/1.2
    Link libraries:              Dynamic load

  Python 2:
    Interpreter:                 /usr/bin/python2.7 (ver 2.7.15)
    Libraries:                   /usr/lib/aarch64-linux-gnu/libpython2.7.so (ver 2.7.15+)
    numpy:                       /usr/lib/python2.7/dist-packages/numpy/core/include (ver 1.13.3)
    packages path:               lib/python2.7/dist-packages

  Python 3:
    Interpreter:                 /usr/bin/python3 (ver 3.6.8)
    Libraries:                   /usr/lib/aarch64-linux-gnu/libpython3.6m.so (ver 3.6.8)
    numpy:                       /usr/local/lib/python3.6/dist-packages/numpy/core/include (ver 1.17.1)
    packages path:               lib/python3.6/dist-packages

  Python (for build):            /usr/bin/python2.7

  Java:
    ant:                         NO
    JNI:                         NO
    Java wrappers:               NO
    Java tests:                  NO

  Install to:                    /usr/local
-----------------------------------------------------------------

Any idea? I don’t really know how to quit from this condition…

If I change line 8 to:

cap = cv2.VideoCapture("rtsp://admin:Password@192.168.0.200:554/")

Visualization seems less fluid but CPU utilization drops down to 60-70%

Hi,
The working pipeline should be

gst = "rtspsrc location=rtsp://admin:Password@192.168.0.200:554/ latency=200 ! queue ! rtph264depay ! queue ! h264parse ! omxh264dec ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink"

Please run ‘sudo tegrastats’ to check CPU usage. And ‘sudo jetson_clocks’ to run in max performance.

Thank you!
I’m trying to undestand GStreamer but is very complex compared to FFMPEG.

Hi,

I have been trying different ideas but can’t figure out how to get smooth motion display from gstreamer output. I tried the gst suggested by @DaneLLL as well. I am using Python with openCV 4.5 compiled with CUDA, tried both nvv4l2decoder and omxh264dec. None of them is giving me a smooth motion video. It seems the openCV call read() in the loop after cv2.VideoCapture() takes periodically longer time and that is why the display showing periodic jerking. I tried placing queue in several places and lastly as suggested by @DaneLLL. My platform is a Nano running JP4.4.

Any help would be greatly appreciated.
Thanks in advance.

Found the solution.
Taking the latency off, solved my problem.