Python GStreamer HW decoding from RTSP not working

Autonomous Machines Jetson & Embedded Systems Jetson Nano
1

Hi all,
I’m trying to capture frames from RTSP stream using GStreamer in order to access HW decoding but it seems not working.

I compiled Opencv 4.0 using this script: https://github.com/AastaNV/JEP/blob/master/script/install_opencv4.0.0_Nano.sh
required cause original Opencv installed in jetson nano distro returns an error using GST pipeline as input

Now using this simple code:

import cv2
import gi
gi.require_version('Gst', '1.0')

if __name__ == '__main__':
    gst = "rtspsrc location=rtsp://admin:Password@192.168.0.200:554/ latency=200 ! queue ! rtph264depay ! h264parse ! queue ! h264parse ! omxh264dec ! videoconvert ! appsink"

    cap = cv2.VideoCapture(gst)
    if not cap.isOpened():
        print("capture failed")
        exit()

    while True:

        ret, frame = cap.read()
        if ret:
            frame = cv2.resize(frame, (1280,720)) # I know it could increase CPU usage but also removing it doesn't change
            cv2.imshow('frame', frame)
            if (cv2.waitKey(1) & 0xFF == ord('q')):
                break;

    cap.release()
    cv2.destroyAllWindows()

If I check CPU utilization with top command I get: 160-190%

but if I launch:

gst-launch-1.0 rtspsrc location=rtsp://admin:Password@192.168.0.200:554 latency=200 ! queue ! rtph264depay ! h264parse ! queue ! h264parse ! omxh264dec ! nveglglessink -e

I have a CPU utilization of: 34%

So seems that using gst-launch-1.0 HW decoding works but not using Python and opencv.

Python building result:

>>> import cv2
>>> print cv2.getBuildInformation()

General configuration for OpenCV 4.0.0 =====================================
  Version control:               unknown

  Extra modules:
    Location (extra):            /home/viper/opencv-4.0.0/opencv_contrib-4.0.0/modules
    Version control (extra):     unknown

  Platform:
    Timestamp:                   2019-09-13T09:09:00Z
    Host:                        Linux 4.9.140-tegra aarch64
    CMake:                       3.10.2
    CMake generator:             Unix Makefiles
    CMake build tool:            /usr/bin/make
    Configuration:               RELEASE

  CPU/HW features:
    Baseline:                    NEON FP16
      required:                  NEON
      disabled:                  VFPV3

  C/C++:
    Built as dynamic libs?:      YES
    C++ Compiler:                /usr/bin/c++  (ver 7.4.0)
    C++ flags (Release):         -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG
    C++ flags (Debug):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -g  -O0 -DDEBUG -D_DEBUG
    C Compiler:                  /usr/bin/cc
    C flags (Release):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-narrowing -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -O3 -DNDEBUG  -DNDEBUG
    C flags (Debug):             -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-narrowing -Wno-comment -Wimplicit-fallthrough=3 -Wno-strict-overflow -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -g  -O0 -DDEBUG -D_DEBUG
    Linker flags (Release):
    Linker flags (Debug):
    ccache:                      NO
    Precompiled headers:         YES
    Extra dependencies:          m pthread cudart_static -lpthread dl rt nppc nppial nppicc nppicom nppidei nppif nppig nppim nppist nppisu nppitc npps cublas cufft -L/usr/local/cuda/lib64 -L/usr/lib/aarch64-linux-gnu
    3rdparty dependencies:

  OpenCV modules:
    To be built:                 aruco bgsegm bioinspired calib3d ccalib core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev datasets dnn dnn_objdetect dpm face features2d flann freetype fuzzy gapi hdf hfs highgui img_hash imgcodecs imgproc java_bindings_generator line_descriptor ml objdetect optflow phase_unwrapping photo plot python2 python3 python_bindings_generator reg rgbd saliency shape stereo stitching structured_light superres surface_matching text tracking video videoio videostab xfeatures2d ximgproc xobjdetect xphoto
    Disabled:                    world
    Disabled by dependency:      -
    Unavailable:                 cnn_3dobj cvv java js matlab ovis sfm ts viz
    Applications:                apps
    Documentation:               NO
    Non-free algorithms:         NO

  GUI:
    GTK+:                        YES (ver 2.24.32)
      GThread :                  YES (ver 2.56.4)
      GtkGlExt:                  NO
    VTK support:                 NO

  Media I/O:
    ZLib:                        /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.11)
    JPEG:                        /usr/lib/aarch64-linux-gnu/libjpeg.so (ver 80)
    WEBP:                        build (ver encoder: 0x020e)
    PNG:                         /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.6.34)
    TIFF:                        build (ver 42 - 4.0.9)
    JPEG 2000:                   build (ver 1.900.1)
    OpenEXR:                     build (ver 1.7.1)
    HDR:                         YES
    SUNRASTER:                   YES
    PXM:                         YES
    PFM:                         YES

  Video I/O:
    DC1394:                      NO
    FFMPEG:                      YES
      avcodec:                   YES (ver 57.107.100)
      avformat:                  YES (ver 57.83.100)
      avutil:                    YES (ver 55.78.100)
      swscale:                   YES (ver 4.8.100)
      avresample:                YES (ver 3.7.0)
    GStreamer:
      base:                      YES (ver 1.14.5)
      video:                     YES (ver 1.14.5)
      app:                       YES (ver 1.14.5)
      riff:                      YES (ver 1.14.5)
      pbutils:                   YES (ver 1.14.5)
    v4l/v4l2:                    linux/videodev2.h

  Parallel framework:            pthreads

  Trace:                         YES (built-in)

  Other third-party libraries:
    Lapack:                      NO
    Eigen:                       YES (ver 3.3.4)
    Custom HAL:                  YES (carotene (ver 0.0.1))
    Protobuf:                    build (3.5.1)

  NVIDIA CUDA:                   YES (ver 10.0, CUFFT CUBLAS)
    NVIDIA GPU arch:             53
    NVIDIA PTX archs:

  OpenCL:                        YES (no extra features)
    Include path:                /home/viper/opencv-4.0.0/opencv-4.0.0/3rdparty/include/opencl/1.2
    Link libraries:              Dynamic load

  Python 2:
    Interpreter:                 /usr/bin/python2.7 (ver 2.7.15)
    Libraries:                   /usr/lib/aarch64-linux-gnu/libpython2.7.so (ver 2.7.15+)
    numpy:                       /usr/lib/python2.7/dist-packages/numpy/core/include (ver 1.13.3)
    packages path:               lib/python2.7/dist-packages

  Python 3:
    Interpreter:                 /usr/bin/python3 (ver 3.6.8)
    Libraries:                   /usr/lib/aarch64-linux-gnu/libpython3.6m.so (ver 3.6.8)
    numpy:                       /usr/local/lib/python3.6/dist-packages/numpy/core/include (ver 1.17.1)
    packages path:               lib/python3.6/dist-packages

  Python (for build):            /usr/bin/python2.7

  Java:
    ant:                         NO
    JNI:                         NO
    Java wrappers:               NO
    Java tests:                  NO

  Install to:                    /usr/local
-----------------------------------------------------------------

Any idea? I don’t really know how to quit from this condition…

2

If I change line 8 to:

cap = cv2.VideoCapture("rtsp://admin:Password@192.168.0.200:554/")

Visualization seems less fluid but CPU utilization drops down to 60-70%

3

Hi,
The working pipeline should be

gst = "rtspsrc location=rtsp://admin:Password@192.168.0.200:554/ latency=200 ! queue ! rtph264depay ! queue ! h264parse ! omxh264dec ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink"

Please run ‘sudo tegrastats’ to check CPU usage. And ‘sudo jetson_clocks’ to run in max performance.

4

Thank you!
I’m trying to undestand GStreamer but is very complex compared to FFMPEG.

5

Hi,

I have been trying different ideas but can’t figure out how to get smooth motion display from gstreamer output. I tried the gst suggested by @DaneLLL as well. I am using Python with openCV 4.5 compiled with CUDA, tried both nvv4l2decoder and omxh264dec. None of them is giving me a smooth motion video. It seems the openCV call read() in the loop after cv2.VideoCapture() takes periodically longer time and that is why the display showing periodic jerking. I tried placing queue in several places and lastly as suggested by @DaneLLL. My platform is a Nano running JP4.4.

Any help would be greatly appreciated.
Thanks in advance.

6

Found the solution.
Taking the latency off, solved my problem.