How to decode videos using opencv python

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
,
#1

I am using this base image nvcr.io/nvidia/deepstream-l4t:5.0.1-20.09-samples

And trying to use hardware decoders present on Jetson to decode rtsp streams using below sample code.

import cv2

def jetson_rtsp_capture(uri, width, height, latency):
    gst_str = ('rtspsrc location={} latency={} ! '
            'rtph264depay ! h264parse ! omxh264dec ! '
            'nvvidconv ! '
            'video/x-raw, width=(int){}, height=(int){}, '
            'format=(string)BGRx ! '
            'videoconvert ! appsink').format(uri, latency, width, height)

    return cv2.VideoCapture(gst_str, cv2.CAP_GSTREAMER)

cap = jetson_rtsp_capture("rtsp://user:password@ip", width=1920, height=1080, latency=200)

while True:
    _, f = cap.read()
    
    if f is not None:
        print(f.shape)

This code snippet works on Jetson but not inside the container, can you please give some pointers on how to make it work inside the container.

Do I need to build opencv or Is there a way to mount host system opencv files inside the container.
Any help will be much appreciated.

Thanks

#3

Hi,
We have deprecated omx plugins. Please use nvv4l2decoder instead of omxh264dec.

#4

Hey @DaneLLL,

Replacing nvv4l2decoder with omxh264dec didn’t help, It is still not able to read frames.

#5

Hi,
Please run gst-inspect-1.0 nvvideoconvert and see if the plugin is present. If yes, please replace nvvidconv with nvvideoconvert and try.

#6

Hey,

I tried this command gst-inspect-1.0 nvvideoconvert, here is the output

root@36fed6a54fa9:/app# gst-inspect-1.0 nvvideoconvert
nvbuf_utils: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbuf_utils: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
2021-07-02 10:44:15.175240: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
nvbufsurftransform: Could not get EGL display connection
2021-07-02 10:44:15.716571: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
No such element or plugin 'nvvideoconvert'

This means nvvideoconvert plugin is not available.
Also since it says Could not get EGL display connection Please check docker run command if I made any mistake

export DISPLAY=:1
xhost +
sudo docker run --rm -it --gpus all \
    -v $(pwd):/app \
    -e DISPLAY=$DISPLAY -v /tmp/.X11-unix/:/tmp/.X11-unix \
    --ipc=host \
    ds_container bash

I tried this command aswell gst-inspect-1.0 nvvidconv, here is the output

root@36fed6a54fa9:/app# gst-inspect-1.0 nvvidconv     
nvbuf_utils: Could not get EGL display connection
Factory Details:
  Rank                     primary (256)
  Long-name                NvVidConv Plugin
  Klass                    Filter/Converter/Video/Scaler
  Description              Converts video from one colorspace to another & Resizes
  Author                   amit pandya <apandya@nvidia.com>

Plugin Details:
  Name                     nvvidconv
  Description              video Colorspace conversion & scaler
  Filename                 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvidconv.so
  Version                  1.2.3
  License                  Proprietary
  Source module            gstreamer-nvvconv-plugin
  Binary package           GStreamer nvvconv Plugin
  Origin URL               http://nvidia.com/

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBaseTransform
                         +----Gstnvvconv

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw(memory:NVMM)
                 format: { (string)I420, (string)I420_10LE, (string)P010_10LE, (string)UYVY, (string)YUY2, (string)YVYU, (string)NV12, (string)NV16, (string)NV24, (string)GRAY8, (string)BGRx, (string)RGBA }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw
                 format: { (string)I420, (string)UYVY, (string)YUY2, (string)YVYU, (string)NV12, (string)NV16, (string)NV24, (string)GRAY8, (string)BGRx, (string)RGBA }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
  
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw(memory:NVMM)
                 format: { (string)I420, (string)I420_10LE, (string)P010_10LE, (string)I420_12LE, (string)UYVY, (string)YUY2, (string)YVYU, (string)NV12, (string)NV16, (string)NV24, (string)GRAY8, (string)BGRx, (string)RGBA }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw
                 format: { (string)I420, (string)UYVY, (string)YUY2, (string)YVYU, (string)NV12, (string)NV16, (string)NV24, (string)P010_10LE, (string)GRAY8, (string)BGRx, (string)RGBA }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SINK: 'sink'
    Pad Template: 'sink'
  SRC: 'src'
    Pad Template: 'src'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "nvvconv0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  qos                 : Handle Quality-of-Service events
                        flags: readable, writable
                        Boolean. Default: false
  silent              : Produce verbose output ?
                        flags: readable, writable
                        Boolean. Default: false
  flip-method         : video flip methods
                        flags: readable, writable, controllable
                        Enum "GstNvVideoFlipMethod" Default: 0, "none"
                           (0): none             - Identity (no rotation)
                           (1): counterclockwise - Rotate counter-clockwise 90 degrees
                           (2): rotate-180       - Rotate 180 degrees
                           (3): clockwise        - Rotate clockwise 90 degrees
                           (4): horizontal-flip  - Flip horizontally
                           (5): upper-right-diagonal - Flip across upper right/lower left diagonal
                           (6): vertical-flip    - Flip vertically
                           (7): upper-left-diagonal - Flip across upper left/lower right diagonal
  output-buffers      : number of output buffers
                        flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state
                        Unsigned Integer. Range: 1 - 4294967295 Default: 4 
  interpolation-method: Set interpolation methods
                        flags: readable, writable, controllable
                        Enum "GstInterpolationMethod" Default: 0, "Nearest"
                           (0): Nearest          - Nearest
                           (1): Bilinear         - Bilinear
                           (2): 5-Tap            - 5-Tap
                           (3): 10-Tap           - 10-Tap
                           (4): Smart            - Smart
                           (5): Nicest           - Nicest
  left                : Pixels to crop at left
                        flags: readable, writable
                        Integer. Range: 0 - 2147483647 Default: 0 
  right               : Pixels to crop at right
                        flags: readable, writable
                        Integer. Range: 0 - 2147483647 Default: 0 
  top                 : Pixels to crop at top
                        flags: readable, writable
                        Integer. Range: 0 - 2147483647 Default: 0 
  bottom              : Pixels to crop at bottom
                        flags: readable, writable
                        Integer. Range: 0 - 2147483647 Default: 0

Thanks

#7

You may try something like :

xhost +
docker run -it --rm --net=host --runtime nvidia -e DISPLAY=$DISPLAY --privileged --ipc=host -v /tmp/.X11-unix/:/tmp/.X11-unix/ ds_container bash
Glmark2 Error: main: could not initialize canvas
#8

I am using remote ssh so without export DISPLAY=:1 it won’t run.

#9

Hi,
The nvvideoconvert plugin is a key plugin in DeepStream SDK. A bit strange you are not able to see it in gst-launch-1.0 command.

Please try this docker:
NVIDIA NGC
In the docker you should be able to use nvvidconv plugin.

#11

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.