How to decode videos using opencv python

NitinRai · July 1, 2021, 10:34am

I am using this base image nvcr.io/nvidia/deepstream-l4t:5.0.1-20.09-samples

And trying to use hardware decoders present on Jetson to decode rtsp streams using below sample code.

import cv2

def jetson_rtsp_capture(uri, width, height, latency):
    gst_str = ('rtspsrc location={} latency={} ! '
            'rtph264depay ! h264parse ! omxh264dec ! '
            'nvvidconv ! '
            'video/x-raw, width=(int){}, height=(int){}, '
            'format=(string)BGRx ! '
            'videoconvert ! appsink').format(uri, latency, width, height)

    return cv2.VideoCapture(gst_str, cv2.CAP_GSTREAMER)

cap = jetson_rtsp_capture("rtsp://user:password@ip", width=1920, height=1080, latency=200)

while True:
    _, f = cap.read()
    
    if f is not None:
        print(f.shape)

This code snippet works on Jetson but not inside the container, can you please give some pointers on how to make it work inside the container.

Do I need to build opencv or Is there a way to mount host system opencv files inside the container.
Any help will be much appreciated.

Thanks

DaneLLL · July 2, 2021, 1:35am

Hi,
We have deprecated omx plugins. Please use nvv4l2decoder instead of omxh264dec.

NitinRai · July 2, 2021, 7:26am

Hey @DaneLLL,

Replacing nvv4l2decoder with omxh264dec didn’t help, It is still not able to read frames.

DaneLLL · July 2, 2021, 8:15am

Hi,
Please run gst-inspect-1.0 nvvideoconvert and see if the plugin is present. If yes, please replace nvvidconv with nvvideoconvert and try.

NitinRai · July 2, 2021, 10:49am

Hey,

I tried this command gst-inspect-1.0 nvvideoconvert, here is the output

root@36fed6a54fa9:/app# gst-inspect-1.0 nvvideoconvert
nvbuf_utils: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbuf_utils: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
2021-07-02 10:44:15.175240: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
nvbufsurftransform: Could not get EGL display connection
2021-07-02 10:44:15.716571: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
nvbufsurftransform: Could not get EGL display connection
No such element or plugin 'nvvideoconvert'

This means nvvideoconvert plugin is not available.
Also since it says Could not get EGL display connection Please check docker run command if I made any mistake

export DISPLAY=:1
xhost +
sudo docker run --rm -it --gpus all \
    -v $(pwd):/app \
    -e DISPLAY=$DISPLAY -v /tmp/.X11-unix/:/tmp/.X11-unix \
    --ipc=host \
    ds_container bash

I tried this command aswell gst-inspect-1.0 nvvidconv, here is the output

root@36fed6a54fa9:/app# gst-inspect-1.0 nvvidconv     
nvbuf_utils: Could not get EGL display connection
Factory Details:
  Rank                     primary (256)
  Long-name                NvVidConv Plugin
  Klass                    Filter/Converter/Video/Scaler
  Description              Converts video from one colorspace to another & Resizes
  Author                   amit pandya <apandya@nvidia.com>

Plugin Details:
  Name                     nvvidconv
  Description              video Colorspace conversion & scaler
  Filename                 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvidconv.so
  Version                  1.2.3
  License                  Proprietary
  Source module            gstreamer-nvvconv-plugin
  Binary package           GStreamer nvvconv Plugin
  Origin URL               http://nvidia.com/

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBaseTransform
                         +----Gstnvvconv

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw(memory:NVMM)
                 format: { (string)I420, (string)I420_10LE, (string)P010_10LE, (string)UYVY, (string)YUY2, (string)YVYU, (string)NV12, (string)NV16, (string)NV24, (string)GRAY8, (string)BGRx, (string)RGBA }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw
                 format: { (string)I420, (string)UYVY, (string)YUY2, (string)YVYU, (string)NV12, (string)NV16, (string)NV24, (string)GRAY8, (string)BGRx, (string)RGBA }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
  
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw(memory:NVMM)
                 format: { (string)I420, (string)I420_10LE, (string)P010_10LE, (string)I420_12LE, (string)UYVY, (string)YUY2, (string)YVYU, (string)NV12, (string)NV16, (string)NV24, (string)GRAY8, (string)BGRx, (string)RGBA }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw
                 format: { (string)I420, (string)UYVY, (string)YUY2, (string)YVYU, (string)NV12, (string)NV16, (string)NV24, (string)P010_10LE, (string)GRAY8, (string)BGRx, (string)RGBA }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SINK: 'sink'
    Pad Template: 'sink'
  SRC: 'src'
    Pad Template: 'src'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "nvvconv0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  qos                 : Handle Quality-of-Service events
                        flags: readable, writable
                        Boolean. Default: false
  silent              : Produce verbose output ?
                        flags: readable, writable
                        Boolean. Default: false
  flip-method         : video flip methods
                        flags: readable, writable, controllable
                        Enum "GstNvVideoFlipMethod" Default: 0, "none"
                           (0): none             - Identity (no rotation)
                           (1): counterclockwise - Rotate counter-clockwise 90 degrees
                           (2): rotate-180       - Rotate 180 degrees
                           (3): clockwise        - Rotate clockwise 90 degrees
                           (4): horizontal-flip  - Flip horizontally
                           (5): upper-right-diagonal - Flip across upper right/lower left diagonal
                           (6): vertical-flip    - Flip vertically
                           (7): upper-left-diagonal - Flip across upper left/lower right diagonal
  output-buffers      : number of output buffers
                        flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state
                        Unsigned Integer. Range: 1 - 4294967295 Default: 4 
  interpolation-method: Set interpolation methods
                        flags: readable, writable, controllable
                        Enum "GstInterpolationMethod" Default: 0, "Nearest"
                           (0): Nearest          - Nearest
                           (1): Bilinear         - Bilinear
                           (2): 5-Tap            - 5-Tap
                           (3): 10-Tap           - 10-Tap
                           (4): Smart            - Smart
                           (5): Nicest           - Nicest
  left                : Pixels to crop at left
                        flags: readable, writable
                        Integer. Range: 0 - 2147483647 Default: 0 
  right               : Pixels to crop at right
                        flags: readable, writable
                        Integer. Range: 0 - 2147483647 Default: 0 
  top                 : Pixels to crop at top
                        flags: readable, writable
                        Integer. Range: 0 - 2147483647 Default: 0 
  bottom              : Pixels to crop at bottom
                        flags: readable, writable
                        Integer. Range: 0 - 2147483647 Default: 0

Thanks

Honey_Patouceul · July 2, 2021, 11:22am

You may try something like :

xhost +
docker run -it --rm --net=host --runtime nvidia -e DISPLAY=$DISPLAY --privileged --ipc=host -v /tmp/.X11-unix/:/tmp/.X11-unix/ ds_container bash

NitinRai · July 2, 2021, 11:26am

I am using remote ssh so without export DISPLAY=:1 it won’t run.

DaneLLL · July 2, 2021, 1:25pm

Hi,
The nvvideoconvert plugin is a key plugin in DeepStream SDK. A bit strange you are not able to see it in gst-launch-1.0 command.

Please try this docker:
NVIDIA L4T Base | NVIDIA NGC
In the docker you should be able to use nvvidconv plugin.

system · September 5, 2021, 7:36am

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
I want to adapt opencv code to gstreamer in jetson Jetson TX1	8	3137	October 18, 2021
Input buffer number of surfaces (0) must be equal to mux->num_surfaces_per_frame (1) DeepStream SDK	15	1611	October 12, 2021
Met error using gstreamer decode jpeg image into opencv on Jetson Nano Jetson Nano opencv , gstreamer	5	1644	October 15, 2021
Unable to get RGB buffer in gstreamer/python pipeline DeepStream SDK	10	2152	October 12, 2021
Nvvidconv colorspace conversion difficulties Jetson Nano	6	1929	October 14, 2021
revising streaming with Xavier Jetson AGX Xavier	12	1254	October 18, 2021
Problems with "... ! nvvidconv ! video/x-raw, format=P010_10LE ! ..." DeepStream SDK gstreamer	10	1626	May 18, 2023
Docker gstreamer Jetson Nano gstreamer , docker	6	2924	October 15, 2021
nvvidconv not working with RGB Jetson TX1	5	5518	October 18, 2021
gst-0.10 nvvidconv sink cap problem Jetson TK1	2	1102	October 18, 2021

How to decode videos using opencv python

Related topics