RTSP Appsink and Appsrc on Jetson TX2 - CPU works, but not GPU

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

Hey guys,

The following code works well without cv2.CAP_GSTREAMER in cv2.VideoCapture. When I add cv2.GSTREAMER however (in app_stream_in), it gets hung, or fails.

Any idea why?

import cv2

gstreamer_appsink = "rtsp://user:pass@192.168.0.153:554/cam/realmonitor?channel=1&subtype=0 latency=0 ! queue ! rtph265depay ! queue ! h265parse ! omxh265dec ! nvvidconv ! video/x-raw,width=1920,height=1080,format=BGRx,framerate=(fraction)30/1 ! videoconvert ! video/x-raw,format=BGR ! appsink "

gstreamer_udpsink = "appsrc ! video/x-raw, format=(string)BGR ! videoconvert ! video/x-raw, format=(string)I420, framerate=(fraction)30/1 ! nvvidconv ! omxh265enc control-rate=2 bitrate=4000000 ! rtph265pay mtu=1400  config-interval=3 pt=96  ! udpsink host=10.0.0.176 port=5000"

print(gstreamer_appsink)
print(gstreamer_udpsink)

app_stream_in = cv2.VideoCapture(gstreamer_appsink)
app_stream_out = cv2.VideoWriter(gstreamer_udpsink, cv2.CAP_GSTREAMER, 0, 30, (1920,1080), True)

print(app_stream_in.isOpened())
print(app_stream_out.isOpened())

if not app_stream_in.isOpened() or not app_stream_out.isOpened():
    print('VideoCapture or VideoWriter not opened')
    exit(0)

n = 0 
while True:
#    print("frame", n)
    ret, frame = app_stream_in.read()

#    print(frame.shape)
#    frame = frame.reshape(1920,1080,3)
    if not ret:
        print('empty frame')
        break

    app_stream_out.write(frame)
    n += 1
2

When I include

app_stream_in = cv2.VideoCapture(gstreamer_appsink, cv2.CAP_GSTREAMER)

This is the console output, and hangs:

Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 279 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 279 
nvbuf_utils: nvbuffer Payload Type not supported
gst_nvvconv_transform: NvBufferGetParams Failed
3

Hi,
Please try cv2.VideoCapture() only.

import sys
import cv2

def read_cam():
    cap = cv2.VideoCapture("uridecodebin uri=rtspt://freja.hiof.no:1935/rtplive/definst/hessdalen02.stream ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! appsink  ")
    if cap.isOpened():
        cv2.namedWindow("demo", cv2.WINDOW_AUTOSIZE)
        while True:
            ret_val, img = cap.read();
            cv2.imshow('demo',img)
            cv2.waitKey(10)
    else:
     print "camera open failed"

    cv2.destroyAllWindows()


if __name__ == '__main__':
    read_cam()

To break down the issue is in VideoCapture() or VideoWrite().

4

Thanks @DaneLLL

The above code works, so as the one I shared for VideoCapture. Somehow though it only works without the cv2.CAP_GSTREAMER command. The one you shared above works with and without cv2.CAP_GSTREAMER with my IP camera.

But when I transcode that with H265 and set up a UDP stream, the UDP stream continuously lags, and after 5 minutes or so the lag becomes very very large and the stream is unusable (about 1 minute). Is this something to do with Python? Is it slow because I’m using OpenCV (Python) as the middle man?

Also, do you have an example for the following case in C or C++ (if Python is too slow I can’t use it).

My general use case is,

IP Camera (RTSP or UDP feed) —> Load to Python for ML Processing (Currently handled through Gstreamer and OpenCV) —> Create a RTSP or UDP stream of the processed feed and make the stream accessible to the local network

5

Hi,
You can run sudo jetson_clocks and add sync=false in appsink.

Besides, if ML processing can be port to CUDA code, you can run gstreamer pipeline like:

uridecodebin ! nvivafilter ! nvvidconv ! nvv4l2h265enc ! h265parse ! rtph265depay ! udpsink

Examples of utilizing nvivafilter:
https://devtalk.nvidia.com/default/topic/1046218/jetson-tx2/unable-to-overlay-text-when-using-udpsrc-/post/5310313/#5310313
https://devtalk.nvidia.com/default/topic/1068451/jetson-nano/new-features-for-nvivafilter/post/5412404/#5412404