How to stream frames using gstreamer, with opencv in python?

Hello there, I want to stream the object detection result frame using gstreamer in my Jetson Xavier, here’s my pipeline:

  1. capture frames from ip camera using opencv-python; √
  2. do the image preprocesing and infence it with mxnet; √
  3. draw the detected bbox on the origin frame; √
  4. stream these frames via gstreamer RTSP, using opencv. ×
  5. open vlc player to watch the real-time frames. ×

I don’t know how to implement step 4, I found some implementation but didn’t work for me:
python-opencv-gstreamer-examples

gst_str_rtp = "appsrc ! videoconvert ! x264enc tune=zerolatency bitrate=500 speed-preset=superfast ! rtph264pay ! udpsink host=10.168.1.177 port=5000"

## do something

out = cv2.VideoWriter(gst_str_rtp, 0, fps, (frame_width, frame_height), True)

while:
    # 1. capture frame
    # 2. inference
    # 3. draw boxes
    out.write(frame)

And then open the VLC player, enter the address: rtp://10.168.1.177:5000, but can not open it.

Why didn’t I try other frameworks or cpp or deepstream?

First, my mobilenet+yolov3 model was trained in MXNet, I don’t want to train it again with other frameworks.

Second, I know the deepstream example app can do the RTSP streaming, but I can’t deploy my MXNet model with deepstream. So I trained a mobilenet + ssd using tensorflow, attempt to deploy it using deepstream, but something blocked my road: ERROR: sample_uff_ssd: Fail to parse

Maybe I can try to write gstreamer pipeline in cpp, but there’s a lot work to do to translate my python inference code to cpp, so I’m seeking python version solution.

Thanks everyone.

You may have a look to this topic.
In recent versions of opencv, the API might have changed. Also have a look to this example.

Thanks for your reply, my opencv is ver 4.1, gstreamer supported.

Here’s my test code, gst pipeline copied from your code:

import cv2

cap = cv2.VideoCapture("rtsp://admin:xxxx@10.168.1.248:554/h264/ch1/main/av_stream")

out = cv2.VideoWriter("appsrc ! video/x-raw, format=BGR ! queue ! videoconvert ! video/x-raw, format=BGRx ! nvvidconv ! omxh264enc ! video/x-h264, stream-format=byte-stream ! h264parse ! rtph264pay pt=96 config-interval=1 ! udpsink host=10.168.1.177 port=50001", cv2.CAP_GSTREAMER, 0, 25.0, (1920,1080))

while cap.isOpened():
    ret, frame = cap.read()
    if ret:
        if out.isOpened():
            out.write(frame)
            print('writing frame')
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break
    else:
        break

# Release everything if job is finished
cap.release()
out.release()

The printed info shows that it actually keep writing frames, but my vlc player can not play it.

I suspect that my capture pipeline is not correct, does it matter?

And, what’s the correct URL put into VLC player in this case? rtp:// or rtsp:// or udp:// ?

1 Like

Your capture is not a gstreamer pipeline, opencv will try to decode itself, but using cpu only is not efficient.
You would try:

cap = cv2.VideoCapture("rtspsrc location=rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mov ! application/x-rtp, media=video ! rtph264depay ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink", cv2.CAP_GSTREAMER)

You may also add:

cv2.imshow(frame)

in the loop so that you can see the image and check your capture.

I have bad experience of vlc on Jetsons so I can’t really help.
From what I’ve understood you would create a sdp file with something like:

c=IN IP4 10.168.1.177
m=video 50001 RTP/AVP 96
a=rtpmap:96 H264/90000

and give a url to this file. Not tried, though.

Anyway you could display your video with gstreamer with:

gst-launch-1.0 -e udpsrc port=50001 ! application/x-rtp, encoding-name=H264, payload=96 ! queue ! rtph264depay ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! xvimagesink

I’ll try the correct capture pipeline.
May I ask how to learn gstreamer efficiently? Since these elements are magic to me, make me so confused when to use them.

You may find some overview looking at this, but you’ll probably learn by trying/correcting yourself (usually not harmful, but when saving to file with filesink, be aware that video may result in huge sizes thus better save to an external disk so that you won’t fill your rootfs).

In short, a gstreamer pipeline has at least a source, a sink, and possibly various other elements depending on what you want to do.
Using gst-launch is the simplest way for trying. Simple pipeline (a test source and display in a X window):

gst-launch-1.0 videotestsrc ! xvimagesink

Between plugin elements in the pipeline, you can also specify caps, which are properties of your stream at that point. Note that if you have to use parenthesis, with gst-launch you would add simple quotes around in order to prevent shell from interpreting these. There should be at least one cap matching both your previous plugin src (output) capabilitity with next plugin sink (input) capability, otherwise the pipeline wouldn’t be able to link plugins.

The following pipeline reads from CSI camera in 640x480@30fps and debayers with ISP into NV12 video format in NVMM memory (DMAable for HW encoders/decoders or GPU or ISP), then uses nvvidconv plugin for converting into YUY2 format and copying into CPU memory with ISP, and finally displays with xvimagesink:

gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM), format=NV12, width=640, height=480, framerate=30/1' ! nvvidconv ! video/x-raw, format=YUY2 ! xvimagesink

For exploring:

gst-inspect-1.0

will list the available plugins, and

gst-inspect-1.0 any_plugin 

would display plugin supported SINK and SRC caps as well as options and default values.

You may use -v (verbose) with gst-launch in order to get details.
You may also use GST_DEBUG, but be aware that it can generate a huge amount of info (here setting debug level 3 for each plugin):

GST_DEBUG=*:3 gst-launch-1.0 -v nvarguscamerasrc ! 'video/x-raw(memory:NVMM), format=NV12, width=640, height=480, framerate=30/1' ! nvvidconv ! video/x-raw, format=YUY2 ! xvimagesink

In case you would be stuck, feel free to create a topic and you may get help.