Gstreamer Pipeline Optimization

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
,
1

I’m trying to use VideoCapture and VideoWriter to stream a camera feed using python. I need to capture the frame in order to do some basic processing before sending it to the writer. However, I noticed that my pipeline has significant 1-2 ms delay compared to the pipeline I run in the terminal. I think it’s because I have more video conversion in the python pipeline, but I’m not sure if I’m able to optimzie/shorten the pipeline.

Here is my python script:


import cv2
import keyboard

if __name__ == '__main__':
    normal_cam = cv2.VideoCapture("v4l2src device=/dev/video2 io-mode=2 ! video/x-raw,format=YUY2,width=2560,height=720,framerate=60/1 ! nvvidconv ! \
        video/x-raw(memory:NVMM) ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink drop=0", cv2.CAP_GSTREAMER)

    #start gstreamer video writer
    gst = "appsrc ! queue ! videoconvert ! video/x-raw, format=RGBA ! nvvidconv ! \
           video/x-raw(memory:NVMM) ! xvimagesink "
    vw = cv2.VideoWriter(gst, cv2.CAP_GSTREAMER, 0, 30, (2560, 720))

    while True:
        if keyboard.is_pressed('q'):
            break
        _, frame = normal_cam.read()
        vw.write(frame)

cv2.destroyAllWindows()

And this is what I run in the terminal:

gst-launch-1.0 v4l2src device=/dev/video2 ! video/x-raw,format=YUY2,width=2560,height=720,framerate=30/1 ! xvimagesink

Obviously the terminal script is much shorter, however I’m unsure if I can shorten the python script while achieving the same functionality.

2

I believe the latency is caused by the videoconvert in my VideoCapture object. Is there another way to convert from BGRx to BGR

3

I found this thread that covers the same issue on a Raspberry Pi - Convert color format using GPU in Gstreamer (alternative to videoconvert). - Raspberry Pi Forums. Is there a way to use v4l2video12convert on the Jetson to reduce the CPU load?

4

You can also use only one nvvidconv

Something like:

nvvidconv ! video/x-raw(memory:NVMM),format=BGRx

Instead of:

nvvidconv !  video/x-raw(memory:NVMM) ! nvvidconv ! video/x-raw,format=BGRx
5

When I try that I get
[ WARN:0] global /tmp/build_opencv/opencv/modules/videoio/src/cap_gstreamer.cpp (1631) writeFrame OpenCV | GStreamer warning: cvWriteFrame() needs images with depth = IPL_DEPTH_8U and nChannels = 3.

6

I also tried changing my pipeline to

"v4l2src device=/dev/video2 io-mode=2 ! video/x-raw,format=YUY2,width=2560,height=720,framerate=60/1 ! nvvidconv ! \
        video/x-raw(memory:NVMM) ! nvvidconv ! video/x-raw,format=BGRx ! h264parse ! v4l2h264dec capture-io-mode=4 ! v4l2convert output-io-mode=5 capture-io-mode=4 ! video/x-raw, format=BGR ! appsink drop=0"

And I get the same error: [ WARN:0] global /tmp/build_opencv/opencv/modules/videoio/src/cap_gstreamer.cpp (1631) writeFrame OpenCV | GStreamer warning: cvWriteFrame() needs images with depth = IPL_DEPTH_8U and nChannels = 3.

8

Hi,
For using xvimagesink plugin, you don’t need to convert to video/x-raw(memory:NVMM). Please try

    gst = "appsrc ! queue ! videoconvert ! xvimagesink "

And it takes significant CPU usage when running OpenCV on Jetson platforms. Please execute

$ sudo nvpmodel -m 0
$ sudo jetson_clocks

to run CPU cores at max clock.

9

I will try both of those but so far there isn’t much noticeable difference, I believe the bottleneck is still the videoconvert command. The nvpmodel command puts my system in 2 core 15W mode. Would there be any difference changing the power to either 20W or 10W?

10

Hi,
Since the overhead is on CPU, performance should be better if more CPU cores are online. Please give it a try.

You can get system status by executing sudo tegrastats.

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.