Low FPS and freeze while streaming high resolution cameras with opencv

[Elad-SH]

I’ve recently purchased a new camera that is capable of streaming 60fps on HD resolution. while using the most basic gst-launch-1.0 line : gst-launch-1.0 v4l2src ! xvimagesink it opens the camera on high res with high FPS with no lag and no freezing.

However, when trying to start the camera on opencv VideoCapture module, anything above 640x480 freezes a lot and gives very low FPS. My best guess is that my pipelines are not ideal and I am not running the camera as intended on HW accelerated pipeline, but rather eating up my CPU which can’t handle the stream, as well as running on an uncompressed format UYVY.

A few pipelines examples that i’ve tried:

cv2.VideoCapture('v4l2src device=/dev/video0 ! video/x-raw,width=1920,height=1080! nvvidconv flip-method=2 ! video/x-raw(memory:NVMM), format=I420! nvvidconv ! video/x-raw, format=BGRx , width=1920, height=1080, framerate=60/1  ! videoconvert ! video/x-raw, format=BGR ! appsink drop=True sync=False')

cv2.VideoCapture('v4l2src device=/dev/video0 !  videoconvert ! video/x-raw, format=BGR ! appsink')

Would appreciate any help and examples of better pipelines, I haven’t found many v4l2src examples if any and non worked well for me.

i should add basic camera information from gst-device-monitor-1.0 :

Device found:

name  : See3CAM_CU30
class : Video/Source
caps  : video/x-raw, format=(string)UYVY, width=(int)2304, height=(int)1536, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 24/1, 12/1 };
        video/x-raw, format=(string)UYVY, width=(int)2048, height=(int)1536, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 42/1, 21/1 };
        video/x-raw, format=(string)UYVY, width=(int)2304, height=(int)1296, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 15/1 };
        video/x-raw, format=(string)UYVY, width=(int)1920, height=(int)1280, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 50/1, 25/1 };
        video/x-raw, format=(string)UYVY, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 60/1, 30/1, 15/1 };
        video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)960, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 58/1, 30/1 };
        video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 60/1, 30/1 };
        video/x-raw, format=(string)UYVY, width=(int)1152, height=(int)768, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 60/1, 30/1 };
        video/x-raw, format=(string)UYVY, width=(int)640, height=(int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 60/1, 30/1 };
        image/jpeg, width=(int)2304, height=(int)1536, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)48/1;
        image/jpeg, width=(int)2048, height=(int)1536, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)50/1;
        image/jpeg, width=(int)2304, height=(int)1296, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)60/1;
        image/jpeg, width=(int)1920, height=(int)1280, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)50/1;
        image/jpeg, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)60/1;
        image/jpeg, width=(int)1280, height=(int)960, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)58/1;
        image/jpeg, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)60/1;
        image/jpeg, width=(int)1152, height=(int)768, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)60/1;
        image/jpeg, width=(int)640, height=(int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)60/1;
properties:
    udev-probed = true
    device.bus_path = platform-3610000.xhci-usb-0:3.4:1.0
    sysfs.path = /sys/devices/3610000.xhci/usb2/2-3/2-3.4/2-3.4:1.0/video4linux/video0
    device.bus = usb
    device.subsystem = video4linux
    device.vendor.id = 2560
    device.vendor.name = "e-con\\x20systems"
    device.product.id = c130
    device.product.name = See3CAM_CU30
    device.serial = e-con_systems_See3CAM_CU30_152B9904
    device.capabilities = :capture:
    device.api = v4l2
    device.path = /dev/video0
    v4l2.device.driver = uvcvideo
    v4l2.device.card = See3CAM_CU30
    v4l2.device.bus_info = usb-3610000.xhci-3.4
    v4l2.device.version = 264701 (0x000409fd)
    v4l2.device.capabilities = 2216689665 (0x84200001)
    v4l2.device.device_caps = 69206017 (0x04200001)
gst-launch-1.0 v4l2src ! ...

[Out_of_the_BOTS]

I have been playing with gstreamer and openCV in python on the Jetson Xaiver (it has been a journey).

It has taken me sometime to get it to work well without lag but I ahve learnt a lot of tuing along the way. First is that it can be difficult to get gstreamer and openCv to work together without these problems.

A few things here to consider:

1. USB cams have much slower bus speeds than CSI cameras as USB is serial and CSI is 8 parallel.

2. you can use USB cam with openCV and not use gstreamer but rather get frames from the v4l2src directly with cap = cv2.VideoCapture(“/dev/video0”, cv2.CAP_V4L2)

3.The other thing that I found was the gstreamer doesn’t like dropping frames but rather tried to buffer them up so that no frames are lost and this causes a lag if at any point openCV isn’t asking for frame quick enough. I found add this to your appsink to drop buffered frames “! appsink drop=true sync=false” then openCV will grab latest frame coming off the cmaera

[Elad-SH]

unfortunately my problem isn’t with delay or lag, it is with very low FPS and freezing, so (3) didn’t really does much, and i already have it implemented in my pipeline.
i know USB cams are slower but i need the cable length that a good CSI cam does not provide, at least not in my experience, mipi cables are very short(<20cm).

The main issue is that OpenCV uses the CPU to handle the camera, or at least it does so with my provided pipeline, i need a GPU based pipeline to grab frames or else i’ll never be able to use high resolution unless i get a very high performence CPU machine.

[ShaneCCC]

v4l2src may involve memory copy hurt the performance. Maybe try nvv4l2camerasrc if help on it.

[Elad-SH]

Might need help building a pipeline that uses that because I can’t seem to launch the camera with it as a source

[ShaneCCC]

I mean modify your cv source to nvvl42camerasrc.

v4l2src device=/dev/video0 ! video/x-raw,width=1920,height=1080!

nvv4l2camerasrc device=/dev/video0 ! video/x-raw(memory:NVMM),width=1920,height=1080

[Out_of_the_BOTS]

Have you tried not using gstreamer but rather get the fram esdirectly from the source

cap = cv2.VideoCapture(“/dev/video0”, cv2.CAP_V4L2)

[Elad-SH]

Can i control camera settings using that without having to use capture.set()? because i am not sure if the actual camera settings are being changed using set, or is it just resize and other changes.