OpenCV extremely slow with gstreamer camera

jetsonnvidia · July 14, 2021, 10:32pm

I have a program in C++ that detects Hough lines using OpenCV and a USB webcam captured at 1280x720 resolution.

I have changed the webcam for a Raspberry Pi camera and so I have to use gstreamer. I have this line to set up the gstreamer capture:

VideoCapture rpicam("nvarguscamerasrc wbmode=4 ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)30/1 ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink");

But it is MUCH slower than the USB webcam. I was getting about 5-10fps with the USB webcam. Now I get 2-3fps with the Raspberry Pi camera.

There is also 3-4 seconds of lag!!! I used to have about 0.2-0.3s lag.

Any ideas what I am doing wrong or how to speed it up?

Thanks.

jrevenko · July 15, 2021, 12:37am

Have you tried: drop=true, after appsink?
DISCLAIMER: i’m a newbie, but it worked for me, (I also compiled OpenCV 4.5.2 with Cuda. Although i dont know if that helped too)

DaneLLL · July 15, 2021, 1:22am

Hi,
Please also execute sudo nvpmodel -m 0 and sudo jetson_clocks. There are BGRx to BGR convertsion and buffer copy in the pipeline. The operations require significant CPU usage, so suggest run CPU cores at max clock.

As comparison, you may also check performance of gst-launch-1.0:

gst-launch-1.0 -v nvarguscamerasrc wbmode=4 ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)30/1 ! nvvidconv ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0

jetsonnvidia · July 15, 2021, 10:50am

I would like to correct myself; the main difference between the USB camera and the Raspberry Pi camera is the massive lag. The frame rate appears similar.

Thanks for your reply. This had no improvement, probably as I am already running at max speed with a decent 4A PSU. Here is my top output:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                                                               
 9613 nano      20   0 1191160 243696  45288 S  34.6  6.0   0:09.92 a.out                                                                                                                                                                                                                 
 9530 root      20   0 10.297g 170252  35576 S  14.4  4.2   0:04.83 nvargus-daemon                                                                                                                                                                                                        
 5847 root      20   0 6542620  45344  30712 S   3.8  1.1   0:34.00 Xorg                                                                                                                                                                                                                  
 9688 nano      20   0    9164   3680   2908 R   3.8  0.1   0:00.38 top

My OpenCV program is only taking 35% of 1 CPU so why is it so slow? Is it memory bandwidth limited? Here is the output of the fakesink line you suggested:

nano@jetson-nano:/nfs/mmapi_samples/samples/ralph_webcam$ gst-launch-1.0 -v nvarguscamerasrc wbmode=4 ! "video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)30/1" ! nvvidconv ! "video/x-raw, format=(string)BGRx" ! videoconvert ! "video/x-raw, format=(string)BGR" ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = false
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)30/1, format=(string)BGRx
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)30/1, format=(string)BGRx
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:src: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)30/1, format=(string)BGR
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:src: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)30/1, format=(string)BGR
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)30/1, format=(string)BGR
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0.GstPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)30/1, format=(string)BGR
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)30/1, format=(string)BGR
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)30/1, format=(string)BGR
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)30/1, format=(string)BGRx
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw, width=(int)1280, height=(int)720, framerate=(fraction)30/1, format=(string)BGRx
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)30/1
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 5 
   Output Stream W = 1280 H = 720 
   seconds to Run    = 0 
   Frame Rate = 120.000005 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = false
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 17, dropped: 0, current: 32.57, average: 32.57
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 33, dropped: 0, current: 30.05, average: 31.29
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 48, dropped: 0, current: 29.97, average: 30.87
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 64, dropped: 0, current: 30.03, average: 30.65
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 80, dropped: 0, current: 30.00, average: 30.52
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 95, dropped: 0, current: 29.97, average: 30.43
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 111, dropped: 0, current: 30.04, average: 30.37
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 127, dropped: 0, current: 30.00, average: 30.32
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 143, dropped: 0, current: 30.00, average: 30.29
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 159, dropped: 0, current: 30.00, average: 30.26
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 175, dropped: 0, current: 30.00, average: 30.23
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 191, dropped: 0, current: 29.99, average: 30.21
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 207, dropped: 0, current: 30.01, average: 30.20
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 223, dropped: 0, current: 30.00, average: 30.18
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 238, dropped: 0, current: 29.99, average: 30.17
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 254, dropped: 0, current: 30.01, average: 30.16
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 269, dropped: 0, current: 30.00, average: 30.15
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 285, dropped: 0, current: 30.00, average: 30.14
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 301, dropped: 0, current: 30.00, average: 30.14
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 317, dropped: 0, current: 30.00, average: 30.13
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 332, dropped: 0, current: 29.98, average: 30.12
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:11.911327355
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
GST_ARGUS: Cleaning up
CONSUMER: Done Success
GST_ARGUS: Done Success
Setting pipeline to NULL ...
Freeing pipeline ...

jetsonnvidia · July 15, 2021, 10:51am

Hi, the reply is much appreciated. Your comment has massively reduced the lag I was seeing.

jetsonnvidia · July 15, 2021, 11:08am

Thank you both for your help. The CPU issue was unrelated to that discussed here but thanks for helping me with the lag issue.

system · September 19, 2021, 7:23am

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.