OpenCV camera lag

Legion · December 13, 2020, 10:49pm

Ok, so I have two cameras connected to my Jetson Nano 2GB board :

Camera 0 :

VIDIOC_ENUM_FMT
Index       : 0
Type        : Video Capture
Pixel Format: 'MJPG' (compressed)
Name        : Motion-JPEG
	Size: Discrete 1920x1080
		Interval: Discrete 0.017s (60.000 fps)
		Interval: Discrete 0.033s (30.000 fps)
		Interval: Discrete 0.040s (25.000 fps)
		Interval: Discrete 0.050s (20.000 fps)
		Interval: Discrete 0.100s (10.000 fps)
	Size: Discrete 1600x1200
		Interval: Discrete 0.017s (60.000 fps)
		Interval: Discrete 0.033s (30.000 fps)
		Interval: Discrete 0.040s (25.000 fps)
		Interval: Discrete 0.050s (20.000 fps)
		Interval: Discrete 0.100s (10.000 fps)
	Size: Discrete 1360x768
		Interval: Discrete 0.017s (60.000 fps)
		Interval: Discrete 0.033s (30.000 fps)
		Interval: Discrete 0.040s (25.000 fps)
		Interval: Discrete 0.050s (20.000 fps)
		Interval: Discrete 0.100s (10.000 fps)
	Size: Discrete 1280x1024
		Interval: Discrete 0.017s (60.000 fps)
		Interval: Discrete 0.033s (30.000 fps)
		Interval: Discrete 0.040s (25.000 fps)
		Interval: Discrete 0.050s (20.000 fps)
		Interval: Discrete 0.100s (10.000 fps)
	Size: Discrete 1280x960
		Interval: Discrete 0.017s (60.000 fps)
		Interval: Discrete 0.033s (30.000 fps)
		Interval: Discrete 0.040s (25.000 fps)
		Interval: Discrete 0.050s (20.000 fps)
		Interval: Discrete 0.100s (10.000 fps)
	Size: Discrete 1280x720
		Interval: Discrete 0.017s (60.000 fps)
		Interval: Discrete 0.020s (50.000 fps)
		Interval: Discrete 0.033s (30.000 fps)
		Interval: Discrete 0.050s (20.000 fps)
		Interval: Discrete 0.100s (10.000 fps)
	Size: Discrete 1024x768
		Interval: Discrete 0.017s (60.000 fps)
		Interval: Discrete 0.020s (50.000 fps)
		Interval: Discrete 0.033s (30.000 fps)
		Interval: Discrete 0.050s (20.000 fps)
		Interval: Discrete 0.100s (10.000 fps)
	Size: Discrete 800x600
		Interval: Discrete 0.017s (60.000 fps)
		Interval: Discrete 0.020s (50.000 fps)
		Interval: Discrete 0.033s (30.000 fps)
		Interval: Discrete 0.050s (20.000 fps)
		Interval: Discrete 0.100s (10.000 fps)
	Size: Discrete 720x576
		Interval: Discrete 0.017s (60.000 fps)
		Interval: Discrete 0.020s (50.000 fps)
		Interval: Discrete 0.033s (30.000 fps)
		Interval: Discrete 0.050s (20.000 fps)
		Interval: Discrete 0.100s (10.000 fps)
	Size: Discrete 720x480
		Interval: Discrete 0.017s (60.000 fps)
		Interval: Discrete 0.020s (50.000 fps)
		Interval: Discrete 0.033s (30.000 fps)
		Interval: Discrete 0.050s (20.000 fps)
		Interval: Discrete 0.100s (10.000 fps)
	Size: Discrete 640x480
		Interval: Discrete 0.017s (60.000 fps)
		Interval: Discrete 0.020s (50.000 fps)
		Interval: Discrete 0.033s (30.000 fps)
		Interval: Discrete 0.050s (20.000 fps)
		Interval: Discrete 0.100s (10.000 fps)
Index       : 1
Type        : Video Capture
Pixel Format: 'YUYV'
Name        : YUYV 4:2:2
	Size: Discrete 1920x1080
		Interval: Discrete 0.200s (5.000 fps)
	Size: Discrete 1600x1200
		Interval: Discrete 0.200s (5.000 fps)
	Size: Discrete 1360x768
		Interval: Discrete 0.125s (8.000 fps)
	Size: Discrete 1280x1024
		Interval: Discrete 0.125s (8.000 fps)
	Size: Discrete 1280x960
		Interval: Discrete 0.125s (8.000 fps)
	Size: Discrete 1280x720
		Interval: Discrete 0.100s (10.000 fps)
	Size: Discrete 1024x768
		Interval: Discrete 0.100s (10.000 fps)
	Size: Discrete 800x600
		Interval: Discrete 0.050s (20.000 fps)
		Interval: Discrete 0.100s (10.000 fps)
		Interval: Discrete 0.200s (5.000 fps)
	Size: Discrete 720x576
		Interval: Discrete 0.040s (25.000 fps)
		Interval: Discrete 0.050s (20.000 fps)
		Interval: Discrete 0.100s (10.000 fps)
		Interval: Discrete 0.200s (5.000 fps)
	Size: Discrete 720x480
		Interval: Discrete 0.033s (30.000 fps)
		Interval: Discrete 0.050s (20.000 fps)
		Interval: Discrete 0.100s (10.000 fps)
		Interval: Discrete 0.200s (5.000 fps)
	Size: Discrete 640x480
		Interval: Discrete 0.033s (30.000 fps)
		Interval: Discrete 0.050s (20.000 fps)
		Interval: Discrete 0.100s (10.000 fps)
		Interval: Discrete 0.200s (5.000 fps)

Camera 1 :

VIDIOC_ENUM_FMT
Index       : 1
Type        : Video Capture
Pixel Format: 'YUYV'
Name        : YUYV 4:2:2
	Size: Discrete 1920x1080
		Interval: Discrete 0.033s (30.000 fps)

If I open them via GStreamer :
For camera 0:

$ gst-launch-1.0 tee name=stream v4l2src device=/dev/video0 ! image/jpeg,width=1280,height=720,framerate=60/1 ! jpegparse ! jpegdec ! xvimagesink sync=false

For camera 1:

$ gst-launch-1.0 v4l2src device=\"/dev/video1\" ! xvimagesink

Both are working perfectly fine - there’s no lag

But if I open them via OpenCV (v4.5) - VideoCapture():

    cv::VideoCapture cap(camId, cv::CAP_V4L);

Camera 0 works still just fine, I can even switch between settings (fourcc, resolution) with cap.set()

But Camera 1 is lagging - not like low framerate, more from time to time
Not much choice witch settings here - from what I understand I can only handle YUYV 1920x1080 if I ask about fps with .get(cv::CAP_PROP_FPS) I get expected 30 fps.

I tried cap.set(cv::CAP_PROP_BUFFERSIZE, 3) but there’s no change.

I also tried (with no success) to open it via VideoCapture GStreamer pipline

cv::VideoCapture cap("v4l2src device=/dev/video1 ! video/x-raw, format=YUY2 ! videoconvert ! video/x-raw, format=BGR ! appsink", cv::CAP_GSTREAMER);

Someone knows why there’s that lag in OpenCV, and how can I solve that? I need to show both cameras at once.

Honey_Patouceul · December 11, 2020, 11:47pm

You may explain how you’re seeing this lag from second camera. Is it using cv::imshow ? Or else ?
Also, please confirm that the following commands don’t show any errors nor lag:

gst-launch-1.0  v4l2src device=/dev/video0 ! image/jpeg,width=1280,height=720,framerate=60/1 ! jpegparse ! jpegdec ! xvimagesink       v4l2src device=/dev/video1 ! xvimagesink 

# or
gst-launch-1.0  v4l2src device=/dev/video0 ! video/x-raw,format=YUY2, width=1280,height=720, framerate=60/1 ! videoconvert! xvimagesink       v4l2src device=/dev/video1 !  video/x-raw,format=YUY2, width=1920,height=1080, framerate=30/1 !  xvimagesink 

# Or:
gst-launch-1.0 -v v4l2src device=/dev/video0 ! video/x-raw,format=YUY2, width=1280,height=720, framerate=60/1 ! videoconvert ! video/x-raw, format=BGR ! fpsdisplaysink video-sink=fakesink text-overlay=0       v4l2src device=/dev/video1 !  video/x-raw,format=YUY2, width=1920,height=1080, framerate=30/1 ! videoconvert ! video/x-raw, format=BGR ! fpsdisplaysink video-sink=fakesink text-overlay=0

Note that these cameras have different framerates. You may first set both to 30 fps and manage futher from opencv.
You may also add a queue before appsink when using these pipelines from opencv.

Legion · December 14, 2020, 9:46am

Yeah I’m showing frames by

while(true)
{
    cv::Mat frame;
    cap >> frame;
    cv::imshow("test", frame);
    cv::waitKey(1);
}

When I run this command:

gst-launch-1.0  v4l2src device=/dev/video0 ! image/jpeg,width=1280,height=720,framerate=60/1 ! jpegparse ! jpegdec ! xvimagesink       v4l2src device=/dev/video1 ! xvimagesink

Video0 is lagging much more than the other one in openCV (it looks like ~2fps), video1 is fine
The log :

legion@legion-desktop:~$ gst-launch-1.0 v4l2src device=/dev/video0 ! image/jpeg,width=1280,height=720,framerate=60/1 ! jpegparse ! jpegdec ! xvimagesink v4l2src device=/dev/video1 ! xvimagesink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.
ERROR: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: Output window was closed
Additional debug info:
xvimagesink.c(555): gst_xv_image_sink_handle_xevents (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0
Execution ended after 0:00:30.373823416
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

The other two give me this:

legion@legion-desktop:~$ gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=YUY2, width=1280,height=720, framerate=60/1 ! videoconvert! xvimagesink v4l2src device=/dev/video1 ! video/x-raw,format=YUY2, width=1920,height=1080, framerate=30/1 ! xvimagesink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:00.000323024
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
legion@legion-desktop:~$ gst-launch-1.0 -v v4l2src device=/dev/video0 ! video/x-raw,format=YUY2, width=1280,height=720, framerate=60/1 ! videoconvert ! video/x-raw, format=BGR ! fpsdisplaysink video-sink=fakesink text-overlay=0 v4l2src device=/dev/video1 ! video/x-raw,format=YUY2, width=1920,height=1080, framerate=30/1 ! videoconvert ! video/x-raw, format=BGR ! fpsdisplaysink video-sink=fakesink text-overlay=0
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink1/GstFakeSink:fakesink1: sync = true
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
ERROR: pipeline doesn't want to preroll.
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

Honey_Patouceul · December 14, 2020, 11:42am

I realize now that in RAW mode, your first camera cannot achieve 60 fps but only 10. So let’s keep with MJPG for now and use 30 fps for both cameras.

Does this work ?

gst-launch-1.0  v4l2src device=/dev/video0 ! image/jpeg,width=1280,height=720,framerate=30/1 ! jpegparse ! jpegdec ! xvimagesink       v4l2src device=/dev/video1 ! xvimagesink

jpegdec is CPU only. You may try to use HW decoder for MJPG decoding:

gst-launch-1.0  v4l2src device=/dev/video0 ! image/jpeg,width=1280,height=720,framerate=30/1 ! jpegparse ! nvv4l2decoder mjpeg=1 ! nvvidconv ! xvimagesink       v4l2src device=/dev/video1 ! xvimagesink

Also note that opencv imshow is slow on Jetson for high resolution * framerate.
You may measure without it.

Legion · December 15, 2020, 1:11pm

Ok, the first command crashed my Jetson and the second command opened cameras like this:

Camera1 opened correctly with no lag, but camera0 has some weird green line, more than half of it’s gray and it’s lagging greatly.

legion@legion-desktop:~$ gst-launch-1.0 v4l2src device=/dev/video0 ! image/jpeg,width=1280,height=720,framerate=30/1 ! jpegparse ! nvv4l2decoder mjpeg=1 ! nvvidconv ! xvimagesink v4l2src device=/dev/video1 ! xvimagesink

Setting pipeline to PAUSED ...

Opening in BLOCKING MODE

Opening in BLOCKING MODE

Pipeline is live and does not need PREROLL ...

Setting pipeline to PLAYING ...

New clock: GstSystemClock

NvMMLiteOpen : Block : BlockType = 277

NVMEDIA: Reading vendor.tegra.display-size : status: 6

NvMMLiteBlockCreate : Block : BlockType = 277

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.

Additional debug info:

gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:

There may be a timestamping problem, or this computer is too slow.

If cv::imshow can cause lagging then how can I preview my video in RT differently?

Also maybe there’s some move around? Like saving the stream to some buffer and then reading it from OpenCV?

JerryChang · December 15, 2020, 6:07am

hello Legion,

please have a try to configure memory swap, you may check Topic 158686 for reference,
thanks

Legion · December 15, 2020, 1:09pm

I think swap memory is not really used that much to pose a problem… For

gst-launch-1.0  v4l2src device=/dev/video0 ! image/jpeg,width=1280,height=720,framerate=30/1 ! jpegparse ! nvv4l2decoder mjpeg=1 ! nvvidconv ! xvimagesink       v4l2src device=/dev/video1 ! xvimagesink

And for my working Gstreamer case

gst-launch-1.0 tee name=stream v4l2src device=/dev/video0 ! image/jpeg,width=1280,height=720,framerate=60/1 ! jpegparse ! jpegdec ! xvimagesink sync=false v4l2src device=\"/dev/video1\" ! xvimagesink

And my laggy OpenCV

Now I see that’s it’s used up quite a lot of CPU, maybe that’s a problem? Can I somehow optimize it?

I also tried to instead of using cv::imshow to just save to file with 30fps, after that saved files seem really fast - 20 seconds of recording make only 3s of video. So it seems that OpenCv is rejecting or not even collecting some frames and that’s the cause of lags?

Honey_Patouceul · December 16, 2020, 5:07pm

I would advise not wasting time with opencv using gstreamer pipeline as long as you are not able to get something working from gst-launch.

Main question would be: do both pipelines only work with separate gst clocks in different processes, or just need sync=false for 1st camera.
What gives:

Same gst-clock and no sync for first display sink

gst-launch-1.0 -v v4l2src device=/dev/video0 ! image/jpeg,width=1280,height=720,framerate=30/1 ! jpegparse ! jpegdec ! xvimagesink sync=false    v4l2src device=/dev/video1 ! video/x-raw, format=YUY2, width=1920,  height=1080, framerate=30/1 ! xvimagesink

Different gst clocks but sync required for first displaysink. Run in 2 separate terminals:

1st terminal:

gst-launch-1.0 -v v4l2src device=/dev/video0 ! image/jpeg,width=1280,height=720,framerate=30/1 ! jpegparse ! jpegdec ! xvimagesink

2nd terminal:

gst-launch-1.0 -v v4l2src device=/dev/video1 ! video/x-raw, format=YUY2, width=1920,  height=1080, framerate=30/1 ! xvimagesink

Not sure, I have no Nano nor 2 USB cams, no MJPG cam for trying, but there may be some USB problems with MJPG cams requesting all USB bandwith…you may also find some info searching this forum.
Not sure it this changes wrt to above, but you may try using RAW (YUY2) mode of first camera with small resolution and framerate (say 640x480@30fps) as second camera only has 1920x1080@fps.

Legion · December 16, 2020, 2:00pm

Sorry for late reply,

gst-launch-1.0 -v v4l2src device=/dev/video0 ! image/jpeg,width=1280,height=720,framerate=30/1 ! jpegparse ! jpegdec ! xvimagesink sync=false    v4l2src device=/dev/video1! video/x-raw, format=YUY2, width=1920,  height=1080, framerate=30/1 ! xvimagesink

Gives me an error :

(gst-launch-1.0:10939): GStreamer-CRITICAL **: 12:05:06.219: gst_element_make_from_uri: assertion 'gst_uri_is_valid (uri)' failed
WARNING: erroneous pipeline: no element "video"

This one

gst-launch-1.0 -v v4l2src device=/dev/video0 ! image/jpeg,width=1280,height=720,framerate=30/1 ! jpegparse ! jpegdec ! xvimagesink

is working but is laggy, and the last one do nothing and you need to Ctrl+C it to stop - after that camera is somehow occupied becouse if i try to use it again somohow I get info that video1 is unavilable.

I started working with OpenCV because this

gst-launch-1.0 tee name=stream v4l2src device=/dev/video0 ! image/jpeg,width=1280,height=720,framerate=60/1 ! jpegparse ! jpegdec ! xvimagesink sync=false v4l2src device=\"/dev/video1\" ! xvimagesink

is working perfectly, but I can’t apply that to my code :(

Generally I need to modify thase both videos together - i would be perfect to do this on Jetson, but if it’s impossible then I would need to send both videos somehow to my computer, but (as you know, cause you are the only one that tried to help) I have problem with it as well :<
Here my post about this: GStreamer - stream video to IP

Honey_Patouceul · December 16, 2020, 5:23pm

I have to apologize I have made again typos… Edited my last post and added a missing space bteween /dev/video1 and ‘!’ that would make gstreamer unable to run the pipeline.

Legion:

I started working with OpenCV because this
gst-launch-1.0 tee name=stream v4l2src device=/dev/video0 ! image/jpeg,width=1280,height=720,framerate=60/1 ! jpegparse ! jpegdec ! xvimagesink sync=false v4l2src device=\"/dev/video1\" ! xvimagesink
is working perfectly, but I can’t apply that to my code :(

Note that tee name=stream should do nothing here and might be removed. If it doesn’t work equally without, then there is a mystery I’d like to understand.

If the following works fine (just reducing first camera framerate to 30 fps:

gst-launch-1.0 -v v4l2src device=/dev/video0 ! image/jpeg,width=1280,height=720,framerate=30/1 ! jpegparse ! jpegdec ! xvimagesink sync=false     v4l2src device=/dev/video1 ! xvimagesink

then you may try to create two videoCaptures from opencv:

cv::VideoCapture cam0("v4l2src device=/dev/video0 ! image/jpeg,width=1280,height=720,framerate=30/1 ! jpegparse ! jpegdec ! videoconvert ! video/x-raw, format=BGR ! queue ! appsink sync=false", cv::CAP_GSTREAMER);
if (!cam0.isOpened()) {
    std::cerr << "Failed to open cam0" << std::endl;
    return -1;
}

cv::VideoCapture cam1("v4l2src device=/dev/video1 ! videoconvert ! video/x-raw, format=BGR ! queue ! appsink", cv::CAP_GSTREAMER);
if (!cam1.isOpened()) {
    std::cerr << "Failed to open cam1" << std::endl;
    cam0.release();
    return -2;
}

If this works, you may be able to read a frame from both cameras in a loop.

Additional note: in some cases it may help for performance to add option io-mode=2 to v4l2src plugin.

Legion · December 16, 2020, 7:18pm

gst-launch-1.0 -v v4l2src device=/dev/video0 ! image/jpeg,width=1280,height=720,framerate=30/1 ! jpegparse ! jpegdec ! xvimagesink sync=false     v4l2src device=/dev/video1 ! xvimagesink

That’s working fine :)
So I tried your code…
if I open cam0 like that it’s really laggy but if I use CAP_V4L it’s working somehow better -> I tried switching to capture fps to 60 and now it’s the best I achieved, far from 30fps thru (look like 15fps).
I also added a switch() hoping that it will optimize somehow and it’s worked - on both devices I noticed an improvement.

And the most important part! Cam1 (after I added io-mode=2) is working fine just like on my PC, with plain GStreamer command it seemed better but it’s not really a problem

Here’s my code

cv::VideoCapture cap, cap1;

cap1.open("v4l2src device=/dev/video1 ! videoconvert ! video/x-raw, format=BGR, io-mode=2 ! queue ! appsink", cv::CAP_GSTREAMER);
if (!cap1.isOpened()) {
    std::cerr << "Failed to open cam1" << std::endl;
    cap.release();
    return -2;
}

// Here I just call .set() after opening with cv::CAP_V4L in outside method
cameras[0].fourcc = cv::VideoWriter::fourcc('M','J','P','G');
cameras[0].fps = 60;
cameras[0].set(cap);

uint stage = 0;

std::chrono::steady_clock::time_point start = std::chrono::steady_clock::now();
while(true)
{
    switch(cv::waitKey(12))
    {
        case 27:
            cap.release();
            cap1.release();
            cout << "Disconnected\n";
            return 0;
    }

    switch (stage)
    {
        case 0:
            if(cap.isOpened())
            {
                cap >> frame;
                cv::imshow("Video0", frame);
                stage = 1;
            }
        break;
        case 1:
            if(cap1.isOpened())
            {
                cap1 >> frame1;
                cv::imshow("Video1", frame1);
                stage = 0;
            }
        break;
    }
}

Is there any way to further improve cam0 ?

Honey_Patouceul · January 6, 2021, 5:13pm

io-mode is not a gstreamer cap, it is a v4l2src option. Try adding it after /dev/video1, separated by a space.

Using V4L API for a MJPG camera means decoding with CPU only. This may be better than gstreamer jpegdec + videoconvert.
If you want to improve it, you may try to run in RAW (YUY2) mode with a lower resolution and framerate or use HW decoder:

gst-launch-1.0 -v v4l2src device=/dev/video0 io-mode=2 ! image/jpeg,width=1280,height=720,framerate=30/1 ! jpegparse ! nvv4l2decoder mjpeg=1 ! nvvidconv ! xvimagesink sync=false

That would turn into opencv input pipeline:

cv::VideoCapture ("v4l2src device=/dev/video0 io-mode=2 ! image/jpeg,width=1280,height=720,framerate=30/1 ! jpegparse ! nvv4l2decoder mjpeg=1 ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink sync=false", cv::CAP_GSTREAMER);

Or:

gst-launch-1.0 -v v4l2src device=/dev/video0 io-mode=2 ! video/x-raw, format=YUY2, width=640, height=480, framerate=30/1 ! xvimagesink sync=false

That would turn into opencv input pipeline:

cv::VideoCapture ("v4l2src device=/dev/video0 io-mode=2 ! video/x-raw, format=YUY2, width=640, height=480, framerate=30/1 ! videoconvert ! video/x-raw, format=BGR ! appsink sync=false", cv::CAP_GSTREAMER);

Be also sure to boost your Nano with:

sudo nvpmodel -m0
sudo jetson_clocks