V4l2 camera with gstreamer: lost frames detected

jacob.podhraski · August 26, 2024, 8:02pm

I am having trouble with my USB Camera capturing without losing frames.

GST_DEBUG=3 gst-launch-1.0 -e v4l2src ! 'video/x-raw,width=1920,height=1080,framerate=30/1' ! videoconvert ! x264enc tune=zerolatency ! mp4mux ! filesink location=test.mp4
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Redistribute latency...
0:00:01.803134525 1500166 0xaaab0937d2a0 FIXME               basesink gstbasesink.c:3246:gst_base_sink_default_event:<filesink0> stream-start event without group-id. Consider implementing group-id handling in the upstream elements
0:00:01.803412094 1500166 0xaaab0937d2a0 WARN                   qtmux gstqtmux.c:2981:gst_qt_mux_start_file:<mp4mux0> Robust muxing requires reserved-moov-update-period to be set
0:00:01.917525558 1500166 0xaaab0937d2a0 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:01.315679075
0:00:02.593731466 1500166 0xaaab0937d2a0 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 4 - ts: 0:00:02.173136435
0:00:03.244768631 1500166 0xaaab0937d2a0 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.824055147
0:00:03.525756389 1500166 0xaaab0937d2a0 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:02.988337139
0:00:03.609405490 1500166 0xaaab0937d2a0 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:03.086927651
0:00:03.712037085 1500166 0xaaab0937d2a0 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:03.185500523
0:00:03.810068554 1500166 0xaaab0937d2a0 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.251249363
0:00:03.906482540 1500166 0xaaab0937d2a0 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 3 - ts: 0:00:03.382660363
0:00:03.992116646 1500166 0xaaab0937d2a0 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:03.481276555
0:00:04.097968710 1500166 0xaaab0937d2a0 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.547006683
0:00:04.257226443 1500166 0xaaab0937d2a0 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:03.645570715
0:00:04.384957789 1500166 0xaaab0937d2a0 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 3 - ts: 0:00:03.777002747
0:00:04.525985768 1500166 0xaaab0937d2a0 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 3 - ts: 0:00:03.908435331
0:00:04.630990147 1500166 0xaaab0937d2a0 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 4 - ts: 0:00:04.072735651
0:00:04.740711518 1500166 0xaaab0937d2a0 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 2 - ts: 0:00:04.171311947
0:00:04.832555201 1500166 0xaaab0937d2a0 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 3 - ts: 0:00:04.302706139

The resulting video ends up being extremely choppy. I have already tried decreasing the resolution to 720p at 50FPS, but no luck there. I see the same output in debug mode.

Here are the device caps:

Device caps -> video/x-raw, format=(string)UYVY, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 60/1, 30/1 }; video/x-raw, format=(string)UYVY, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 80/1, 50/1 }; video/x-raw, format=(string)UYVY, width=(int)640, height=(int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)120/1; image/jpeg, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)100/1; image/jpeg, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)100/1; image/jpeg, width=(int)640, height=(int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)120/1

Using a Jetson Xavier with Ubuntu 20.
Gstreamer version: 1.16.3

I tried out a CSI camera to see if that made a difference and surely enough, there was no frame loss.

Still, I need to get the USB camera working for this project.

JerryChang · August 27, 2024, 3:19am

hello jacob.podhraski,

please check your USB camera capability with… $ v4l2-ctl -d /dev/video0 --list-formats-ext
you may also running with preview disabled to examine the output frame-rate.
for instance,
$ gst-launch-1.0 v4l2src device=/dev/video1 num-buffers=900 ! 'video/x-raw, width=1920, height=1080, framerate=30/1, format=YUY2' ! fpsdisplaysink text-overlay=0 name=sink_0 video-sink=fakesink sync=0 -v

jacob.podhraski · August 27, 2024, 4:55pm

Sure, here is my usb information:

ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture

	[0]: 'UYVY' (UYVY 4:2:2)
		Size: Discrete 1920x1080
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 640x480
			Interval: Discrete 0.008s (120.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.013s (80.000 fps)
			Interval: Discrete 0.020s (50.000 fps)
	[1]: 'MJPG' (Motion-JPEG, compressed)
		Size: Discrete 1920x1080
			Interval: Discrete 0.010s (100.000 fps)
		Size: Discrete 640x480
			Interval: Discrete 0.008s (120.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.010s (100.000 fps)

And here is the output with the average frame rate being output to the console:

GST_DEBUG=3 gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=900 ! 'video/x-raw, width=1920, height=1080, framerate=30/1' ! fpsdisplaysink text-overlay=0 name=sink_0 video-sink=fakesink sync=0 -v
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0/GstFakeSink:fakesink0: sync = false
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)UYVY, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)2:4:5:1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)UYVY, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)2:4:5:1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)UYVY, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)2:4:5:1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0/GstFakeSink:fakesink0.GstPad:sink: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)UYVY, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)2:4:5:1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0.GstGhostPad:sink: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)UYVY, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)2:4:5:1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, format=(string)UYVY, pixel-aspect-ratio=(fraction)1/1, colorimetry=(string)2:4:5:1, interlace-mode=(string)progressive
0:00:00.191439817 24815 0xaaaacf339580 WARN          v4l2bufferpool gstv4l2bufferpool.c:809:gst_v4l2_buffer_pool_start:<v4l2src0:pool:src> Uncertain or not enough buffers, enabling copy threshold
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0/GstFakeSink:fakesink0: sync = false
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 17, dropped: 0, current: 32.35, average: 32.35
0:00:02.334946363 24815 0xaaaacf339580 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.202420132
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 24, dropped: 0, current: 7.32, average: 16.19
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 40, dropped: 0, current: 30.45, average: 19.92
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 56, dropped: 0, current: 30.43, average: 22.10
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 72, dropped: 0, current: 30.43, average: 23.53
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 88, dropped: 0, current: 30.44, average: 24.55
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 104, dropped: 0, current: 30.43, average: 25.30
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 120, dropped: 0, current: 30.43, average: 25.88
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 136, dropped: 0, current: 30.43, average: 26.34
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 152, dropped: 0, current: 30.44, average: 26.72
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 168, dropped: 0, current: 30.43, average: 27.04
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 184, dropped: 0, current: 30.43, average: 27.30
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 200, dropped: 0, current: 30.43, average: 27.53
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 216, dropped: 0, current: 30.43, average: 27.72
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 232, dropped: 0, current: 30.43, average: 27.90
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 248, dropped: 0, current: 30.43, average: 28.05
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 264, dropped: 0, current: 30.44, average: 28.18
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 280, dropped: 0, current: 30.43, average: 28.30
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 296, dropped: 0, current: 30.43, average: 28.41
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 312, dropped: 0, current: 30.44, average: 28.50
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 328, dropped: 0, current: 30.43, average: 28.59
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 344, dropped: 0, current: 30.43, average: 28.67
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 360, dropped: 0, current: 30.43, average: 28.75
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 376, dropped: 0, current: 30.43, average: 28.82
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 392, dropped: 0, current: 30.43, average: 28.88
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 408, dropped: 0, current: 30.44, average: 28.94
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 424, dropped: 0, current: 30.43, average: 28.99
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 440, dropped: 0, current: 30.43, average: 29.04
0:00:16.996709004 24815 0xaaaacf339580 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:16.864127068

It seems like it is dropping less frames. But as soon as I add the video encoder: x264enc it returns back to losing frames.

enrique.ramirez · August 27, 2024, 10:36pm

Hi @jacob.podhraski

Can you try using the hardware encoder?

gst-launch-1.0 -e v4l2src device=/dev/video0 ! 'video/x-raw,width=1920,height=1080,framerate=30/1,forma=YUY2' ! nvvidconv ! nvv4l2h264enc ! h264parse ! qtmux ! filesink location=test.mp4

Regards,

Enrique Ramirez
Embedded SW Engineer at RidgeRun
Contact us: support@ridgerun.com
Developers wiki: https://developer.ridgerun.com
Website: www.ridgerun.com

jacob.podhraski · August 28, 2024, 6:17pm

That seemed to add an improvement! In addition, I was wondering if there was a similar hardware encoder for saving jpeg images to a directory? I am currently losing a bunch of frames using this pipeline:

GST_DEBUG=3 gst-launch-1.0 v4l2src device=/dev/video0 ! nvvidconv ! jpegenc ! multifilesink location="/tmp/frames/frame_%05d.jpg"

Is there anything I can use to accelerate this?

enrique.ramirez · August 28, 2024, 7:25pm

Yes, Jetson has a GStreamer element to use the hardware JPEG encoder:

Regards,
-Enrique

jacob.podhraski · August 28, 2024, 8:48pm

It is performing a little better but, I am still losing frames with this method:

GST_DEBUG=3 gst-launch-1.0 v4l2src device=/dev/video0 ! nvvidconv ! nvjpegenc ! multifilesink location="/tmp/frames/frame_%05d.jpg"
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
0:00:00.250864827 66834 0xaaaabcf9d0c0 WARN          v4l2bufferpool gstv4l2bufferpool.c:809:gst_v4l2_buffer_pool_start:<v4l2src0:pool:src> Uncertain or not enough buffers, enabling copy threshold
NvMMLiteBlockCreate : Block : BlockType = 1
0:00:01.824585299 66834 0xaaaabcf9d0c0 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:01.637763056
0:00:02.390826876 66834 0xaaaabcf9d0c0 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.204007960
0:00:03.166788248 66834 0xaaaabcf9d0c0 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:02.979999184
0:00:03.738827556 66834 0xaaaabcf9d0c0 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:03.552020928
0:00:05.303423112 66834 0xaaaabcf9d0c0 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:05.116632768
0:00:06.298782818 66834 0xaaaabcf9d0c0 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:06.111976864
0:00:07.600861836 66834 0xaaaabcf9d0c0 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:07.414080960
0:00:08.233276782 66834 0xaaaabcf9d0c0 WARN                 v4l2src gstv4l2src.c:978:gst_v4l2src_create:<v4l2src0> lost frames detected: count = 1 - ts: 0:00:08.046447056

I still want to get to the bottom of this. Is there anything else in my pipeline that can be optimized for better performance? Or at this point is there anything else I need to consider?

enrique.ramirez · August 30, 2024, 3:56pm

According to this post, the JPEG encoder seems to be used for a single frame capture and not for a MJPEG stream:
https://forums.developer.nvidia.com/t/tx2-jpeg-compression-at-high-framerate-and-configuration/54342

system · October 9, 2024, 3:16am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Gstreamer Jetson Nano gstreamer	19	1526	October 15, 2021
What is limiting the camera framerate (gstreamer) Jetson Nano camera , gstreamer	11	3029	December 22, 2021
Camera's frame rate unstable Jetson AGX Xavier camera	22	4616	October 18, 2021
Problem using gstreamer with opencv and cuda on TX2 with Jetpack 4.3 Jetson TX2 rtsp , opencv , cuda , gstreamer	21	3333	October 18, 2021
Gstreamer issue in NX: Nano is faster than NX! Jetson Xavier NX gstreamer	16	1986	October 18, 2021
Gstreamer fpsdisplaysink reports double framerate Jetson Nano camera , gstreamer	8	3043	December 22, 2021
Failing to get input from CSI camera (IMX219-83) Jetson Nano camera , ubuntu , gstreamer	29	1485	September 25, 2023
Simple Gstream USB Camera Live View Jetson AGX Orin gstreamer	12	2967	July 25, 2023
Help with gstreamer speed improvement via using userptr buffer allocations Jetson TX2	10	4539	May 17, 2018
Gstreamer reports Raspberry Pi camera streaming at 120fps when in reality it is only 60fps Jetson Nano camera , gstreamer	53	3823	July 19, 2022

V4l2 camera with gstreamer: lost frames detected

Related topics