MJPG Camera Pipeline is Super laggy although it reports it's opening at 30fps

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
,
#1

Hey guys, I’ve been scowering the forums for a few days trying to figure this out but to no avail. I’m wondering if someone can help me with a pipeline for OpenCV. The webcam will open with this script, and it reports a size of 1280x720@30FPS but when i wave my hand in front of it it is clearly not 30 FPS, more like 5 or 10.

This is the output of v4l2-ctl --list-formats-ext

ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'MJPG' (compressed)
	Name        : Motion-JPEG
		Size: Discrete 320x240
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.040s (25.000 fps)
		Size: Discrete 480x272
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.040s (25.000 fps)
		Size: Discrete 640x360
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.040s (25.000 fps)
		Size: Discrete 640x480
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.040s (25.000 fps)
		Size: Discrete 800x600
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.040s (25.000 fps)
		Size: Discrete 1024x576
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.040s (25.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.040s (25.000 fps)
		Size: Discrete 1440x1080
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.040s (25.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.040s (25.000 fps)

	Index       : 1
	Type        : Video Capture
	Pixel Format: 'YUYV'
	Name        : YUYV 4:2:2
		Size: Discrete 320x240
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 480x272
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 640x360
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 640x480
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 800x600
			Interval: Discrete 0.067s (15.000 fps)
		Size: Discrete 1024x576
			Interval: Discrete 0.100s (10.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.100s (10.000 fps)

and this is the pipeline:

cap = cv2.VideoCapture('v4l2src device=/dev/video0 io-mode=2 ! image/jpeg,framerate=30/1,width=1280,height=720,format=MJPG ! nvjpegdec ! videoconvert ! appsink', cv2.CAP_GSTREAMER')

I ran this in gst-launch-1.0 (I can’t remember the pipeline at the moment but will be looking for it) and it seems to have no issue when operating at 30FPS.

This is the basic python script I am using to open the camera to test the pipeline before I put it into my actual code.

import cv2
cap = cv2.VideoCapture('v4l2src device=/dev/video0 io-mode=2 ! image/jpeg,framerate=30/1,width=1280,height=720,format=MJPG ! nvjpegdec ! videoconvert ! appsink', cv2.CAP_GSTREAMER)
width= int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
height= int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
fps = cap.get(cv2.CAP_PROP_FPS)

print('Src opened, %dx%d @ %d fps' % (width,height,fps))

#writer= cv2.VideoWriter('video.mp4', cv2.VideoWriter_fourcc(*'DIVX'), 20, (width,height))


while True:
    time.sleep(0.5)
    ret,frame= cap.read()

 #   writer.write(frame)

    cv2.imshow('frame', frame)

    if cv2.waitKey(1) & 0xFF == 27:
        break


cap.release()
#writer.release()
cv2.destroyAllWindows()

If anyone could please help me refine this pipeline I would be forever grateful.

#2

You may try option 2 from this post.

#3

Thank you for your reply. I tried both pipelines from option 2 in that post and the first one gave me some errors, while the second would in fact open, and report that it was running at 30fps although the stream was clearly operating at around 2-5 fps.

Pipeline #1 tried from that post (had to add appsink instead of xvimagesink as it complained appsink was missing )

cap = cv2.VideoCapture('v4l2src device=/dev/video0 ! image/jpeg, format=MJPG, width=640, height=480, framerate=30/1 ! nvjpegdec ! \'video/x-raw(memory:NVMM), format=I420\' ! nvvidconv ! appsink')

with an error output of:

(python3:15189): GStreamer-CRITICAL **: 12:00:35.932: gst_element_make_from_uri: assertion 'gst_uri_is_valid (uri)' failed

(python3:15189): GStreamer-CRITICAL **: 12:00:35.936: gst_element_link_pads_filtered: assertion 'GST_IS_BIN (parent)' failed
[ WARN:0] global /home/chase/opencv/modules/videoio/src/cap_gstreamer.cpp (854) open OpenCV | GStreamer warning: Error opening bin: syntax error
[ WARN:0] global /home/chase/opencv/modules/videoio/src/cap_gstreamer.cpp (597) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Src opened, 0x0 @ 0 fps
Traceback (most recent call last):
  File "recordVideo.py", line 23, in <module>
    cv2.imshow('frame', frame)
cv2.error: OpenCV(4.5.2) /home/chase/opencv/modules/highgui/src/window.cpp:404: error: (-215:Assertion failed) size.width>0 && size.height>0 in function 'imshow'

and Pipeline 2 (same thing about appsink vs xvimagesink)

cap = cv2.VideoCapture('v4l2src device=/dev/video0 ! image/jpeg, format=MJPG, width=640, height=480, framerate=30/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! appsink')

with output of this, reporting 30FPS but visually 2-10FPS ( hard to tell)

Opening in BLOCKING MODE
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 277 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 277 
[ WARN:0] global /home/chase/opencv/modules/videoio/src/cap_gstreamer.cpp (1081) open OpenCV | GStreamer warning: Cannot query video position: status=1, value=7, duration=-1
Src opened, 640x480 @ 30 fps

Again, thank you for your reply!

EDIT:

I also took out the

\'

in the first pipeline, which was likely causing the syntax error, and it managed to open the stream, but behaves the same as pipeline 2 in that it reports 30FPS but is actually closer to that 2 - 10

#5

Hi,
Please execute sudo nvpmodel -m 0 and sudo jetson_clocks, and try the two pipelines:
Set camera decoder in OpenCV on Jetson Nano - #6 by DaneLLL
Set camera decoder in OpenCV on Jetson Nano - #8 by DaneLLL

Please modify width,height,format,framerate per your camera.

#6

Your code is sleeping half a second between each capture.
That is very likely why your capture rate is 2 FPS.

2 Likes
#8

Thank you for the reply Snarky, I noticed that shortly after the above posters commented. Unfortunately the pipelines posted above did not work even with the sleep removed from the code. I will post the working pipeline and edit this comment when I am able to provide the pipeline that did end up working!

#9