Gstreamer use MJPEG codec

Autonomous Machines Jetson & Embedded Systems Jetson Nano
,
1

Hello,

I am using this camera for a project and am trying to capture full resolution (3264x2448) frames at 15 FPS using the MJPEG codec. I am using OpenCV version 4.1.1 (the default version installed on the Jetson Nano) and am trying to use gstreamer when I initialize my VideoCapture object. The following code works as expected and I am able to capture full resolution images:

cap = cv2.VideoCapture('v4l2src device=/dev/video0 ! video/x-raw, width=(int)800, height=(int)600 ! videoconvert !  video/x-raw, format=(string)BGR ! appsink', cv2.CAP_GSTREAMER)

However, since this video capture object is capturing frames in raw, the max rate at which I can capture frames is 2 FPS. Looking at the ā€œProduct Descriptionā€ section of the page for my camera, I see that the max supported frame rate using YUY2 at 3264x2448 is 2 FPS, and I also see the the max supported frame rate using MJPEG is 15 FPS. To try to achieve this frame rate, Iā€™ve modified my VideoCapture object initialization to look like this:

cap = cv2.VideoCapture('v4l2src device=/dev/video0 ! image/jpeg, width=(int)3264, height=(int)2448 !  nvjpegdec ! video/x-raw ! appsink', cv2.CAP_GSTREAMER)

But, when I run the above line, I get the following error:

[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (1757) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module v4l2src15 reported: Internal data stream error.
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (886) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created

What is the correct way to use OpenCV and gstreamer to achieve my objective?

2

As an experiment, I adapted some of the c++ gstreamer examples from jetson-utils (specifically these files: 1 & 2) to create the following simple python test script:

import cv2
def check_capture_object(cap):
	r, f = cap.read()
	print(f'Frame read was successful: {r}')
	if r:
		print(f'Frame rate: cap.get(cv2.CAP_PROP_FPS)')
		print(f'Image dimensions: {f.shape}')

def create_gst_string(codec_as_str, video_options):
	gst = f'v4l2src device=/dev/video0 ! {codec_as_str}, width=(int)3264, height=(int)2448 ! {video_options}, format=(string)YUY2 ! appsink'
	print(f'Full gst string: {gst}')
	return gst

def test_all_gst_strings():
	all_codec_strs = [
		'video/x-h264',
		'video/x-h265',
		'video/x-vp8',
		'video/x-vp9',
		'video/mpeg, mpegversion=(int)2',
		'video/mpeg, mpegversion=(int)4',
		'image/jpeg',
	]
	all_video_options = [
		'h264parse ! omxh264dec ! video/x-raw',
		'h265parse ! omxh265dec ! video/x-raw',
		'omxvp8dec ! video/x-raw',
		'omxvp9dec ! video/x-raw',
		'mpegvideoparse ! omxmpeg2videodec ! video/x-raw',
		'mpeg4videoparse ! omxmpeg4videodec ! video/x-raw',
		'nvjpegdec ! video/x-raw',
	]
	for codec_str, vid_option in zip(all_codec_strs, all_video_options):
		check_capture_object(cv2.VideoCapture(create_gst_string(codec_str, vid_option), cv2.CAP_GSTREAMER))
		print('-----------')

None of the VideoCapture objects work as expected. See below for the output of the script:

Full gst string: v4l2src device=/dev/video0 ! video/x-h264, width=(int)3264, height=(int)2448 ! h264parse ! omxh264dec ! video/x-raw, format=(string)YUY2 ! appsink
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (1757) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module v4l2src22 reported: Internal data stream error.
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (886) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Frame read was successful: False
-----------
Full gst string: v4l2src device=/dev/video0 ! video/x-h265, width=(int)3264, height=(int)2448 ! h265parse ! omxh265dec ! video/x-raw, format=(string)YUY2 ! appsink
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (711) open OpenCV | GStreamer warning: Error opening bin: could not link v4l2src23 to h265parse2, v4l2src23 can't handle caps video/x-h265, width=(int)3264, height=(int)2448
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Frame read was successful: False
-----------
Full gst string: v4l2src device=/dev/video0 ! video/x-vp8, width=(int)3264, height=(int)2448 ! omxvp8dec ! video/x-raw, format=(string)YUY2 ! appsink
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (1757) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module v4l2src24 reported: Internal data stream error.
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (886) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Frame read was successful: False
-----------
Full gst string: v4l2src device=/dev/video0 ! video/x-vp9, width=(int)3264, height=(int)2448 ! omxvp9dec ! video/x-raw, format=(string)YUY2 ! appsink
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (1757) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module v4l2src25 reported: Internal data stream error.
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (886) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Frame read was successful: False
-----------
Full gst string: v4l2src device=/dev/video0 ! video/mpeg, mpegversion=(int)2, width=(int)3264, height=(int)2448 ! mpegvideoparse ! omxmpeg2videodec ! video/x-raw, format=(string)YUY2 ! appsink
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (1757) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module v4l2src26 reported: Internal data stream error.
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (886) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Frame read was successful: False
-----------
Full gst string: v4l2src device=/dev/video0 ! video/mpeg, mpegversion=(int)4, width=(int)3264, height=(int)2448 ! mpeg4videoparse ! omxmpeg4videodec ! video/x-raw, format=(string)YUY2 ! appsink
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (1757) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module v4l2src27 reported: Internal data stream error.
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (886) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Frame read was successful: False
-----------
Full gst string: v4l2src device=/dev/video0 ! image/jpeg, width=(int)3264, height=(int)2448 ! nvjpegdec ! video/x-raw, format=(string)YUY2 ! appsink
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (711) open OpenCV | GStreamer warning: Error opening bin: could not link nvjpegdec8 to appsink28, nvjpegdec8 can't handle caps video/x-raw, format=(string)YUY2
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Frame read was successful: False
-----------
4

Hi,
Please check if gst-launch-1.0 command works:

gst-launch-1.0 v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=(int)3264, height=(int)2448 !  nvjpegdec ! video/x-raw ! nvvidconv ! 'video/x-raw(memory:NVMM),width=1920,height=1080' ! nvoverlaysink
5

Running this command in the command line on my Nano gives the error -bash: syntax error near unexpected token ('`. Am I missing something here?

When I try to initialize a video capture object with that gstreamer config this is how I set it up:

import cv2
cap = cv2.VideoCapture('v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=(int)3264, height=(int)2448 !  nvjpegdec ! video/x-raw ! nvvidconv ! video/x-raw(memory:NVMM),width=1920,height=1080 ! nvoverlaysink', cv2.CAP_GSTREAMER)

Which gives the errors:

[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (801) open OpenCV | GStreamer warning: cannot find appsink in manual pipeline
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
6

Hi,
You may try

gst-launch-1.0 v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=(int)3264, height=(int)2448 !  nvjpegdec ! video/x-raw ! videoconvert ! video/x-raw,format=BGR ! fakesink

If above command works, you can then try

cap = cv2.VideoCapture('v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=(int)3264, height=(int)2448 !  nvjpegdec ! video/x-raw ! videoconvert ! video/x-raw,format=BGR ! appsink', cv2.CAP_GSTREAMER)
7

@DaneLLL Thanks for that, the commands worked! It looks like io-mode=2 is what made the difference here. What does io-mode=2 actually do?

8

Hi,
You can get information of all properties by executing $ gst-inspect-1.0 v4l2src:

  io-mode             : I/O mode
                        flags: readable, writable
                        Enum "GstV4l2IOMode" Default: 0, "auto"
                           (0): auto             - GST_V4L2_IO_AUTO
                           (1): rw               - GST_V4L2_IO_RW
                           (2): mmap             - GST_V4L2_IO_MMAP
                           (3): userptr          - GST_V4L2_IO_USERPTR
                           (4): dmabuf           - GST_V4L2_IO_DMABUF
                           (5): dmabuf-import    - GST_V4L2_IO_DMABUF_IMPORT
1 Like
9

Hi @DaneLLL, After testing this further, Iā€™m seeing that the images being returned are always empty (all pixels have value of 0). Do you know why this might be?

10

Hi,
Please check if you can run the pipeline and see preview:

$ export DISPLAY=:0 (or DISPLAY=:1)
$ gst-launch-1.0 v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=(int)3264, height=(int)2448 !  nvjpegdec ! video/x-raw ! xvimagesink
12

If this works fine in command line, how to deploy it in Opencv code? Any help?

13

Hi,
We have verified the steps with Logitech C615/C930 and E-Con See3CAM CU135. You may try other cameras and check if the issue is specific to the camera.

14

Hi I tried

cap = cv2.VideoCapture(ā€˜v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=(int)1920, height=(int)1080 ! nvjpegdec ! video/x-raw ! videoconvert ! video/x-raw,format=BGR ! appsinkā€™, cv2.CAP_GSTREAMER)

and the fps is only around 20 fps.

While, if I tried

cap = cv2.VideoCapture(ā€˜v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=(int)1920, height=(int)1080 ! nvjpegdec ! video/x-raw ! nvvidconv ! video/x-raw,format=I420 ! appsinkā€™, cv2.CAP_GSTREAMER)

It could reach 30 fps, because it utilize the GPU acceleration.
However, if I need to run on opencv, I need to use BGR format, and if convert I420 to BGR, it costs CPU. So the fps drop down to 20fps.

Is there any methods that directly use GPU to convert video to BGR format?
I have no idea. If I used both nvvidconv and video/x-raw,format=I420, it does not work and gives me an error. It shows:
could not link nvvconv0 to appsink0, nvvconv0 canā€™t handle caps video/x-raw, format=BGR.

15

Hi,
This is limitation of hardware converter. There is a similar discussion in

So for gstreamer+ OpenCV, you need to use software converter for converting to BGR. Please execute sudo jetson_clocks to run CPUs at max lcocks.

16

Hello there,
to decode my ELP MJPEG camera to RGB I used the following pipeline and it works perfectly but It is not exactly what you are trying to do, I hope it can help you :

"input_images": {
   "pipeline": {
    "pipeline": "gst-launch-1.0 -v v4l2src device=/dev/video0  ! avdec_mjpeg ! videoconvert ! video/x-raw,format=RGB,height=960,wight=2560,framerate=30/1 ! appsink name = acquired_image "
   }
},
17

You may try (assuming your cam has 1080p@30fps mode):

cap = cv2.VideoCapture(ā€˜v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=(int)1920, height=(int)1080, framerate=30/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsinkā€™, cv2.CAP_GSTREAMER)

so MJPEG to BGRx is performed by HW decoder, and videoconvert just removes the 4th byte for each pixel.

Be aware that opencvio may be the bottleneck. If youā€™ve installed jetson-utils, it may give a better framerate.
Last time Iā€™ve looked to it, I think it was using nvjpegdec for mjpeg sources, but this may have changed since then.
If nvv4l2decoder better works for your case, you may also try to modify this jetson-utils caseā€¦but I havenā€™t tested, though.

2 Likes
19

Hello, I tried your Pipeline and it works nice, but when Iā€™m trying to write the frame, cap.read() slows down ā€“ 5 sec video is recording like 40 sec and its like 2 fps and output video contains like 5 frames. What is doing it please?

I manually lower fps on writer to 5fps, and finaly its moving normal speed, but its skiping frames, I really dont get what is happeningā€¦