Gstreamer use MJPEG codec

aast226 · August 4, 2020, 11:53pm

Hello,

I am using this camera for a project and am trying to capture full resolution (3264x2448) frames at 15 FPS using the MJPEG codec. I am using OpenCV version 4.1.1 (the default version installed on the Jetson Nano) and am trying to use gstreamer when I initialize my VideoCapture object. The following code works as expected and I am able to capture full resolution images:

cap = cv2.VideoCapture('v4l2src device=/dev/video0 ! video/x-raw, width=(int)800, height=(int)600 ! videoconvert !  video/x-raw, format=(string)BGR ! appsink', cv2.CAP_GSTREAMER)

However, since this video capture object is capturing frames in raw, the max rate at which I can capture frames is 2 FPS. Looking at the “Product Description” section of the page for my camera, I see that the max supported frame rate using YUY2 at 3264x2448 is 2 FPS, and I also see the the max supported frame rate using MJPEG is 15 FPS. To try to achieve this frame rate, I’ve modified my VideoCapture object initialization to look like this:

cap = cv2.VideoCapture('v4l2src device=/dev/video0 ! image/jpeg, width=(int)3264, height=(int)2448 !  nvjpegdec ! video/x-raw ! appsink', cv2.CAP_GSTREAMER)

But, when I run the above line, I get the following error:

[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (1757) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module v4l2src15 reported: Internal data stream error.
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (886) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created

What is the correct way to use OpenCV and gstreamer to achieve my objective?

aast226 · August 4, 2020, 11:57pm

As an experiment, I adapted some of the c++ gstreamer examples from jetson-utils (specifically these files: 1 & 2) to create the following simple python test script:

import cv2
def check_capture_object(cap):
	r, f = cap.read()
	print(f'Frame read was successful: {r}')
	if r:
		print(f'Frame rate: cap.get(cv2.CAP_PROP_FPS)')
		print(f'Image dimensions: {f.shape}')

def create_gst_string(codec_as_str, video_options):
	gst = f'v4l2src device=/dev/video0 ! {codec_as_str}, width=(int)3264, height=(int)2448 ! {video_options}, format=(string)YUY2 ! appsink'
	print(f'Full gst string: {gst}')
	return gst

def test_all_gst_strings():
	all_codec_strs = [
		'video/x-h264',
		'video/x-h265',
		'video/x-vp8',
		'video/x-vp9',
		'video/mpeg, mpegversion=(int)2',
		'video/mpeg, mpegversion=(int)4',
		'image/jpeg',
	]
	all_video_options = [
		'h264parse ! omxh264dec ! video/x-raw',
		'h265parse ! omxh265dec ! video/x-raw',
		'omxvp8dec ! video/x-raw',
		'omxvp9dec ! video/x-raw',
		'mpegvideoparse ! omxmpeg2videodec ! video/x-raw',
		'mpeg4videoparse ! omxmpeg4videodec ! video/x-raw',
		'nvjpegdec ! video/x-raw',
	]
	for codec_str, vid_option in zip(all_codec_strs, all_video_options):
		check_capture_object(cv2.VideoCapture(create_gst_string(codec_str, vid_option), cv2.CAP_GSTREAMER))
		print('-----------')

None of the VideoCapture objects work as expected. See below for the output of the script:

Full gst string: v4l2src device=/dev/video0 ! video/x-h264, width=(int)3264, height=(int)2448 ! h264parse ! omxh264dec ! video/x-raw, format=(string)YUY2 ! appsink
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (1757) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module v4l2src22 reported: Internal data stream error.
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (886) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Frame read was successful: False
-----------
Full gst string: v4l2src device=/dev/video0 ! video/x-h265, width=(int)3264, height=(int)2448 ! h265parse ! omxh265dec ! video/x-raw, format=(string)YUY2 ! appsink
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (711) open OpenCV | GStreamer warning: Error opening bin: could not link v4l2src23 to h265parse2, v4l2src23 can't handle caps video/x-h265, width=(int)3264, height=(int)2448
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Frame read was successful: False
-----------
Full gst string: v4l2src device=/dev/video0 ! video/x-vp8, width=(int)3264, height=(int)2448 ! omxvp8dec ! video/x-raw, format=(string)YUY2 ! appsink
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (1757) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module v4l2src24 reported: Internal data stream error.
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (886) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Frame read was successful: False
-----------
Full gst string: v4l2src device=/dev/video0 ! video/x-vp9, width=(int)3264, height=(int)2448 ! omxvp9dec ! video/x-raw, format=(string)YUY2 ! appsink
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (1757) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module v4l2src25 reported: Internal data stream error.
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (886) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Frame read was successful: False
-----------
Full gst string: v4l2src device=/dev/video0 ! video/mpeg, mpegversion=(int)2, width=(int)3264, height=(int)2448 ! mpegvideoparse ! omxmpeg2videodec ! video/x-raw, format=(string)YUY2 ! appsink
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (1757) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module v4l2src26 reported: Internal data stream error.
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (886) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Frame read was successful: False
-----------
Full gst string: v4l2src device=/dev/video0 ! video/mpeg, mpegversion=(int)4, width=(int)3264, height=(int)2448 ! mpeg4videoparse ! omxmpeg4videodec ! video/x-raw, format=(string)YUY2 ! appsink
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (1757) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module v4l2src27 reported: Internal data stream error.
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (886) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Frame read was successful: False
-----------
Full gst string: v4l2src device=/dev/video0 ! image/jpeg, width=(int)3264, height=(int)2448 ! nvjpegdec ! video/x-raw, format=(string)YUY2 ! appsink
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (711) open OpenCV | GStreamer warning: Error opening bin: could not link nvjpegdec8 to appsink28, nvjpegdec8 can't handle caps video/x-raw, format=(string)YUY2
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Frame read was successful: False
-----------

DaneLLL · August 5, 2020, 2:49am

Hi,
Please check if gst-launch-1.0 command works:

gst-launch-1.0 v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=(int)3264, height=(int)2448 !  nvjpegdec ! video/x-raw ! nvvidconv ! 'video/x-raw(memory:NVMM),width=1920,height=1080' ! nvoverlaysink

aast226 · August 5, 2020, 4:54pm

DaneLLL:

gst-launch-1.0 v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=(int)3264, height=(int)2448 !  nvjpegdec ! video/x-raw ! nvvidconv ! 'video/x-raw(memory:NVMM),width=1920,height=1080' ! nvoverlaysink

Running this command in the command line on my Nano gives the error -bash: syntax error near unexpected token ('`. Am I missing something here?

When I try to initialize a video capture object with that gstreamer config this is how I set it up:

import cv2
cap = cv2.VideoCapture('v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=(int)3264, height=(int)2448 !  nvjpegdec ! video/x-raw ! nvvidconv ! video/x-raw(memory:NVMM),width=1920,height=1080 ! nvoverlaysink', cv2.CAP_GSTREAMER)

Which gives the errors:

[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (801) open OpenCV | GStreamer warning: cannot find appsink in manual pipeline
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created

DaneLLL · August 5, 2020, 11:23pm

Hi,
You may try

gst-launch-1.0 v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=(int)3264, height=(int)2448 !  nvjpegdec ! video/x-raw ! videoconvert ! video/x-raw,format=BGR ! fakesink

If above command works, you can then try

cap = cv2.VideoCapture('v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=(int)3264, height=(int)2448 !  nvjpegdec ! video/x-raw ! videoconvert ! video/x-raw,format=BGR ! appsink', cv2.CAP_GSTREAMER)

aast226 · August 6, 2020, 7:43pm

@DaneLLL Thanks for that, the commands worked! It looks like io-mode=2 is what made the difference here. What does io-mode=2 actually do?

DaneLLL · August 6, 2020, 11:13pm

Hi,
You can get information of all properties by executing $ gst-inspect-1.0 v4l2src:

  io-mode             : I/O mode
                        flags: readable, writable
                        Enum "GstV4l2IOMode" Default: 0, "auto"
                           (0): auto             - GST_V4L2_IO_AUTO
                           (1): rw               - GST_V4L2_IO_RW
                           (2): mmap             - GST_V4L2_IO_MMAP
                           (3): userptr          - GST_V4L2_IO_USERPTR
                           (4): dmabuf           - GST_V4L2_IO_DMABUF
                           (5): dmabuf-import    - GST_V4L2_IO_DMABUF_IMPORT

aast226 · August 7, 2020, 8:15pm

Hi @DaneLLL, After testing this further, I’m seeing that the images being returned are always empty (all pixels have value of 0). Do you know why this might be?

DaneLLL · August 12, 2020, 1:01am

Hi,
Please check if you can run the pipeline and see preview:

$ export DISPLAY=:0 (or DISPLAY=:1)
$ gst-launch-1.0 v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=(int)3264, height=(int)2448 !  nvjpegdec ! video/x-raw ! xvimagesink

003021lqy · August 13, 2020, 7:23am

DaneLLL:

Hi,
Please check if you can run the pipeline and see preview:

$ export DISPLAY=:0 (or DISPLAY=:1)
$ gst-launch-1.0 v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=(int)3264, height=(int)2448 !  nvjpegdec ! video/x-raw ! xvimagesink

If this works fine in command line, how to deploy it in Opencv code? Any help?

DaneLLL · August 13, 2020, 8:11am

Hi,
We have verified the steps with Logitech C615/C930 and E-Con See3CAM CU135. You may try other cameras and check if the issue is specific to the camera.

003021lqy · August 14, 2020, 3:27am

Hi I tried

cap = cv2.VideoCapture(‘v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=(int)1920, height=(int)1080 ! nvjpegdec ! video/x-raw ! videoconvert ! video/x-raw,format=BGR ! appsink’, cv2.CAP_GSTREAMER)

and the fps is only around 20 fps.

While, if I tried

cap = cv2.VideoCapture(‘v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=(int)1920, height=(int)1080 ! nvjpegdec ! video/x-raw ! nvvidconv ! video/x-raw,format=I420 ! appsink’, cv2.CAP_GSTREAMER)

It could reach 30 fps, because it utilize the GPU acceleration.
However, if I need to run on opencv, I need to use BGR format, and if convert I420 to BGR, it costs CPU. So the fps drop down to 20fps.

Is there any methods that directly use GPU to convert video to BGR format?
I have no idea. If I used both nvvidconv and video/x-raw,format=I420, it does not work and gives me an error. It shows:
could not link nvvconv0 to appsink0, nvvconv0 can’t handle caps video/x-raw, format=BGR.

DaneLLL · August 14, 2020, 5:06am

Hi,
This is limitation of hardware converter. There is a similar discussion in

So for gstreamer+ OpenCV, you need to use software converter for converting to BGR. Please execute sudo jetson_clocks to run CPUs at max lcocks.

Planktos · August 14, 2020, 6:44am

Hello there,
to decode my ELP MJPEG camera to RGB I used the following pipeline and it works perfectly but It is not exactly what you are trying to do, I hope it can help you :

"input_images": {
   "pipeline": {
    "pipeline": "gst-launch-1.0 -v v4l2src device=/dev/video0  ! avdec_mjpeg ! videoconvert ! video/x-raw,format=RGB,height=960,wight=2560,framerate=30/1 ! appsink name = acquired_image "
   }
},

Honey_Patouceul · August 14, 2020, 6:58pm

You may try (assuming your cam has 1080p@30fps mode):

cap = cv2.VideoCapture(‘v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=(int)1920, height=(int)1080, framerate=30/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink’, cv2.CAP_GSTREAMER)

so MJPEG to BGRx is performed by HW decoder, and videoconvert just removes the 4th byte for each pixel.

Be aware that opencvio may be the bottleneck. If you’ve installed jetson-utils, it may give a better framerate.
Last time I’ve looked to it, I think it was using nvjpegdec for mjpeg sources, but this may have changed since then.
If nvv4l2decoder better works for your case, you may also try to modify this jetson-utils case…but I haven’t tested, though.

dakurtz422 · January 9, 2021, 2:00am

Hello, I tried your Pipeline and it works nice, but when I’m trying to write the frame, cap.read() slows down – 5 sec video is recording like 40 sec and its like 2 fps and output video contains like 5 frames. What is doing it please?

I manually lower fps on writer to 5fps, and finaly its moving normal speed, but its skiping frames, I really dont get what is happening…

Topic		Replies	Views
Multiple stream from one camera via gstreamer Jetson Nano camera , gstreamer	11	6364	October 15, 2021
CSI-Camera Raspberry Pi v2 not work on Jetson Nano Jetson Nano	11	13603	October 14, 2021
Opencv Camera Properties : CAP_PROP_FOURCC Not working Jetson Nano opencv	28	4021	October 15, 2021
Can I capture still image during streaming video in gstreamer? Jetson Nano gstreamer	22	10716	October 18, 2021
openCV video writer to gstreamer appsrc Jetson Nano jetpack	22	14188	October 15, 2021
How does the camera control work ? Jetson TX2	13	2747	October 18, 2021
Reading mp4 file via gstreamer in opencv Jetson Nano opencv	20	14449	September 1, 2021
GStreamer pipeline for accelerated streaming of USB camera Jetson Nano gstreamer	8	9028	December 5, 2022
Optimizing opencv gstreamer with mipi and usb uvc cameras Jetson Nano opencv , gstreamer	24	11464	October 18, 2021
Unable to use the correct GStreamer pipeline for e-CAM130_CUXVR with Jetson AGX Xavier in OpenCV Python Jetson AGX Xavier	6	2547	October 18, 2021

Gstreamer use MJPEG codec

Related topics