MIPI camera + OpenCV

I decided to get Khadas Vim3 to my Jetson Xavier, but I got a problem with using MIPI camera, I would be very glad if you’ll help me, because khadas community doesn’t seems to answer on my topic.

The situation:

I have khadas vim3 and mioi cam from there official store, I plugged in the cam, but when I tried usual command:
cap = cv2.VideoCapture('/dev/video0')
the program just stucked in the infinite loop without showing a picture, then i tried this:


So he doesn’t see anything
And yes, I’m sure that camera plugged in correctly and it’s video0 because this command:
v4l2_test -c 1 -p 0 -F 0 -f 0 -D 0 -R 1 -r 2 -d 2 -N 1000 -n 800 -w 0 -e 1 -I 0 -b /dev/fb0 -v /dev/video0
shows me the camera frame
This is info about my cam:

Type: Video Capture Multiplanar

[0]: 'RGB4' (32-bit A/XRGB 8-8-8-8)
Size: Discrete 1280x720
Size: Discrete 1920x1080
Size: Discrete 3840x2160
[1]: 'RGB3' (24-bit RGB 8-8-8)
Size: Discrete 1280x720
Size: Discrete 1920x1080
Size: Discrete 3840x2160
[2]: 'NV12' (Y/CbCr 4:2:0)
Size: Discrete 1280x720
Size: Discrete 1920x1080
Size: Discrete 3840x2160
[3]: 'Y444' (16-bit A/XYUV 4-4-4-4)
Size: Discrete 1280x720
Size: Discrete 1920x1080
Size: Discrete 3840x2160
[4]: 'YUYV' (YUYV 4:2:2)
Size: Discrete 1280x720
Size: Discrete 1920x1080
Size: Discrete 3840x2160
[5]: 'UYVY' (UYVY 4:2:2)
Size: Discrete 1280x720
Size: Discrete 1920x1080
Size: Discrete 3840x2160
[6]: 'GREY' (8-bit Greyscale)
Size: Discrete 1280x720
Size: Discrete 1920x1080
Size: Discrete 3840x2160
[7]: 'BYR2' (16-bit Bayer BGBG/GRGR (Exp.))
Size: Discrete 1280x720
Size: Discrete 1920x1080
Size: Discrete 3840x2160```

I would be very glad if someone will recommend me what to do

Happy New Year to everyone!

Did you try this?

And this link would be helpful to start:

1 Like

Thanks! I’ll try it later, but are you sure that it’ll help? Because these are instructions for NVIDIA Jetson + raspberry mipi

Both Jetson Nano and Xavier NX devkits use 15-pin MIPI connector, and especially IMX219 based RPi camera module v2.

I opened the repo, I know this guy and already was on Jetson hacks repo and tried this commands, but Khadas returns error when I try nvarguscamerasrc bcs he doesn’t know what is it

I apologize for my misunderstandings.
Pls forget my previous comment.

  1. HW: not a Jetson, so you’d better contact the vendor.

    • if camera device driver is provided
    • how to use v4l2 or gstreamer
  2. MIPI Camera: Both Jetson Nano and Xavier NX use 15-pin RPi v2 camera as I said.
    But VIM3 uses different camera, 30-pin connector.

1 Like

Thanks)
I left my question on khadas community, but nobody is responding, that’s why I decided to try it here(

You refer this link, don’t you?
https://docs.khadas.com/products/sbc/vim3/configurations/os08a10-mipi-camera

  1. guvcview works?
  2. enable & disable
  3. OpenCV packages for Python

Wow
I haven’t seen it before, but I was looking for solution for 3 days…
Thank you!!!
I’ll check it on Khadas as soon as I get home!

G’luck and Happy new year !

1 Like

Hey!
I tried guvcview and camera app and it works well, but when I’m running opencv cammand, he just stucks in infinite loop before while loop, do you know why it can be?

Pls check your OpenCV version. And IF possible, install it again.

Already did it…
I was working on OpenCV 4.6, then I installed OpenCV 4.2, result is the same…

Maybe I should instal cpp OpenCV, how do you think?

I don’t think so. Python library is enough to run the example I think.
Would you check if

  • normal USB webcam does work in this way
  • which version of cv2 is loaded by import cv2

I’d suggest to check what videoio backends are available from the opencv lib running in your python env:

import cv2
print(cv2.getBuildInformation())

and check for Video I/O such as:

  Video I/O:
    DC1394:                      NO
    FFMPEG:                      YES
      avcodec:                   YES (58.54.100)
      avformat:                  YES (58.29.100)
      avutil:                    YES (56.31.100)
      swscale:                   YES (5.5.100)
      avresample:                YES (4.0.0)
    GStreamer:                   YES (1.16.3)
    v4l/v4l2:                    YES (linux/videodev2.h)

For camera capture, you may use V4L:

# Using V4L2 backend:
cap = cv2.VideoCapture(0, cv2.CAP_V4L)
# or
# cap = cv2.VideoCapture('/dev/video0', cv2.CAP_V4L)

Or gstreamer:

# Using v4l2src plugin that outputs into system memory, you may use RGB format from your camera that you would convert into BGR for opencv:
cap = cv2.VideoCapture("v4l2src device=/dev/video0 ! video/x-raw,format=RGB,width=1280,height=720,framerate=30/1 ! videoconvert ! video/x-raw,format=BGR ! appsink drop=1', cv2.CAP_GSTREAMER)

# As your camera is MIPI with UYVY format, you may also try nvv4l2camerasrc outputting into NVMM memory, use HW convert into BGRx and finally CPU converter for BGR format:
cap = cv2.VideoCapture('nvv4l2camerasrc device=/dev/video0 ! video/x-raw(memory:NVMM),formay=UYVY,width=1280,height=720,framerate=30/1 ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink drop=1', cv2.CAP_GSTREAMER)

Also, you may test if the capture did successfully open and check current framing:

if not cap.isOpened():
	print('Failed to open camera');
	exit(-1)
w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
fps = float(cap.get(cv2.CAP_PROP_FPS))
print('camera opened, framing %dx%d@%f fps' % (w,h,fps))

If ok you would be able to run an opencv loop with imshow for displaying result such as:

import cv2

cap = cv2.VideoCapture(0, cv2.CAP_V4L)
if not cap.isOpened():
	print('Failed to open camera');
	exit(-1)
w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
fps = float(cap.get(cv2.CAP_PROP_FPS))
print('camera opened, framing %dx%d@%f fps' % (w,h,fps))

while True:
	ret,frame = cap.read()
	if not ret:
		print('Failed to read from camera')
		cap.release()
		exit(-3)
	cv2.imshow('Test', frame)
	cv2.waitKey(1)
	
cap.release()

Try the available backends for camera capture and when you have one working, try to set up the writer.
Note that the size and framerate of the writer should match the size of the frames you’re pushing into it and the rate you’re pushing these. In your case, you’re pushing 1280x720 frames into a 640x480 writer, that may be wrong.

If your requirement is XVID codec, you may use FFMPEG backend if available:

fourcc = cv2.VideoWriter_fourcc(*'XVID')
out = cv2.VideoWriter("./test.avi", cv2.CAP_FFMPEG, fourcc, fps, (w,h), True)

Or you may use HW encoder for H264 with gstreamer backend:

# BGR frames from opencv are first converted into BGRx with CPU, then resized into 640x480 and converted into NV12 into NVMM memory for HW encoder, and H264 stream is put into AVI container:
out = cv2.VideoWriter("appsrc ! video/x-raw,format=BGR ! queue ! videoconvert ! video/x-raw,format=BGRx ! nvvidconv ! video/x-raw(memory:NVMM),width=640,height=480 ! nvv4l2h264enc ! h264parse ! avimux ! filesink location=test.avi", cv2.CAP_GSTREAMER, 0, fps, (w,h), True)

# Or:
out = cv2.VideoWriter("appsrc ! video/x-raw,format=BGR ! queue ! videoscale ! video/x-raw,width=640,height=480 ! videoconvert ! video/x-raw,format=BGRx ! nvvidconv ! video/x-raw(memory:NVMM),format=NV12 ! nvv4l2h264enc ! h264parse ! avimux ! filesink location=test.avi", cv2.CAP_GSTREAMER, 0, fps, (w,h), True)

Again, test if the writer has successfully been created:

if not out.isOpened():
	print('Failed to open writer')
	cap.release()
	exit(-2)

And add the write in the loop (you may remove imshow/waitKey):

import cv2

cap = cv2.VideoCapture(0, cv2.CAP_V4L)
if not cap.isOpened():
	print('Failed to open camera');
	exit(-1)
w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
fps = float(cap.get(cv2.CAP_PROP_FPS))
print('camera opened, framing %dx%d@%f fps' % (w,h,fps))

# Using FFMPEG backend
fourcc = cv2.VideoWriter_fourcc(*'XVID')
out = cv2.VideoWriter("./test.avi", cv2.CAP_FFMPEG, fourcc, fps, (w,h), True)

# Or using GStreamer backend resizing into 640x480
#out = cv2.VideoWriter("appsrc ! video/x-raw,format=BGR ! queue ! videoconvert ! video/x-raw,format=BGRx ! nvvidconv ! video/x-raw(memory:NVMM),width=640,height=480 ! nvv4l2h264enc ! h264parse ! avimux ! filesink location=test.avi", cv2.CAP_GSTREAMER, 0, fps, (w,h), True)
#out = cv2.VideoWriter("appsrc ! video/x-raw,format=BGR ! queue ! videoscale ! video/x-raw,width=640,height=480 ! videoconvert ! video/x-raw,format=BGRx ! nvvidconv ! video/x-raw(memory:NVMM),format=NV12 ! nvv4l2h264enc ! h264parse ! avimux ! filesink location=test.avi", cv2.CAP_GSTREAMER, 0, fps, (w,h), True)
if not out.isOpened():
	print('Failed to open writer')
	cap.release()
	exit(-2)

while True:
	ret,frame = cap.read()
	if not ret:
		print('Failed to read from camera')
		out.release()
		cap.release()
		exit(-3)
	out.write(frame)
	cv2.imshow('Test', frame)
	cv2.waitKey(1)
	
out.release()
cap.release()

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.