Greyscale only images when NVIDIA accelerated gstreamer pipeline used in OpenCV cv2.VideoCapture()

I am able to use the simple, non-accelerated gstreamer pipeline in Python 3.6 OpenCV cv.VideoCapture().

cv2.VideoCapture( “v4l2src device=/dev/video0 ! video/x-raw, width=640, height=360 ! videoconvert ! appsink”)

Unfortunately, this simplistic pipeline does not use any NVIDIA hardware decoders or accelerations, so it uses too much CPU.
Furthermore, my cameras support faster MJPEG framerates than YUY2 raw, and it would be nice to use NVJPG hardware accelerated decoding.

The following pipelines work when launched from script, but using them inside cv2.VideoCapture() either does not work at all, or shows only greyscale image.

  1. NVJPG decoding (I can see NVJPG turning on in jtop)

gst-launch-1.0 v4l2src device=/dev/video2 io-mode=2 ! image/jpeg, width=1280, height=720, framerate=30/1, format=MJPG ! nvjpegdec ! video/x-raw ! nvvidconv ! ‘video/x-raw(memory:NVMM), width=1280, height=720’ ! nvoverlaysink

  1. Similar to above, but visualizing with xvimagesink

gst-launch-1.0 v4l2src device=/dev/video0 io-mode=2 ! image/jpeg,width=640,height=360,framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! video/x-raw ! videoconvert ! video/x-raw ! xvimagesink

  1. Also works with nvv4l2decoder

gst-launch-1.0 v4l2src device=/dev/video2 io-mode=2 ! image/jpeg, width=1280, height=720, framerate=30/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! video/x-raw ! videoconvert ! video/x-raw ! xvimagesink

All of the above do not work inside cv2.VideoCapture().

  1. Number one in OpenCV:

cap = cv2.VideoCapture(“v4l2src device=/dev/video0 io-mode=2 ! image/jpeg,width=640,height=360,framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! video/x-raw ! videoconvert ! video/x-raw ! appsink”, cv2.CAP_GSTREAMER)

gives error:

(python3:23088): GStreamer-CRITICAL **: 22:32:32.992: gst_element_make_from_uri: assertion ‘gst_uri_is_valid (uri)’ failed

(python3:23088): GStreamer-CRITICAL **: 22:32:32.995: gst_element_link_pads_filtered: assertion ‘GST_IS_BIN (parent)’ failed

(python3:23088): GStreamer-CRITICAL **: 22:32:32.996: gst_element_link_pads_filtered: assertion ‘GST_IS_BIN (parent)’ failed
[ WARN:0] global /home/datemike/opencv/modules/videoio/src/cap_gstreamer.cpp (713) open OpenCV | GStreamer warning: Error opening bin: syntax error
[ WARN:0] global /home/datemike/opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created

  1. Number two and three in OpenCV:

cap = cv2.VideoCapture(“v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=640, height=360, framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! video/x-raw ! videoconvert ! video/x-raw ! appsink”)

cap = cv2.VideoCapture(“v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=640, height=360, framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! video/x-raw ! videoconvert ! video/x-raw ! appsink”)

Displays black and white only!
Looking at the output array, the ‘color’ dimension is only size 1, instead of 3.

I tried adding and deleting the ‘cv2.CAP_GSTREAMER’ argument at the end, and it makes no difference.

In your cases 2 and 3, it sends NV12 format. NV12 format is one channel Mat (subsampled color info is added below the luminance) and cv2.imshow thinks it is monochrome.

For opencv appsink, you would use input caps ‘video/x-raw, format=BGR’. However be aware that videoconvert is CPU only, so it is better to keep its task simple using nvvidconv to prepare into BGRx format:

cap = cv2.VideoCapture("v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=640, height=360, framerate=60/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink drop=1", cv2.CAP_GSTREAMER)

Thank you very much! It fixed my issue :)