Latency problems converting from NV12 to GRAY8 with Gstreamer

[bogusszymon2000]

Hello,
I’m trying to convert a video stream from a camera into gray-scale. My basic pipeline, which doesn’t do any conversion but works with a satisfying latency looks like this:

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1280, height=720, framerate=60/1, format=NV12' ! nvvidconv ! xvimagesink

Starting with this pipeline I tried modify it so I receive stream in gray-scale.
I was really trying to avoid using videoconvert element as it uses CPU but I ended up with only pipeline like this working as I desire, however there is a visible lag in-between frames:

gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM),width=1280, height=720, framerate=60/1, format=NV12' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=I420' ! nvvidconv ! 'video/x-raw, format=GRAY8' ! videoconvert ! xvimagesink

What I’m hoping to get is something that utilizes only nvvidconv element :

gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM),width=1280, height=720, framerate=60/1, format=NV12' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=GRAY8' ! ....some sink

but I’m not sure it that can work standalone with xvimagesink or later with appsink as primarly I plan to emded this pipeline in code so I can extract the gray-scale frame into cv::Mat object.

I was also experimenting with nveglglessink but it doens’t support GRAY8…

UPDATE:

gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM),width=1280, height=720, framerate=60/1, format=NV12' ! nvvidconv compute-hw=GPU nvbuf-memory-type=nvbuf-mem-cuda-device ! 'video/x-raw, format=GRAY8' ! videoconvert ! xvimagesink

This pipeline works faster with the GPU-driven decoding but I still cannot eliminate videoconvert.

[Honey_Patouceul]

I think that the issue is in xvimagesink that doesn’t specify its sink format caps. GRAY8 seems not supported and it causes an error upstream in the source.

When adding videoconvert it works because it converts into YUY2 format. Note that this conversion might not very expensive as it just needs to append constant null chrominances planes to received luminance plane.

However, if your intention is to process these frames from an application, nvarguscamerasrc and nvvidconv would be ok. Try:

gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM),format=NV12,width=1280,height=720,framerate=60/1' ! nvvidconv ! video/x-raw,format=GRAY8 ! queue ! fpsdisplaysink text-overlay=0 video-sink=fakesink -v

You may also try the following python example using opencv for reading monochrome frames and displaying:

import cv2

#cap = cv2.VideoCapture("videotestsrc ! video/x-raw,width=640,height=480,framerate=30/1,format=GRAY8 ! queue ! appsink drop=1", cv2.CAP_GSTREAMER)
cap = cv2.VideoCapture("nvarguscamerasrc ! nvvidconv ! video/x-raw, format=GRAY8 ! queue ! appsink drop=1", cv2.CAP_GSTREAMER)
if not cap.isOpened():
   print('failed to open video capture')
   exit(-1)

while True:
   ret_val, img = cap.read();
   if not ret_val:
      break

   cv2.imshow('Test GRAY8', img)	
   if cv2.waitKey(1) == ord('q'):
      break
 
cap.release()

This would probably be less efficient than videoconvert + xvimagesink.
This example is just intended to show that appsink can work reading GRAY8 frames, hope it helps!

[bogusszymon2000]

@Honey_Patouceul thank you for your answer!
For people interested in this topic I found a decent solution for now, based on this Nvidia documnetation
My new pipeline implements GPU accelerated encoding and queue elements. It works with the performance of the first pipeline I included into my first message here:

gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM),width=1280, height=720, framerate=60/1, format=NV12' ! queue ! nvvidconv compute-hw=GPU nvbuf-memory-type=nvbuf-mem-cuda-device ! 'video/x-raw, format=GRAY8' ! queue ! videoconvert ! xvimagesink

However I will check also the solution of @Honey_Patouceul