Custom mipi driver pixelformat

I have a custom made MIPI camera, and a custom driver, connected to the Xavier NX. I get MIPI video from my camera and am able to stream it over RTSP. The pixelformat, from my MIPI driver, is YUYV (YUY2 in GStreamer).

The encoder, omxh264enc, only accepts NV12 and I420 pixelformats. Using nvvidconv or videoconvert adds too much latency, so I’ve looked into glcolorconvert to do the color conversion on the GPU. The problem is that by going from YUY2 to any other pixelformat results in bad colors with the glcolorconvert element. It’s reproduceable on a Ubuntu18.04 desktop, so the bug is in GStreamer and fixed in the later versions (tested GStreamer1.14 vs GStreamer1.16).

Is it possible to do the color conversion in v4l2 instead of GStreamer, or give more pixelformats possibilities? Ideally the video format needed in GStreamer would be either NV12 or I420. If updating to a later version of GStreamer is more feasible, how can that be done?

Kind regards,
Goran Broeckaert

Did you use v4l2src? Could you try nvv4l2camerasrc

nvv4l2camerasrc does not work. I can’t get any frames when using this element.

My current pipeline (to test whether the color conversion is working) is:

gst-launch1.0 v4l2src device=/dev/video0 num-buffers=1000 ! glupload ! glcolorconvert ! gldownload ! video/x-raw,format=NV12 ! omxh264enc ! filesink location=test.h264

I read the file on a desktop and see that the colors are different. The video format outputted by v4l2src is YUY2. I have also looked with videotestsrc that output YUY2 video, the results are the same. The resolution of the video does not have any influence and the encoder is also not the influence for the weird behavior.

Hi,
The source format is YUYV so you would need to customize nvv4l2camerasrc plugin to support the format. Please refer to
Macrosilicon USB - #5 by DaneLLL

And then you can run like:

nvv4l2camerasrc ! 'video/x-raw(memory:NVMM),format=YUY2,width=_SOURCE_W_,width=_SOURCE_H_.framerate=_SOURCE_FR_' ! nvvidconv ! nvv4l2h264enc ! h264parse ! matroskamux ! filesink location=test.mkv

The goal is to also stream the video using RTSP. The filesink is so I can iterate easier. Does nvvidconv add a lot of latency when using nvv4l2camerasrc?

Hi,
For setting up RTSP server, please refer to this post:
RTSP stream not recognizing USB camera - #3 by DaneLLL

It works with v4l2src and there is a memory copy from CPU buffer to NVMM buffer. If you don’t hit performance issue, it is good to use v4l2src. For further improvement, you can customize nvv4l2camerasrc to support YUYV. This would eliminate the memory copy.