Efficient conversion of RAW12 to 16-bit image

Hi,

I’m capturing images via the MIPI bus using V4L2 from a monochrome sensor that outputs 5MPIX RAW12 at 30PFS. But because of the way the TX2 arranges the bits in memory I have to rearrange and bit shift the data in order to form a 16-bit image which can then be stored in an OpenCV Mat (CV_16UC1) for further processing. With some optimization, I can get this operation down to approx 300ms per frame which is still way to slow.

Is there some way to transform the T_L16_F format to 16-bit little endian efficiently, or can the video pipeline be configured to arrange the data in the desired way?

Below is a screenshot of the memory format for reference (Parker TRM - Chapter 27.10.6 RAW Memory Formats):

hello pp28bgsbm,

it’s an internal buffer formats for CSI RAW.
you might go through ISP to have hardware support to convert it as YUV.
furthermore, we had enable monochrome sensor supports starting from l4t-r32.3.1,
please also contact with Jetson Preferred Partners for camera solutions.
thanks

Hi Jerry

What is changed in l4t-r32.3.1 to enable monochrome sensors? Is it something that will solve my problem?

Can you point me to a location where I can read more about this? The release notes do not mention anything about this.

I would like to know more before I upgrade (from L4T 28.2.1), as my sensor driver probably has to be ported to work with L4T 32.3.1 ?

hello pp28bgsbm,

your original request were converting memory formats, but it cannot have configuration since it’s an internal buffer formats usage.

my suggestion in comment #2 were going through ISP to produce YUV images.
there are camera software stack updates for l4t-r32.3.1, which add supports for monochrome sensor types.
since it had enable monochrome sensor going through Camera Core, you’ll also need to contact with Jetson Preferred Partners for camera ISP configurations.

however, you’re still able to enable monochrome sensors without ISP supports, and I think you already done verification.
you might refer to similar discussion threads, such as Topic 1037601, and Topic 1036708.
thanks

Hi Jerry

Thank you for clarification. So the conclusion is that if I want to continue using V4L2, I will have to manually convert from the T_L16_F raw memory format, as I cannot use the ISP in combination with V4L2?

correct

Update: For anyone interested in this topic we have managed to bring down the time it takes to convert a 12-bit image stored in T_L16_F format to 16-bit image format to 9ms. I can’t share the code but it can be done with normal bitwise operations in C++, using a 64bit pointer to the V4L2 frame buffer.