support for grayscale sensors is missing

I try to use a grayscale sensor with l4t 28.2.1, but I came to the conclusion that this is not supported :

In drivers/media/platform/tegra/camera/sensor_common.c, extract_pixel_format seems to support only bayer pixel types. I’d need 8-bits, 10-bits and 12-bits grayscale support.

Is a patch available ?

hello phdm,

may I know what’s your sensor?
please try to implement your sensor drive as general bayer format, and retain only Y information for output result.
thanks

E2V ev76c560 or E2V ev76c570 or SONY imx264 : really grayscale (not bayer) sensors, and I need access to all the original pixels, not after demosaic and rgb to yuv conversion.

hello phdm,

please refer to [Camera Architecture Stack] in the [Release 28.2 Development Guide]
you should be able to use standard v4l2 control to access camera sensors, and dump the raw files to retain all the original pixels.
for example,

v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=10 --stream-to=test.raw

Hello JerryChang,

which file do you refer to precisely ? I don’t find a file called ‘Development Guide’ in the download center.

hello phdm,

please access the L4T documentation from Jetson Download Center,
https://developer.nvidia.com/embedded/dlc/l4t-documentation-28-2-ga

Hi JerryChang,

  1. AFAICS the command line you gave does not do what I need. A conversion still seems to happen.

  2. Thank you for the link. I successfully downloaded the file. Starting to read now :)

There’s no occurrence of the words ‘grey’, ‘gray’, ‘greyscale’ or ‘grayscale’ in the documentation.

What I need to get is either GRAY8, GRAY16_LE, or GRAY16_BE frames using v4l2src. It seems that starts with the ability to tell the nvidia layer that my sensor is a pure grayscale sensor, which seems currently impossible. I would already be satisfied with a GRAY8 stream even if that implies a truncation of the lsb’s of the pixels given by the sensor. GRAY16* would also be good even if pixels are not rescaled to the 0-65535 range.

hello phdm,

you should able to dump the raw files with v4l2 standard controls in comment #4,
please use below command to list the sensor output color format.

$ v4l2-ctl -d /dev/videoX --list-formats

moreover,
here’s commands for your reference, please try to access camera sensor via v4l2src.

$ gst-launch-1.0 -v v4l2src num-buffers=1 device=/dev/video1 ! video/x-raw,framerate=30/1,width=640,height=480 ! nvjpegenc ! filesink location=sample.jpg

Hello JerryChang,

as my sensor is a pure grayscale sensor, I managed to tell that to the kernel to get

s_data->sensor_props->sensor_modes[0].image_properties.pixel_format = V4L2_PIX_FMT_Y12;

This is modeled after what extract_pixel_format does for other sensor formats

after that, my kernel simply crashes :

[    4.138854] Unable to handle kernel NULL pointer dereference at virtual address 00000010
...
[    4.138934] PC is at tegra_channel_update_format+0x14/0xa0
[    4.138944] LR is at tegra_channel_s_ctrl+0x19c/0x230

I then added the following entry to ‘static const struct tegra_video_format vi2_video_formats’ in drivers/media/platform/tegra/camera/vi/vi2_formats.h :

TEGRA_VIDEO_FORMAT(RAW12, 12, Y12_1X12, 2, 1, T_R16_I,
                                RAW12, Y12, "GRAY12"),

With that, I can start linux and list formats :

nvidia@tegra-ubuntu:~$ v4l2-ctl --list-formats
ioctl: VIDIOC_ENUM_FMT
        Index       : 0
        Type        : Video Capture
        Pixel Format: 'Y12 '
        Name        : 12-bit Greyscale

nvidia@tegra-ubuntu:~$

I can also run the command from comment #4, with either RG12 or Y12 as format; that gives me per frame a size of 1920x1080x2 (2 bytes per pixel), but this is clearly not the raw output from my sensor as I can recognize a repeated pattern in the images, which is not in the observed scene.

v4l2src is unusable for Y12; it accepts only GRAY8, GRAY16_BE and GRAY16_LE, so I added other entries to static const struct tegra_video_format vi2_video_formats :

TEGRA_VIDEO_FORMAT(RAW12, 12, Y12_1X12, 1, 1, T_L8,
                                RAW12, GREY, "GRAY8"),
        TEGRA_VIDEO_FORMAT(RAW12, 12, Y12_1X12, 2, 1, T_R16_I,
                                RAW12, Y16, "GRAY16"),

I now have

nvidia@tegra-ubuntu:~$ v4l2-ctl --list-formats
ioctl: VIDIOC_ENUM_FMT
        Index       : 0
        Type        : Video Capture
        Pixel Format: 'GREY'
        Name        : 8-bit Greyscale

        Index       : 1
        Type        : Video Capture
        Pixel Format: 'Y16 '
        Name        : 16-bit Greyscale

nvidia@tegra-ubuntu:~$

That makes v4l2src happy, but the images I get with either v4l2-ctl or a ‘gst-launch-1.0 v4l2src’ pipeline are still garbled.

hello phdm,

please also refer to the [Tegra X1 Technical Reference Manual], from Jetson Download Center.
please check the [Video Input (VI)]-> [Data Formatting]-> [YUV Pixel Data Formatting] chapter for more details about memory format.
moreover, suggest you to use 3rdparty tools (i.e. 7yuv) to analysis raw format in hex values.
thanks

Hi JerryChang,

the command from comment #4 seems to work with the ov5693 camera mounted on the tx1 evk. Can you
confirm that this really is the raw data sent by the sensor and not some reconstructed data after two transformations ?

I ask that because when I issue the same command (with RG10 replaced by RG12) with a monochrome imx264 sensor, I still get images with some structure in it, that does not seem to come from the sensor.

I have noticed that TEGRA_VI_CSI_IMAGE_DEF is configured with BYPASS_PXL_TRANSFORM set to 0, and IMAGE_DEF_FORMAT set to 32 (TEGRA_IMAGE_FORMAT_T_R16_I) while TEGRA_VI_CSI_IMAGE_DT is set to 44 (TEGRA_IMAGE_DT_RAW12). I have not found if something is done with vf_code (=4 = TEGRA_VF_RAW12)

With that, I get e.g. in a zone of my image where the real intensity is more or less constant the following byte series :

0000000 f0 00 f0 00 f1 00 3c 00 f1 00 ef 00 f0 00 3c 04
0000020 ef 00 f0 00 f0 00 3b 04 ee 00 ef 00 f1 00 3c 0c
0000040 f0 00 f0 00 f0 00 3c 00 f1 00 f0 00 f1 00 3c 00
0000060 f0 00 f1 00 f1 00 3c 00 f0 00 f0 00 f0 00 3b 04
0000100 f0 00 ef 00 f1 00 3c 0c f0 00 f1 00 f0 00 3c 04
0000120 ee 00 f1 00 f0 00 3c 04 ef 00 ee 00 f0 00 3c 04
0000140 f0 00 f0 00 f1 00 3c 04 f1 00 f0 00 f0 00 3c 00
0000160 f1 00 f0 00 ed 00 3c 04 f0 00 f1 00 f0 00 3c 00

One can observe two repetitive phenomenons :

  • bytes alternating between some real measure and 00
  • every fourth 2-bytes column a value beginning with ‘3’ instead of ‘f’

I really have a 12-bit monochrome sensor. How can I get the untransformed pixels in v4l2 buffers ?

This is with 28.2.1 on a tx1 board

hello phdm,

suggest you to use 3rdparty tools, such as raw2bmp, or 7yuv to convert raw files into images.
the commands from comment #4 is raw files dumped by the VI driver, which is following the RG10 pixel format and writing into memory.
you may also refer to [Camera Architecture Stack] in the [Release 28.2 Development Guide] to understand the hardware blocks and buffer flows.
thanks

I split and convert the raw output file to pgm, which is the simplest image file format using the following script :

v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RG12 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=20 --stream-to=test.raw
for i in $(seq 10 19)
do
        for m in 4095 32767 65535
        do
                (
                echo P5
                echo 1920 1080
                echo $m
                dd if=test.raw conv=swab bs=4147200 skip=$i count=1
                ) > frame-$i-swab-$m.pgm
                (
                echo P5
                echo 1920 1080
                echo $m
                dd if=test.raw bs=4147200 skip=$i count=1
                ) > frame-$i-raw-$m.pgm
        done
done

I skip the first 10 frames to give some time to the sensor to stabilize, and try every combination of big-endian or little-endian way of encoding 12 bits on 2 bytes and not rescaled (highest pixel value is 4095), rescaled to 15 bits (highest pixel value is 32767) or rescaled to 16 bits (highest pixel value is 65535), but no combination gives me a perfect raw monochrome image.

hello phdm,

could you please attach the *.pgm file, let’s see what’s it looks like.
thanks

Hi JerryChang,

I’d like to attach a .pgm file, but I don’t find how to do that.
frame-10-raw-65535.pgm.gz (2.28 MB)

You can attach a file to a previous post.
Moving your mouse cursor in the upper right corner of a previous post (being logged in), you will see a paper clip icon for this purpose.

OK, file is attached to comment #16 (as .pgm.gz, because plain .pgm is refused)

It’s the best combination I’ve found of swab or notswab and maxpixel = 2^12-1, 2^15-1 or 2^16-1.
Surprisingly, 2^16-1 is the best maxval but that does not match documentation (2^15-1 or 2^12-1) as I read it

Have you tried qv4l2 ? Can you get correct capture with it ?

Furthermore, it may have changed, but I think that gstreamer v4l2src plugin was only working with 8 bits (rggb, maybe gray8) format.

hello phdm,

I had check your pgm file,
could you please update the kernel driver to modify the TEGRA_STRIDE_ALIGNMENT to 64,

for example,

--- a/drivers/media/platform/tegra/camera/vi/core.h
+++ b/drivers/media/platform/tegra/camera/vi/core.h
@@ -22,8 +22,10 @@
-#define TEGRA_STRIDE_ALIGNMENT 256
+#define TEGRA_STRIDE_ALIGNMENT 64