support for grayscale sensors is missing

Hi JerryChang,

here is an image taken with TEGRA_STRIDE_ALIGNMENT 64. I cannot tell if it is better or worse, but it is not good :(

This time, the best looking picture was the raw (bytes not swapped) one.
frame-10-65535-raw.pgm.gz (2.57 MB)

I seems to me that what I get with the command in comment #4 is not the raw output from the sensor, but what I get is the output from the sensor processed by some other step (probably the ISP). Is that true ?

hello phdm,

the commands in comment #4 dump the raw files, which process by the V4L2 media framework.
please also refer to the [Camera Architecture Stack] from [L4T Documentation]-> [NVIDIA Tegra Linux Driver Package]-> [Release 28.2 Development Guide]-> [Camera Development] chapter-> [Camera Software Development Solution] session.

Sorry for some noise here : actually the connection to my monochrome sensor was faulty : some mipi lane was garbled. I now have another sensor that does not have that bug. Command in comment #4 (updated to my 12-bit sensor) actually gives a dump of the images sent by the sensor, with pixels stored as 16-bit big endian values : each 12-bit pixel is shifted by 4 positions to give a 16-bit value.

After adding the following entries to vi2_video_formats

TEGRA_VIDEO_FORMAT(RAW12, 12, Y12_1X12, 2, 1, T_R16_I,
                        RAW12, Y16_BE, "GRAY16_BE"),
TEGRA_VIDEO_FORMAT(RAW12, 12, Y12_1X12, 1, 1, T_L8,
                        RAW12, GREY, "GRAY8"),

and of course my driver telling that its mediabus_format is MEDIA_BUS_FMT_Y12_1X12, I now have the following list of formats :

nvidia@cam5-ubuntu:~$ v4l2-ctl --list-formats
        Index       : 0
        Type        : Video Capture
        Pixel Format: 'Y16 -BE'
        Name        : 16-bit Greyscale BE

        Index       : 1
        Type        : Video Capture
        Pixel Format: 'GREY'
        Name        : 8-bit Greyscale

Unfortunately, using it in a gstreamer pipeline ‘gst-launch-1.0 v4l2src …’ only gives me 2 fps.

If I let my driver pretend that its media_bus_format is MEDIA_BUS_FMT_SRGGB12_1X12, and use a gstreamer pipeline ‘gst-launch-1.0 nvcamerasrc …’ I get at least 20 fps. Why such a poor performance with v4l2src ?

Hello JerryChang,

I checked the above command on a freshly installed tx1 devkit with jetpack-3.2.1 (l4t 28.2) and equipped with a ov5693 sensor provided by nvidia. It confirms a bug in the handling of rawN sensors, with N > 8 : LSB bits are discarded !

Here is the test I made :

nvidia@tegra-ubuntu:/run/user/1001$ v4l2-ctl --list-formats-ext
        Index       : 0
        Type        : Video Capture
        Pixel Format: 'BG10'
        Name        : 10-bit Bayer BGBG/GRGR
                Size: Discrete 2592x1944
                        Interval: Discrete 0.033s (30.000 fps)
nvidia@tegra-ubuntu:/run/user/1001$ v4l2-ctl --set-fmt-video=width=2592,height=1944,pixelformat=BG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=20 --stream-to=bg10.raw
nvidia@tegra-ubuntu:/run/user/1001$ hexdump -C bg10.raw
00000000  32 00 65 00 3a 00 5d 00  33 00 62 00 2e 00 5f 00  |2.e.:.].3.b..._.|
00000010  35 00 6d 00 39 00 57 00  3f 00 6e 00 4b 00 5d 00  |5.m.9.W.?.n.K.].|

As one can see, all the odd bytes are 0[0-3]. This is unexpected from a 10-bit sensor.
This is confirmed bye the following test

nvidia@tegra-ubuntu:/run/user/1001$ hexdump -Cv bg10.raw | grep -v '0[0-3] .. 0[0-3] .. 0[0-3] .. 0[0-3]  .. 0[0-3] .. 0[0-3] .. 0[0-3] .. 0[0-3]'

Do I misunderstand ?
Does the CSI-VI pair truncate the incoming pixels to 8 bits while also replicating the most significant bits ?
Is the image stored in memory in big endian or little endian mode ?
Is the RAW10 value shifted into 16 bits to produce values between 0 and 0xffc0, or are the 10-bits not shifted at all and produce values between 0 and 3ff ? I have read chapters 29 and 32 of the TX1 TRM, but experimental results seem not to match documentation.

You might read this post.

Hi Honey_Patouceul

Thank you.

I have read, and this seems to suggest that the raw10 images of the ov5693 sensor, when acquired with

v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=10 --stream-to=test.raw

are actually stored in 16 bytes word, in little endian form, without any shift occuring, thus with values between 0 and 1023 (2^10-1).

This does not seem to match the TX1 TRM “Raw Pixel Formatting to Memory”/“T_R16_I Formatting for RAW10, RAW12 and RAW14”, where RAW10 pixels are shown written in memory as

[ 0 | 0 | D9| D8| D7| D6| D5| D4| D3| D2| D1| D0| D9| D8| D7| 0 ]

, thus shifted 4 places to the left with lsb filled with some msb replication.

In a CSI->VI->MEM path, how are RAW10 or RAW12 pixels actually transformed and written into memory ?

Sorry I cannot tell much more, but the 16 bits format may only be used by option stream-to which writes to disk.
v4l2-ctl would have read into memory in RG10 format as explained by TRM, and then made the 16 bits format for disk storage with stream-to. So the raw file may not be an exact image of the memory stream. It is much easier to read just 2 bytes and get ‘short’ values for further computing, than having to manage the shifts and mask.


I am currently trying to add grayscale format support to a Jetson Xavier with L4T 31.1 by following the steps described in the above comment for the sensor OV24A1B. I started from the driver for the sensor OV5693.

In ov5693.c, I modified the default data format to match the required format (Gray 10 bits)


In sensor_common.c, I set the pixel format to V4L2_PIX_FMT_Y10

static int extract_pixel_format(
	const char *pixel_t, u32 *format)
	 *format = V4L2_PIX_FMT_Y10;
	return 0;

In camera_common.c, I added a camera common color formats for Grayscale

static const struct camera_common_colorfmt camera_common_color_fmts[] = {

        // Grayscale support

And as mentionned in the above comment, in vi2_formats.h, I added other entries to static const struct tegra_video_format vi2_video_formats :

TEGRA_VIDEO_FORMAT(RAW10, 10, Y10_1X10, 2, 1, T_R16_I,
                                RAW10, Y10, "GRAY10"),			
	TEGRA_VIDEO_FORMAT(RAW10, 10, Y10_1X10, 2, 1, T_R16_I,
                                RAW10, Y12, "GRAY12"),
	TEGRA_VIDEO_FORMAT(RAW10, 10, Y10_1X10, 2, 1, T_R16_I,
                                RAW10, Y16, "GRAY16"),

However, I still get a kernel panic when running this kernel with the stack pointing to nvidia/drivers/media/platform/tegra/camera/vi/channel.c

[    9.025262] Process kworker/1:0 (pid: 18, stack limit = 0xffffffc3ed81c028)
[    9.032165] Call trace:
[    9.034285] [<ffffff8008b5e8cc>] tegra_channel_fmts_bitmap_init+0x17c/0x238
[    9.040920] [<ffffff8008b60754>] tegra_channel_init_subdevices+0x15c/0x738
[    9.047654] [<ffffff8008b61a1c>] tegra_vi_graph_notify_complete+0x53c/0x678
[    9.053885] [<ffffff8008b45f88>] v4l2_async_test_notify+0x108/0x120
[    9.059911] [<ffffff8008b460cc>] v4l2_async_notifier_register+0x12c/0x1a0
[    9.066287] [<ffffff8008b626a8>] tegra_vi_graph_init+0x210/0x290
[    9.071633] [<ffffff8008b5ca98>] tegra_vi_media_controller_init+0x1a8/0x1e0
[    9.078545] [<ffffff8008536340>] vi5_probe+0x340/0x390
[    9.083096] [<ffffff80087afc50>] platform_drv_probe+0x60/0xc8
[    9.088342] [<ffffff80087ad3dc>] driver_probe_device+0x26c/0x420
[    9.094380] [<ffffff80087ad76c>] __device_attach_driver+0xb4/0x160
[    9.100679] [<ffffff80087aae34>] bus_for_each_drv+0x6c/0xa8
[    9.106363] [<ffffff80087acfb4>] __device_attach+0xcc/0x160
[    9.111965] [<ffffff80087ad894>] device_initial_probe+0x24/0x30
[    9.117913] [<ffffff80087ac108>] bus_probe_device+0xa0/0xa8
[    9.123429] [<ffffff80087ac760>] deferred_probe_work_func+0x90/0xe0
[    9.129995] [<ffffff80080d3f60>] process_one_work+0x1e0/0x478
[    9.135769] [<ffffff80080d4444>] worker_thread+0x24c/0x4d0
[    9.141279] [<ffffff80080dabd8>] kthread+0xe8/0x100
[    9.145918] [<ffffff8008083500>] ret_from_fork+0x10/0x50
[    9.151607] ---[ end trace 6425dc7be0d116c3 ]---

Do you have any ideas which step(s) am I missing to add grayscale support ?

Thanks in advance,


I do not have a Xavier, only TX1 and TX2, and while the TX1 uses the vi2_formats.h file, TX2 uses vi4_formats.h. Additions for grayscale must go there also for Xavier, I surmise.

Also, the stack trace is not the most interesting part of the error message (except for the name of the function where it happens : ‘tegra_channel_fmts_bitmap_init’ here), but there is also a panic message (probably about a null pointer). ‘printk’ is your friend :)

Oops :)

Actually, seeing ‘vi5_probe’ in your stack trace, you probably have a ‘vi5_formats.h’

Hello phdm,

Thank you for your fast answer.

I tried to put the new formats in vi5_formats.h instead of vi2_formats.h and it made v4l2-ctl happy without any kernel panic.

Thanks again,