PIXEL_FMT_RAW16 support

Autonomous Machines Jetson & Embedded Systems Jetson TX1
1

In the Argus Camera API Release Notes, it lists PIXEL_FMT_YCbCr_420_888 as the only supported format:

* Current EGLStream Buffer Format Support:

    Format                     State
    =======================    ==========
    PIXEL_FMT_Y8               Not Supported
    PIXEL_FMT_Y16              Not Supported
    PIXEL_FMT_YCbCr_420_888    Works with all consumers.
    PIXEL_FMT_YCbCr_422_888    Not Supported
    PIXEL_FMT_YCbCr_444_888    Not Supported
    PIXEL_FMT_JPEG_BLOB        Not Supported (TODO: Remove format)
    PIXEL_FMT_RAW16            Not Supported

I would like to have raw bayer data (PIXEL_FMT_RAW16). I can currently get it from v4l2 through /dev/video0, but it would be nice if I could use argus instead.

Can we have support for this format please?

2

hello jholt,

may i know your use-case about go through argus api with raw data?

argus api received isp output, hence it’s only support PIXEL_FMT_YCbCr_420_888 format so far.
thanks

3

Thanks Jerry. My request probably seems weird since most of argus deals with the ISP, and I’m trying to avoid the ISP and get raw bayer data. Therefore many nice features in argus will not be relevant to me.

I’m working with Leopard Imaging on a possible bug in the v4l2 driver for their tx1 camera module. They are very helpful, but they suggested that argus is the preferred library for accessing the camera. So that made me think that raw bayer support will have better luck for everyone if it’s supported in argus. Otherwise it’s easy for low-level features to be skipped in the v4l2 driver.

(Ideally I’d also like to program the sensor registers directly without any abstraction layer, but right now I’m very happy to have raw bayer data).

4

hello jholt,

had you tried to get raw data by v4l2-ctl ?

v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count==100 --stream-to=video.raw
5

Yes, with the Nvidia OV5693 board I get good results with both v4l-ctl and my C++ v4l2 code. (With the Leopard Imaging sensor I get all 0’s on frames except from buffer index 0, and some corruption of image data. --verbose flag to v4l2-ctl shows buffer index)

6

hello jholt,

v4l2-ctl is the standard v4l2 control to access camera sensor, what’s the commands you were using?
did you able to launch camera by argus but failed by v4l2-ctl ?

7

nvgstcapture and tegra_multimedia_api/samples/09_camera_jpeg_capture both produce OK frames with the Leopard IMX274. But v4l2-ctl produces empty frames:

ubuntu@tegra-ubuntu:~/tmp$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=1 --stream-to=video.raw
<
ubuntu@tegra-ubuntu:~/tmp$ od -x video.raw |head
0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
17644000

If we add the --verbose flag and capture 3 frames, we see the first two frames have the ‘error’ flag set, which is no problem. Those two frames are ignored. The next three frames get stored in video.raw, and they come from buffer indexes 2, 3, and 0:

ubuntu@tegra-ubuntu:~/tmp$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=3 --stream-to=video.raw --verbose 
VIDIOC_QUERYCAP: ok
VIDIOC_S_EXT_CTRLS: ok
VIDIOC_G_FMT: ok
VIDIOC_S_FMT: ok
Format Video Capture:
	Width/Height      : 1920/1080
	Pixel Format      : 'RG10'
	Field             : None
	Bytes per Line    : 3840
	Size Image        : 4147200
	Colorspace        : sRGB
	Transfer Function : Default
	YCbCr Encoding    : Default
	Quantization      : Default
	Flags             : 
VIDIOC_REQBUFS: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_STREAMON: ok
	Index    : 0
	Type     : Video Capture
	Flags    : mapped, done, error
	Field    : None
	Sequence : 0
	Length   : 4147200
	Bytesused: 4147200
	Timestamp: 248312.526287s (Monotonic, End-of-Frame)

	Index    : 1
	Type     : Video Capture
	Flags    : mapped, done, error
	Field    : None
	Sequence : 1
	Length   : 4147200
	Bytesused: 4147200
	Timestamp: 248312.544060s (Monotonic, End-of-Frame)

	Index    : 2
	Type     : Video Capture
	Flags    : mapped, done
	Field    : None
	Sequence : 2
	Length   : 4147200
	Bytesused: 4147200
	Timestamp: 248312.562128s (Monotonic, End-of-Frame)

	Index    : 3
	Type     : Video Capture
	Flags    : mapped, done
	Field    : None
	Sequence : 3
	Length   : 4147200
	Bytesused: 4147200
	Timestamp: 248312.579850s (Monotonic, End-of-Frame)

	Index    : 0
	Type     : Video Capture
	Flags    : mapped, done
	Field    : None
	Sequence : 4
	Length   : 4147200
	Bytesused: 4147200
	Timestamp: 248312.597679s (Monotonic, End-of-Frame)

VIDIOC_STREAMOFF: ok
buntu@tegra-ubuntu:~/tmp$ ls -l video.raw
-rw-rw-r-- 1 ubuntu ubuntu 12441600 Feb  3 21:44 video.raw

We can use dd to break out the three individual frames, and we see that only the last frame (which comes from buffer index 0) has data in it:

ubuntu@tegra-ubuntu:~/tmp$ dd if=video.raw of=frame1.raw bs=4147200 count=1 skip=0
1+0 records in
1+0 records out
4147200 bytes (4.1 MB, 4.0 MiB) copied, 0.0290863 s, 143 MB/s
ubuntu@tegra-ubuntu:~/tmp$ dd if=video.raw of=frame2.raw bs=4147200 count=1 skip=1
1+0 records in
1+0 records out
4147200 bytes (4.1 MB, 4.0 MiB) copied, 0.0430302 s, 96.4 MB/s
ubuntu@tegra-ubuntu:~/tmp$ dd if=video.raw of=frame3.raw bs=4147200 count=1 skip=2
1+0 records in
1+0 records out
4147200 bytes (4.1 MB, 4.0 MiB) copied, 0.0561688 s, 73.8 MB/s
ubuntu@tegra-ubuntu:~/tmp$ ls -l frame?.raw
-rw-rw-r-- 1 ubuntu ubuntu 4147200 Feb  3 21:50 frame1.raw
-rw-rw-r-- 1 ubuntu ubuntu 4147200 Feb  3 21:50 frame2.raw
-rw-rw-r-- 1 ubuntu ubuntu 4147200 Feb  3 21:50 frame3.raw
ubuntu@tegra-ubuntu:~/tmp$ od -x frame1.raw |head
0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
17644000
ubuntu@tegra-ubuntu:~/tmp$ od -x frame2.raw |head
0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
17644000
ubuntu@tegra-ubuntu:~/tmp$ od -x frame3.raw |head
0000000 0047 0062 004b 0065 004a 0063 004a 0062
0000020 0048 0063 0044 0064 004d 0066 004b 0062
0000040 004b 0064 0047 0061 004a 0060 0050 005b
0000060 0047 005d 004f 0069 0041 0062 004c 005c
0000100 004b 0060 0046 005e 004b 005a 004f 0067
0000120 004c 005e 004b 005b 004a 0064 004f 0063
0000140 0046 0066 0049 005b 0047 0064 004a 0063
0000160 004a 0062 004e 0062 0048 0061 004c 0062
0000200 004f 0061 004c 005e 004a 0060 0047 0065
0000220 004a 0059 0046 0060 004c 005e 004c 0063

(The data in frame3 is also not a good image, but I think that is possibly a different issue from the buffer bug. I think the buffer bug is maybe a simple fix.)

8

hello jholt,

seems your camera driver is IMX274.

  1. could you try to enable the test pattern mode in the imx274_s_stream() function.
  2. if the test pattern mode works correctly, please check the control.value of the gain/ frame length/ coarse time.
9

I tried a variety of sensor_mode values and none of them produced a test pattern:

v4l2-ctl -d /dev/video0 --set-ctrl sensor_mode=6 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=3 --stream-to=video.raw --verbose

In the kernel Leopard sent me, they commented out the test pattern mode and instead made a compile-time define for TEST_PATTERN. From their tri_streaming_imx274_base24.2.1_NVidia_20161207.patch in imx274_mode_tbls.h:

+struct imx274_reg_struct imx274_mode_1920x1080_60fps_reg[] = {
+ {0x3000,  0,  1,       0x1},
+ {0x3000,  1,  1,       0x0},
... (lots of register settings)
+ #ifdef TEST_PATTERN
+ {0x303C, 0x11}, 
+ {0x303D, 0x0A}, // color bar
+ //{0x303D, 0, 8, 0x03}, // gray image
+ {0x370E, 0, 8, 0x01}, 
+ {0x377F, 0, 8, 0x01}, 
+ {0x3781, 0, 8, 0x01}, 
+ {0x370B, 0, 8, 0x11}, // enable
+#endif

and note commented-out mode here:

+static const imx274_reg *mode_table[] = {
+ [IMX274_MODE_3840X2160]     = imx274_mode1_3840x2160_raw10,
+ [IMX274_MODE_1920X1080]     = imx274_mode3_1920x1080_raw10,
+ 
+ [IMX274_MODE_START_STREAM_1]    = imx274_start_1,
+ [IMX274_MODE_START_STREAM_2]    = imx274_start_2,
+ [IMX274_MODE_START_STREAM_3]    = imx274_start_3,
+ [IMX274_MODE_START_STREAM_4]    = imx274_start_4,
+ [IMX274_MODE_STOP_STREAM]     = imx274_stop,
+ [IMX274_MODE_2_LANE_MIPI]     = imx274_2_lane_conf,
+ //[IMX274_MODE_TEST_PATTERN]    = tp_colorbars,
10

hello jholt,

please try to rebuild the kernel image by enable the test pattern.
you should replace /boot/Image with the Image build by your own, and execute warm reboot to refresh the kernel.

you can also adding debug message to check the control.value of the gain/ frame length/ coarse time.
BTW, if the test pattern not works, you should contact your sensor vendor for further suggestion. thanks

11

Thanks for your ongoing help, Jerry. Ok, I am compiling a new kernel now. My intuition says that the buffer bug probably won’t be fixed by having the sensor emit a test pattern. I know that having wrong frame length settings can cause the sensor to have alternating good/bad frames. But in this bug, I can set the number of v4l2 buffers to any number, and it will always give empty frames except when it wraps around to buffer 0. So that makes me think it has little to do with the sensor settings, and is instead a problem with a buffer pointer somewhere related to tegra/camera/channel.c.

12

jholt,

according to your previous comment #5, this looks like sensor driver issue.

Yes, with the Nvidia OV5693 board I get good results with both v4l-ctl and my C++ v4l2 code. (With the Leopard Imaging sensor I get all 0’s on frames except from buffer index 0, and some corruption of image data. --verbose flag to v4l2-ctl shows buffer index)

i would suggest you digging into your imx274 sensor driver to check below functions works well first.
static int imx274_s_stream(struct v4l2_subdev *sd, int enable){…}
static int imx274_s_ctrl(struct v4l2_ctrl *ctrl){…}

13

I added #define TEST_PATTERN in imx274_mode_tbls.h and now I get a garbled test pattern instead of garbled image data, but the buffer bug persists (zeros on all frames except from buffer 0).

14

hello jholt,

besides digging into the sensor driver, could you please try this on previous release. (L4T 24.2 or L4T 24.1)

15

hello jholt,

had you contact your sensor vendor to rule out the problem?
is this still an open topic?

16

A few weeks ago Leopard said they would look into it. I haven’t heard from them since. I instrumented their imx274 driver to printk the way argus configured the sensor vs. how v4l2 configured the sensor, and they seemed to be identical. I’m done trying to fix it, so feel free to close the topic if you like. Thanks for your help.