Custom driver for csi-2 raw bayer sensor. (Use ISP for format conversion)

Autonomous Machines Jetson & Embedded Systems Jetson TX1
#1

Dear Sirs. We need your assistance.
There are custom carrier board for JetsonTX1 module and custom camera board with CSI-2 sensor.
http://i.imgur.com/vl7KgbZ.jpg

The driver was created according to L4T-docs(24.2.1) chapter “Sensor Driver Programming Guide” (based on OV5693 example).
We can see image(extracted from raw bayer data) from sensor
http://i.imgur.com/sI03bH4.png

There are problems:

  1. The alternation of empty(zero) and normal(full) frames is observed.
    http://i.imgur.com/iLWzqgq.jpg

  2. ‘Mipi cal timeout’ and ‘MW_ACK_DONE syncpoint time out’ are single warnings at start of the stream.
    http://pastebin.com/72Sy4dw5

  3. ‘nvcamera-daemon’ fault (using nvgstcapture or gst-launch)
    http://pastebin.com/56J9EFqA

Please, help. Can you check our DeviceTree dtsi-file (see below)? How to fix these problems?

Lane clock osc is continuous (HP, differential state always), Lane data osc has HP<>LP transitions.
Module pins: G24: CSI2_CK_N ; G25: CSI2_CK_P ; F25: CSI2_D0_N ; F26: CSI2_D0_P ; H23: CSI2_D1_N ; H24: CSI2_D1_P ; H8: SENSOR_RESET ; C6: I2C_SCL ; D6: I2C_SDA
Master clock is not used, because sensor module has onboard 27MHz clock-gen. Power control is not used, because camera supply is always on. Only reset line control is used for sensor startup. Camera module has no EEPROM, so we include dtsi-file manually (see patches below). Sensor has no onboard ISP, so the output is only RawBayer. We need to utilize TX1 ISP for ‘Raw->NV12’ conversion. Does ‘nvcamerasrc’ do this automatically without any configs?

‘drivers/media/platform/tegra/camera/camera_common.c’ has no support for 12bit mode, see the end of “camera_common_try_fmt”. It return -22(-EINVAL) for V4L2_MBUS_FMT_SRGGB12_1X12. See patch below.

Here is workflow for JetPack_3.0, R24.2.1. (for TX1)
http://pastebin.com/MZAmHjdc

Makefile
http://pastebin.com/HmizaxvC

Patch for include sensor in config
http://pastebin.com/pxcVwWRL

Patch for config
http://pastebin.com/tWTSaQvw

Patch for camera_common.c (enable raw_12bit support)
http://pastebin.com/LKgsKu6M

DeviceTree dtsi-file
http://pastebin.com/1mpmxAF8

dmesg at start
http://pastebin.com/WP0tLhSF

v4l2-compliance (v4l2 utils via apt-get install)
http://pastebin.com/81AbyVrZ

v4l2-ctl capture log (please look at mipi timeout message near the end of log)
http://pastebin.com/sgTHqVNt

yavta test
http://pastebin.com/tcyKaDiV

gst-inspect
http://pastebin.com/g3RVxBq5

nvgstcapture-1.0 fail with dmesg output
http://pastebin.com/NVnkfA9R

gst-launch-1.0 nvcamerasrc ! ‘video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)30/1’ ! fakesink
[dmesg and gst debug log]
http://pastebin.com/NaR7hYNn

Thank you in advance. We really hope for your help as soon as possible.

#2

Hi
There’s few thing you can try. And min_gain_val should not be 0 try to modify is as 1.

  1. Check the MIPI timing is fit the MIPI spec.
  2. try discontinuous mode
  3. increase the pix_clk_hz to “160000000” and disable embedded_metadata if it’s possible
#3

Hi, ShaneCCC.

It seems that the DeviceTree nodes like ‘pix_clk_hz’,‘line_length’,‘active_w’,‘active_h’ are not parsed.

There is only one module called ‘camera_common.c’ where these features parsed.

$ grep -r 'pix_clk_hz' sources/kernel/drivers/*

sources/kernel/drivers/media/platform/tegra/camera/camera_common.c: of_property_read_string(node, "pix_clk_hz", &str);
sources/kernel/drivers/media/platform/tegra/camera/camera_common.c: dev_err(&client->dev, "Failed to read mode pix_clk_hz\n");
sources/kernel/drivers/media/platform/tegra/camera/camera_common.c: dev_err(&client->dev, "Failed to convert mode pix_clk_hz\n");

Function called as ‘camera_common_parse_sensor_mode()’.

There is only one sensor driver do this call.

$ grep -r -i 'camera_common_parse_sensor_mode' sources/kernel/drivers/*
 
sources/kernel/drivers/media/i2c/imx185.c: err = camera_common_parse_sensor_mode(client, board_priv_pdata);
sources/kernel/drivers/media/platform/tegra/camera/camera_common.c: int camera_common_parse_sensor_mode(struct i2c_client *client,
sources/kernel/drivers/media/platform/tegra/camera/camera_common.c: EXPORT_SYMBOL(camera_common_parse_sensor_mode);
sources/kernel/include/media/camera_common.h: int camera_common_parse_sensor_mode(struct i2c_client *client,

In my case, this call was not used, because it was not found in the reference example of the driver ‘media/i2c/ov5693.c’.

Feature ‘pix_clk_hz’ is read to ‘struct camera_common_pdata *pdata’ to ‘pdata->mode_info[i].pixel_clock’ (see ‘camera_common_parse_sensor_mode’).
But only ‘imx185.c’ is use it for frame_length calc.

$ grep -r '].pixel_clock' sources/kernel/drivers/media/*
sources/kernel/drivers/media/i2c/imx185.c:	frame_length = mode[s_data->mode].pixel_clock *
sources/kernel/drivers/media/i2c/imx185.c:	coarse_time = mode[s_data->mode].pixel_clock * val /
sources/kernel/drivers/media/platform/tegra/camera/camera_common.c:		err = kstrtoll(str, 10, &pdata->mode_info[i].pixel_clock);
sources/kernel/drivers/media/platform/tegra/camera/camera_common.c:			pdata->mode_info[i].pixel_clock);

I can call ‘camera_common_parse_sensor_mode’ from ‘parse_dt’ function, and these fields are parsed, but what is the profit?

It seems that other dtsi-file fields are also not used.
The following fields are found only in files located in the directory ‘arch/arm64/boot/dts’ but we can’t see any mentions in c-files located at ‘drivers/media’.

$ grep -r \
-e 'min_exp_time' -e 'max_exp_time' \
-e 'min_framerate' -e 'max_framerate' \
-e 'min_hdr_ratio' -e 'max_hdr_ratio' \
-e 'min_gain_val' -e 'max_gain_val' \
-e 'inherent_gain' \
-e 'embedded_metadata_height' -e 'mclk_multiplier' \
-e 'readout_orientation' -e 'pixel_t' \
-e 'dpcm_enable' -e 'cil_settletime' \
-e 'discontinuous_clk' -e 'num_lanes' \
-e 'tegra_sinterface' -e 'mclk_khz' \
sources/kernel/drivers/*

There are nothing interesting here.
http://pastebin.com/bkP3qx5A

So, I’m very confused now. Where are CSI-2 settings like ‘embedded_metadata’, ‘tegra_sinterface’ located in drivers?

#4

Hi
Most of them are parser by the user space un-public camera lib when you launch the nvcamerasrc(gstreamer).

#5

Hi!
Thank you for info. I see them inside ‘libnvodm_imager.so’.

target $ dd if=/usr/lib/aarch64-linux-gnu/tegra/libnvodm_imager.so ibs=1 skip=4970752 bs=1 count=512 | hexdump -C
00000000  65 72 61 74 65 00 00 00  6d 69 6e 5f 65 78 70 5f  |erate...min_exp_|
00000010  74 69 6d 65 00 00 00 00  6d 61 78 5f 65 78 70 5f  |time....max_exp_|
00000020  74 69 6d 65 00 00 00 00  6d 69 6e 5f 67 61 69 6e  |time....min_gain|
00000030  5f 76 61 6c 00 00 00 00  6d 61 78 5f 67 61 69 6e  |_val....max_gain|
00000040  5f 76 61 6c 00 00 00 00  6d 69 6e 5f 68 64 72 5f  |_val....min_hdr_|
00000050  72 61 74 69 6f 00 00 00  6d 61 78 5f 68 64 72 5f  |ratio...max_hdr_|
00000060  72 61 74 69 6f 00 00 00  6d 63 6c 6b 5f 6b 68 7a  |ratio...mclk_khz|
00000070  00 00 00 00 00 00 00 00  6d 63 6c 6b 5f 6d 75 6c  |........mclk_mul|
00000080  74 69 70 6c 69 65 72 00  63 69 6c 5f 73 65 74 74  |tiplier.cil_sett|
00000090  6c 65 74 69 6d 65 00 00  64 69 73 63 6f 6e 74 69  |letime..disconti|
000000a0  6e 75 6f 75 73 5f 63 6c  6b 00 00 00 00 00 00 00  |nuous_clk.......|
000000b0  65 6d 62 65 64 64 65 64  5f 6d 65 74 61 64 61 74  |embedded_metadat|
000000c0  61 5f 68 65 69 67 68 74  00 00 00 00 00 00 00 00  |a_height........|
000000d0  64 70 63 6d 5f 65 6e 61  62 6c 65 00 00 00 00 00  |dpcm_enable.....|
000000e0  74 72 75 65 00 00 00 00  78 5f 73 74 61 72 74 00  |true....x_start.|
000000f0  79 5f 73 74 61 72 74 00  78 5f 65 6e 64 00 00 00  |y_start.x_end...|
00000100  79 5f 65 6e 64 00 00 00  68 5f 73 63 61 6c 69 6e  |y_end...h_scalin|
00000110  67 00 00 00 00 00 00 00  76 5f 73 63 61 6c 69 6e  |g.......v_scalin|
00000120  67 00 00 00 00 00 00 00  73 65 6e 73 6f 72 5f 6d  |g.......sensor_m|
00000130  6f 64 65 6c 00 00 00 00  25 73 3a 20 50 61 73 73  |odel....%s: Pass|
00000140  65 64 20 69 6e 20 69 6e  76 61 6c 69 64 20 4e 55  |ed in invalid NU|
00000150  4c 4c 20 70 61 72 61 6d  65 74 65 72 73 00 00 00  |LL parameters...|
00000160  25 73 3a 20 43 61 6e 20  6e 6f 74 20 61 6c 6c 6f  |%s: Can not allo|
00000170  63 61 74 65 20 63 6f 6e  74 65 78 74 20 30 78 25  |cate context 0x%|
00000180  58 00 00 00 00 00 00 00  25 73 3a 20 68 44 65 76  |X.......%s: hDev|
00000190  20 54 61 62 6c 65 20 6e  6f 74 20 69 6e 69 74 69  | Table not initi|
000001a0  61 6c 69 7a 65 64 00 00  76 69 2d 6f 75 74 70 75  |alized..vi-outpu|
000001b0  74 00 00 00 00 00 00 00  25 73 3a 20 44 65 76 69  |t.......%s: Devi|
000001c0  63 65 20 69 73 20 6e 6f  74 20 6f 66 20 43 53 49  |ce is not of CSI|
000001d0  20 74 79 70 65 00 00 00  75 73 65 5f 64 65 63 69  | type...use_deci|
000001e0  62 65 6c 5f 67 61 69 6e  00 00 00 00 00 00 00 00  |bel_gain........|
000001f0  75 73 65 5f 73 65 6e 73  6f 72 5f 6d 6f 64 65 5f  |use_sensor_mode_|

Also I can see that sensors are hardcoded inside this library.

target $ hexdump -C /usr/lib/aarch64-linux-gnu/tegra/libnvodm_imager.so | grep imx
000bf380  6f 64 65 00 00 00 00 00  2f 64 65 76 2f 69 6d 78  |ode...../dev/imx|
000bf3c0  20 25 73 0a 00 00 00 00  69 6d 78 31 33 32 3a 20  | %s.....imx132: |
000bf3f0  69 6d 78 31 33 32 3a 20  43 61 6e 20 6e 6f 74 20  |imx132: Can not |
000bf480  5f 69 6d 78 31 33 32 2e  63 00 00 00 00 00 00 00  |_imx132.c.......|
000bf4b0  20 25 73 0a 00 00 00 00  45 72 72 3a 69 6d 78 31  | %s.....Err:imx1|
000e7c30  72 20 4f 70 65 6e 0a 00  2f 64 65 76 2f 69 6d 78  |r Open../dev/imx|
000e7c70  20 25 73 0a 00 00 00 00  45 72 72 3a 69 6d 78 31  | %s.....Err:imx1|
000e7da0  5f 69 6d 78 31 33 35 2e  63 00 00 00 00 00 00 00  |_imx135.c.......|
00122220  6f 64 65 00 00 00 00 00  2f 64 65 76 2f 69 6d 78  |ode...../dev/imx|
001222b0  6f 72 5f 62 61 79 65 72  5f 69 6d 78 31 37 39 2e  |or_bayer_imx179.|
001222c0  63 00 00 00 00 00 00 00  45 72 72 3a 69 6d 78 31  |c.......Err:imx1|
0014d170  2f 64 65 76 2f 69 6d 78  32 30 38 00 00 00 00 00  |/dev/imx208.....|
0014d220  6f 72 5f 62 61 79 65 72  5f 69 6d 78 32 30 38 2e  |or_bayer_imx208.|
0014d230  63 00 00 00 00 00 00 00  45 72 72 3a 69 6d 78 32  |c.......Err:imx2|
0018a540  2f 64 65 76 2f 69 6d 78  32 31 34 00 00 00 00 00  |/dev/imx214.....|
0018a5d0  5f 69 6d 78 32 31 34 2e  63 00 00 00 00 00 00 00  |_imx214.c.......|
0018a5e0  45 72 72 3a 69 6d 78 32  31 34 20 6f 64 6d 3a 25  |Err:imx214 odm:%|
0018a700  5f 45 52 53 5f 69 6d 78  32 31 34 0a 23 2a 20 49  |_ERS_imx214.#* I|
001b78e0  6f 64 65 00 00 00 00 00  2f 64 65 76 2f 69 6d 78  |ode...../dev/imx|
001b7960  64 2e 0a 00 00 00 00 00  45 72 72 3a 69 6d 78 32  |d.......Err:imx2|
001b7a30  6f 72 5f 62 61 79 65 72  5f 69 6d 78 32 31 39 2e  |or_bayer_imx219.|
001b7af0  69 6d 78 32 31 39 0a 23  2a 20 53 65 6e 73 6f 72  |imx219.#* Sensor|
002b2b80  65 31 36 33 33 5f 69 6d  78 32 30 38 00 00 00 00  |e1633_imx208....|
002b2b90  65 31 36 33 33 5f 69 6d  78 32 31 39 00 00 00 00  |e1633_imx219....|
002b2ba0  69 6d 78 32 37 34 5f 66  72 6f 6e 74 5f 41 36 56  |imx274_front_A6V|
002b2c60  69 6d 78 31 38 35 5f 62  6f 74 74 6f 6d 5f 6c 69  |imx185_bottom_li|
002b2c70  69 6d 78 31 38 35 00 00  23 2a 20 3d 3d 3d 3d 3d  |imx185..#* =====|

I see that nvcamera-daemon is linked against this library:

target $ ldd /usr/sbin/nvcamera-daemon | grep imager
	libnvodm_imager.so => /usr/lib/aarch64-linux-gnu/tegra/libnvodm_imager.so (0x0000007f787c6000)

So, the question is:
(1): How to add new sensor to ‘libnvodm_imager.so’ library?

Also, I need to solve the problem of alternating useful and zero frames when using v4l2-ctl capture.

(2): Which type of ‘v4l-utils’ package should be used (apt-get source or nvidia source-pack)?

Remove apt-get installed binary version.

target $ sudo apt-get remove v4l-utils

So, we can see TWO variants of v4l-utils comes from sources.

One comes from Nvidia download center as ‘sources’ pack
http://developer.download.nvidia.com/embedded/L4T/r24_Release_v2.1/BSP/sources_r24.2.1.tbz2

55f301507f978f23a868437515d041f0d6e8aa68 /opt/tx1/r24.2.1/sources_r24.2.1.tbz2

It contains

81ef29a2c171bad749c89b3658ff5f7ce3bafdeb sources/v4l2-utils_src.tbz2

This type of sources fails to compile on target!

CXXLD    qv4l2
../v4l2-ctl/qv4l2-vivid-tpg.o: In function `vzalloc':
/home/ubuntu/build/v4l2utils/utils/qv4l2/../v4l2-ctl/vivid-tpg.c:92: undefined reference to `v4l2_page_size'
/home/ubuntu/build/v4l2utils/utils/qv4l2/../v4l2-ctl/vivid-tpg.c:92: undefined reference to `v4l2_page_size'

Please, see the workflow of how to build and log here:
http://pastebin.com/EvqA5LkJ

Did anything special rules exist to build this packet? I can’t see any special Nvidia readme file.

Second version of ‘v4l-utils’ comes from ubuntu-repository.

target $ mkdir /home/ubuntu/build
target $ cd /home/ubuntu/build
target $ apt-get source v4l-utils
target $ cd v4l-utils-1.10.0
target $ ./bootstrap.sh
target $ ./configure
target $ make
target $ sudo make install

These sources are built without problems.

(3): How to set mipi csi settings if we use v4l2-ctl capture?

target $ ldd /usr/local/bin/v4l2-ctl
	linux-vdso.so.1 =>  (0x0000007f87c90000)
	libv4l2.so.0 => /usr/lib/aarch64-linux-gnu/libv4l2.so.0 (0x0000007f87c56000)
	libstdc++.so.6 => /usr/lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000007f87ac7000)
	libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000007f87aa5000)
	libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007f8795f000)
	/lib/ld-linux-aarch64.so.1 (0x0000005562797000)
	libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000007f87933000)
	libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000007f8791f000)
	libv4lconvert.so.0 => /usr/lib/aarch64-linux-gnu/libv4lconvert.so.0 (0x0000007f8789a000)
	libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000007f877ef000)
	librt.so.1 => /lib/aarch64-linux-gnu/librt.so.1 (0x0000007f877d7000)
	libjpeg.so.8 => /usr/lib/aarch64-linux-gnu/libjpeg.so.8 (0x0000007f87790000)

It does not use libnvodm_imager library.

Thank you in advance.

#6
  1. You don’t need to add any sensor to libnvodm_imager.so that is for old version and not remove it.
  2. Both of V4l2-utils can work. You can just verify it in the reference sensor board first.
  3. Need to modify the csi2_fop.c current it hard code as oxA, search the “TEGRA_CSI_CIL_PHY_CONTROL” to find it.
void csi2_start_streaming(struct tegra_csi_device *csi,
                          enum tegra_csi_port_num port_num)
{
        struct tegra_csi_port *port = &csi->ports[port_num];

        csi2_write(csi, TEGRA_CSI_CLKEN_OVERRIDE, 0, port_num >> 1);

        /* Clean up status */
        csi2_pp_write(port, TEGRA_CSI_PIXEL_PARSER_STATUS, 0xFFFFFFFF);
        csi2_cil_write(port, TEGRA_CSI_CIL_STATUS, 0xFFFFFFFF);
        csi2_cil_write(port, TEGRA_CSI_CILX_STATUS, 0xFFFFFFFF);

        csi2_cil_write(port, TEGRA_CSI_CIL_INTERRUPT_MASK, 0x0);

        /* CIL PHY registers setup */
        csi2_cil_write(port, TEGRA_CSI_CIL_PAD_CONFIG0, 0x0);
        csi2_cil_write(port, TEGRA_CSI_CIL_PHY_CONTROL,
                       BYPASS_LP_SEQ | 0xA);
#7

Hi. Thank you very much for your answer. I’m playing with sources under ‘drivers/media/platform/tegra’.
I just insert many of pr_info to see of what’s happening.
Please, don’t leave this thread. I will post new info here asap. Thank you.

#8

Hello!
I solved the problem about alternating blank/full frames and I got a series of full only raw bayer images using other settings(Width,Height,WordCount in csi regs).
Here is result:
http://i.imgur.com/bzZ7vOT.jpg

But now there is a problem with image resolution mismatch. Please, help me.

There are two types of resolution (in my case):
*) Capture image resolution = 1280x720@16bit (is what I want to set in capture command, is what I want to see at output)
*) CSI-2 frame resolution = 1312x729@10bit (it includes color processing margin, ignored OB data etc).
Please, see my sensor CSI-2 frame structure here:

In my case, there is only one correct set of values for csi-2(10bit).
Width=1312
WordCount=1640
Height=729

If I set WordCount other then 1640, the capture fails.
If I set Width or Height other then 1312x729, I see errors like “line is too short/long” or “frame height is too short/long” in ‘TEGRA_VI_CSI_ERROR_STATUS’ register
(Tegra X1 TRM name is 31.7.27 ‘VI_CSI_0_ERROR_STATUS_0’ page 2538).

There is only one place where the registers are set. It’s ‘drivers/media/platform/tegra/camera/vi2_fops.c’ function ‘vi2_channel_capture_setup’.
Here is we can see settings for ‘TEGRA_VI_CSI_IMAGE_SIZE’ (TRM 31.7.7 page 2534) and ‘TEGRA_VI_CSI_IMAGE_SIZE_WC’(TRM 31.7.8)

Just for testing I hardcoded these settings inside this func:

//inside 'vi2_channel_capture_setup()', just for test
    height=729;
    width=1312;
    word_count=1640;
  //word_count=1312*10/8=1312*5/4=1640
    vi2_channel_csi_write(chan, index, TEGRA_VI_CSI_IMAGE_SIZE_WC, word_count);
  //CSI frame size Width=1312, Height=729 (but capture frame size is 1280x720)
    vi2_channel_csi_write(chan, index, TEGRA_VI_CSI_IMAGE_SIZE, (height << IMAGE_SIZE_HEIGHT_OFFSET) | width);

So it works for CSI-2 and ‘TEGRA_VI_CSI_ERROR_STATUS’ has zero(non-error) status. And captured raw video has 1280x720 resolution with series of full frames output.
But csi-2 1312x729 resolution now conflicts with capture 1280x720 res.
Now I can see periodic memory errors, like this:

[  514.767963] [drivers/media/platform/tegra/camera/vi2_fops.c]: [vi2_channel_capture_frame]:
[  514.767968] [drivers/media/platform/tegra/camera/vi2_fops.c]: [vi2_channel_csi_write]:
[  514.800176] smmu_dump_pagetable(): fault_address=0x00000000803ce000 pa=0x0000000000000000 bytes=1000 #pte=619 in L2
...
[  514.801022] [drivers/media/platform/tegra/camera/vi2_fops.c]: [vi2_channel_capture_frame]:
[  514.801024] [drivers/media/platform/tegra/camera/vi2_fops.c]: [vi2_channel_csi_write]:
[  514.810699] mc-err: (18) csw_viw: EMEM decode error on PDE or PTE entry
[  514.817669] mc-err:   status = 0x60010072; addr = 0x803ce000
[  514.823386] mc-err:   secure: no, access-type: write, SMMU fault: nr-nw-s
[  514.833475] smmu_dump_pagetable(): fault_address=0x00000000805ce000 pa=0x0000000000000000 bytes=1000 #pte=924 in L2

I did not deal in detail with this, but I suppose that these errors are related to different image sizes (1280x720@16bit as request, 1313x729@10bit as fact).

How to I can resolve this image resolution conflict?
Can I set working(capture) area inside raw bayer area? (Is it possible to set left and top offset or setup color processing margin?)

Thanks in advance.

#9

Hi senchuss
Seems this sensor always output 1312x796, the solution should be get the 1312x796 and post process as what you want.

#10

Hello!
I have got successful capture for four modes without errors (using sensor_mode_id):

sensor_mode:
0: 1312x729 RawBayer(RG..GB..) 10bit -> 16bit(T_R16_I)
1: 1312x977 RawBayer(RG..GB..) 10bit -> 16bit(T_R16_I)
2: 1312x729 RawBayer(RG..GB..) 12bit -> 16bit(T_R16_I)
3: 1312x977 RawBayer(RG..GB..) 12bit -> 16bit(T_R16_I)

Various controls (gain, shutter_time etc) also work.
At this moment I use app-space C-program for capture data (using IOCTL).

I need to utilize ISP (for conversion raw_bayer to yuv) inside my C-program.

I can see ‘/dev/nvhost-isp’ and ‘/dev/nvhost-ctrl-isp’. If I try to open them then:

open(/dev/nvhost-isp) -> [drivers/video/tegra/host/isp/isp.c]: [nvhost_isp_t210_finalize_poweron]
enable_irq(tegra_isp->irq);

open(/dev/nvhost-ctrl-isp) -> [drivers/video/tegra/host/isp/isp.c]: [isp_open]

close(/dev/nvhost-ctrl-isp) -> [drivers/video/tegra/host/isp/isp.c]: [isp_release]

close(/dev/nvhost-isp) -> [drivers/video/tegra/host/isp/isp.c]: [nvhost_isp_t124_prepare_poweroff]
disable_irq(tegra_isp->irq);

(1) I can see ‘static int __init isp_init(void)’ but this func is not called. Where I need to setup this type of call?

(2) How to do ISP IOCTL? Which type of ctrl_class is used? Which type of dev is used for ioctl (nvhost-isp or nvhost-ctrl-isp) ?
If I try code (it previously work with camera ioctl)

extCtrls.ctrl_class = V4L2_CTRL_CLASS_CAMERA; // ????
extCtrls.error_idx = 0;
extCtrls.count = 1;
extCtrls.controls = &extCtrl;
extCtrl.id = NVHOST_ISP_IOCTL_GET_ISP_CLK; //from [KERNEL_SRC/include/linux/nvhost_isp_ioctl.h]
extCtrl.value = 0;
extCtrl.value64 = extCtrl.value;
//extCtrl.string = 
extCtrl.size = sizeof(extCtrl);
if (-1 == xioctl(fd_isp, VIDIOC_S_EXT_CTRLS, &extCtrls)) { //if error
  perror("NVHOST_ISP_IOCTL_GET_ISP_CLK");
  exit(EXIT_FAILURE);
}//if error
printf("ISP_CLK=%d\n",extCtrl.value); //???

I got perror
NVHOST_ISP_IOCTL_GET_ISP_CLK: Bad address
for both cases (where fd_isp is ‘/dev/nvhost-isp’ or ‘/dev/nvhost-ctrl-isp’).

(3) Can you post any C-code example of how to use ISP?

(4) According to Tegra TX1 TRM page 2533, I see 31.7.4 VI_CSI_0_IMAGE_DEF_0 .

bit 2: DEST_ISPB
bit 1: DEST_ISPA
bit 0: DEST_MEM

I can see [drivers/media/platform/tegra/camera/vi2_fops.c]: [vi2_channel_capture_frame_enable]
is set it only as DEST_MEM

vi2_channel_csi_write(chan, index,TEGRA_VI_CSI_IMAGE_DEF, val | IMAGE_DEF_DEST_MEM);

Did I need set both (DEST_MEM and DEST_ISPA) or DEST_ISPA only?

Thank you.

#11

Hi senchuss
TX1 ISP pipeline are not public you can’t use ISP for the format conversion. Current you can use gstreamer bayer2rgb element then VIC to YUV format. There’s a plan for the RAW from memory to ISP pipeline but not know which release will have it.

#12

Hi, Shane

I’m having same issue as senchuss had, where I get one valid image for every 4 frames and 3 blank images. I tried to adjust 0xA value you mentioned, it has some effect as whether my image# can match the sequence# I generated, however it doesn’t change the 3 blank images. I don’t see any error printed in the dmesg. Do you have any additional suggestions on how could I debug this issue?

thanks
Xiaoyong

#13

@xiaoyongtijee
If there’s no any message from the kernel message that means CSI/VI have done the capture normally.

#14

Thanks Shane, that was I suspect too, I’m looking into v4l2 buffering part of the driver, which looks like in channel.c and vi2_fops.c under drivers/media/platform/tegra/camera. Trying to understand why 3 out of 4 of them are blank. Are there any other loggings I can enable?

thanks

#15

While we R&D our Custom Boards with many different IMXxxx sensors we eiter encounter these V4L interface problems:

  1. The alternation of empty(zero) and normal(full) frames is observed.

  2. ‘Mipi cal timeout’ and ‘MW_ACK_DONE syncpoint time out’ are single warnings at start of the stream.
    And ISP pipeline interface problems:

  3. No any data from ISP, and random kernel opps when ISP pipeline consumer application killed by SIGKILL.

  4. ISP supporting code logic completely lock kernel in case of timeout MIPI CSI EOF message.

  5. ISP and V4L sources does not work simultaniusly, V4L interface for data capture stop working after temporary usage of ISP interface for capturing data.

    1. was resolved by
  • correct configuration of "line_length" in video sensor driver
  • increasing delay between video lines data send via CSI
  • correct settings for min and max FPS
  1. was resolved by observing real CSI data parameters by oscilloscope and then
  • correct configuration of "line_length" in DTS file
  • correct settings for pix_clk_hz in DTS file
  • correct settings for min and max FPS in DTS file
  • correct settings for mclk_khz and mclk_multiplier (which used for ISP clock) in DTS file

Example:

mode0 {				/* mode IMX250_MODE_2048X1552_12BIT */
				num_lanes	  = "4";
				tegra_sinterface  = "serial_a";
				dpcm_enable       = "false";
				cil_settletime    = "0";
 
 				dynamic_pixel_bit_depth = "12";
				pixel_t = "bayer_grbg12";	/* 12bit GRGB BAYER */
 
				line_length = "2680";		/*     13'400 ns */
				active_w    = "2048";		/*     10'300 ns */
				active_h    = "1552";		/* 20'796'800 ns */
 
 				readout_orientation = "0";
 
 				inherent_gain = "1";
 
				/* Sync Params */
 				discontinuous_clk = "no";
				mclk_khz        = "50000";	/* CSI_MCLK out frequency (not used in our design) */
				mclk_multiplier = "5";		/* multiplayer for ISP clock */
				pix_clk_hz      = "200000000";	/* 200MHz Sensor Pixel clock */
 
				min_framerate = "1.0";
				max_framerate = "35.604";	/* 2097 lines 28'099'800 ns */
				min_exp_time  = "30";
				max_exp_time  = "200000";
...
  1. was solved by generating fake EOF by FPGA if it missed due to glitches or occasional sensor board disconnetction.
#16

Thanks F1inx, this is a good summary of what we experienced.