Jetson Orin Nano - CSI/VI Timeout in Raw12 mode (works in raw10)

Autonomous Machines Jetson & Embedded Systems Jetson Orin Nano
1

Hello all,

I’m currently facing an issue on my Jetson Orin Nano dev kit (JetPack 6.1/ L4T 36.4) trying to capture a MIPI CSI stream from 2 FPGA-based cameras. So far, I have modified original IMX477 sensor driver to have four main modes : SXGA and FHD, each declined in raw10 and raw12 pixel formats. The FPGAs emulate two IMX477 cameras and change formats with the same i2c register adresses, sending a test pattern for the CSI to read. The software changes have been made in imx477_modes_tbls.h and the DT, with some additionnal printk statements in the driver itself.

Everything looks good while capturing in both raw10 modes using v4l2-ctl (and gstreamer). But I have timeouts errors for the raw12 modes with v4l2-ctl in bypass mode. See below :

v4l2-ctl -d /dev/video0 --set-fmt-video=width=1280,height=1024,pixelformat=RG12 --stream-mmap --verbose --set-ctrl sensor_mode=2,bypass_mode=0 
VIDIOC_QUERYCAP: ok
VIDIOC_S_EXT_CTRLS: ok
VIDIOC_G_FMT: ok
VIDIOC_S_FMT: ok
Format Video Capture:
	Width/Height      : 1280/1024
	Pixel Format      : 'RG12' (12-bit Bayer RGRG/GBGB)
	Field             : None
	Bytes per Line    : 2560
	Size Image        : 2621440
	Colorspace        : sRGB
	Transfer Function : Default (maps to sRGB)
	YCbCr/HSV Encoding: Default (maps to ITU-R 601)
	Quantization      : Default (maps to Full Range)
	Flags             : 
		VIDIOC_REQBUFS returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_STREAMON returned 0 (Success)
cap dqbuf: 0 seq:      0 bytesused: 2621440 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 0 seq:      0 bytesused: 2621440 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 1 seq:      1 bytesused: 2621440 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 2 seq:      2 bytesused: 2621440 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 3 seq:      3 bytesused: 2621440 ts: 0.000000 (error, ts-monotonic, ts-src-eof)

Looking further I followed these topics : https://forums.developer.nvidia.com/t/how-to-get-vi-and-csi-traces-in-jetson-nano/161648 and https://docs.nvidia.com/drive/drive-os-5.2.0.0L/drive-os/index.html#page/DRIVE_OS_Linux_SDK_Development_Guide/NvMedia/nvmedia_debugging_csi_errors.html
to see and figure out what was happening at VI and CSI level. Trace log looks like this :

<...>-9397    [001] .......  4174.687536: tegra_channel_open: vi-output, imx477 9-001a
        v4l2-ctl-9397    [000] .......  4174.703956: tegra_channel_set_power: imx477 9-001a : 0x1
        v4l2-ctl-9397    [000] .......  4174.704081: camera_common_s_power: status : 0x1
        v4l2-ctl-9397    [000] .......  4175.004458: tegra_channel_set_power: 13e00000.host1x:nvcsi@15a00000- : 0x1
        v4l2-ctl-9397    [000] .......  4175.004476: csi_s_power: enable : 0x1
        v4l2-ctl-9397    [000] .......  4175.005042: tegra_channel_capture_setup: vnc_id 0 W 1280 H 1024 fmt c4
 vi-output, imx4-9398    [000] .......  4175.006779: vi_task_submit: class_id:48 ch:0 syncpt_id:26 syncpt_thresh:522 pid:9398 tid:9398
 vi-output, imx4-9398    [000] .......  4175.006796: vi_task_submit: class_id:48 ch:0 syncpt_id:26 syncpt_thresh:522 pid:9398 tid:9398
 vi-output, imx4-9398    [000] .......  4175.006797: vi_task_submit: class_id:48 ch:0 syncpt_id:26 syncpt_thresh:522 pid:9398 tid:9398
 vi-output, imx4-9398    [000] .......  4175.006798: vi_task_submit: class_id:48 ch:0 syncpt_id:26 syncpt_thresh:522 pid:9398 tid:9398
        v4l2-ctl-9397    [001] .......  4175.006831: tegra_channel_set_stream: enable : 0x1
        v4l2-ctl-9397    [001] .......  4175.011272: tegra_channel_set_stream: 13e00000.host1x:nvcsi@15a00000- : 0x1
        v4l2-ctl-9397    [001] .......  4175.011281: csi_s_stream: enable : 0x1
        v4l2-ctl-9397    [001] .......  4175.012085: tegra_channel_set_stream: imx477 9-001a : 0x1
     kworker/2:2-8370    [002] .......  4175.027425: rtcpu_vinotify_event: tstamp:131667852423 cch:0 vi:1 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:4213354453152 data:0x799e300010000000
     kworker/2:2-8370    [002] .......  4175.027433: rtcpu_vinotify_event: tstamp:131667852683 cch:0 vi:1 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:4213354510112 data:0x0000000031000001
     kworker/2:2-8370    [002] .......  4175.027434: rtcpu_vinotify_event: tstamp:131667852969 cch:0 vi:1 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:4213354528224 data:0x799e2d0010000000
     kworker/2:2-8370    [002] .......  4175.027435: rtcpu_vinotify_event: tstamp:131667853217 cch:0 vi:1 tag:VIFALC_ACTIONLST channel:0x23 frame:0 vi_tstamp:4213354548544 data:0x0000000007020001
     kworker/2:2-8370    [002] .......  4175.027435: rtcpu_vinotify_event: tstamp:131667853558 cch:0 vi:1 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:4213354590912 data:0x0000000031000002
 vi-output, imx4-9399    [002] .......  4177.540462: tegra_channel_capture_setup: vnc_id 0 W 1280 H 1024 fmt c4
 vi-output, imx4-9398    [000] .......  4177.540825: vi_task_submit: class_id:48 ch:0 syncpt_id:26 syncpt_thresh:522 pid:9398 tid:9398
 vi-output, imx4-9398    [000] .......  4177.540845: vi_task_submit: class_id:48 ch:0 syncpt_id:26 syncpt_thresh:522 pid:9398 tid:9398
 vi-output, imx4-9398    [000] .......  4177.540849: vi_task_submit: class_id:48 ch:0 syncpt_id:26 syncpt_thresh:522 pid:9398 tid:9398
 vi-output, imx4-9398    [000] .......  4177.540852: vi_task_submit: class_id:48 ch:0 syncpt_id:26 syncpt_thresh:522 pid:9398 tid:9398
     kworker/2:2-8370    [002] .......  4177.551339: rtcpu_vinotify_event: tstamp:131746522436 cch:0 vi:1 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:4215888456032 data:0x799e300010000000
     kworker/2:2-8370    [002] .......  4177.551351: rtcpu_vinotify_event: tstamp:131746522729 cch:0 vi:1 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:4215888533504 data:0x0000000031000001
     kworker/2:2-8370    [002] .......  4177.551352: rtcpu_vinotify_event: tstamp:131746522982 cch:0 vi:1 tag:VIFALC_ACTIONLST channel:0x23 frame:0 vi_tstamp:4215888555168 data:0x0000000007020001
     kworker/2:2-8370    [002] .......  4177.607323: rtcpu_vinotify_event: tstamp:131747063292 cch:0 vi:1 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:4215888748256 data:0x799e2d0010000000
     kworker/2:2-8370    [002] .......  4177.607337: rtcpu_vinotify_event: tstamp:131747063542 cch:0 vi:1 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:4215888791008 data:0x0000000031000002
[...]

Activating dev_dbg() on csi5_fops.c and vi5_fops.c did not show me much additionnal info in dmesg, as it seems FS and FE packets are not recognized by CSI :

imx477 power on
[  +0,300175] t194-nvcsi 13e00000.host1x:nvcsi@15a00000: csi5_power_on
[  +0,016461] t194-nvcsi 13e00000.host1x:nvcsi@15a00000: csi5_start_streaming: csi_pt=2, st_id=2, vc_id=0, pg_mode=0x0
[  +0,000015] t194-nvcsi 13e00000.host1x:nvcsi@15a00000: csi5_stream_set_config: stream_id=2, csi_port=2
[  +0,000003] t194-nvcsi 13e00000.host1x:nvcsi@15a00000: cil_settingtime is pulled from device
[  +0,000574] imx477: writing table
[  +0,053764] imx477: Setting mode to mode2
[  +0,000009] imx477: writing table
[  +0,017859] imx477: start streaming
[  +0,000007] imx477: writing table
[  +2,499283] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  +0,000032] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  +0,000952] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  +2,558914] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  +0,000031] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  +0,000949] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  +2,558930] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  +0,000026] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  +0,001148] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel 
[...]
[  +0,000078] imx477: stop streaming
[  +0,000002] imx477: writing table
[  +0,000195] t194-nvcsi 13e00000.host1x:nvcsi@15a00000: csi5_stop_streaming: csi_pt=2, st_id=2, vc_id=0, pg_mode=0x0
[  +0,003336] imx477 power off
[  +0,000056] t194-nvcsi 13e00000.host1x:nvcsi@15a00000: csi5_power_off

I have read the CSI Debug Counters as well using devmem, particularly events 0-8. The counters did not increase value either in raw12 capture, but did in raw10 capture. I ended up reading registers indicated by TRM, namely NVCSI_STREAM_X_ERROR_STATUS2VI_VCY_0 and NVCSI_STREAM_X_INTR_STATUS_VCY_0 where X and Y are indicated by kernel debug logs. The values read are always 0.

So my main questions are :

  • Is there another way to debug this kind of issue ? Using an oscilloscope I could confirm there were FS, Frames and FS packets in the electric signal (which was about ~20% larger), so I should be able to see something at register level. I am yet to find a trace of data handled (or at least discarded) by NVCSI.
  • Could this silent error be caused by base timeout value, or any kind of specific raw12 requirements in the DT or driver ?
  • Am I missing an important detail here that renders me unable to handle data ? If I understood correctly, bypassing the ISP should make V4L2 handle raw12 signal and it is compatible.

I really would appreciate guidance on this, as I am very new to camera dev and am quite out of clues right now. Thanks a lot.
Best regards.

notes_nv.txt (17.2 KB)

3

Does your sensor driver support raw10 and raw12 in the same driver?

v4l2-ctl --list-formats-ext

4

I believe it does :

$ v4l2-ctl --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture

	[0]: 'RG10' (10-bit Bayer RGRG/GBGB)
		Size: Discrete 1920x1080
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x1024
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x1024
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.017s (60.000 fps)
	[1]: 'RG12' (12-bit Bayer RGRG/GBGB)
		Size: Discrete 1920x1080
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x1024
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x1024
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.017s (60.000 fps)

However it there are 4 modes per pixel format. I never paid attention as to why, so maybe it’s causing my problem ? As I said I didn’t change much in driver code.

5

Could you remove raw10 only report raw12 to verify.

Thanks

6

Thank you for your response, I have made appropriate changes to DT and driver to only support raw12

v4l2-ctl --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture

	[0]: 'RG12' (12-bit Bayer RGRG/GBGB)
		Size: Discrete 1280x1024
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.017s (60.000 fps)

Tried again the capture w mode 0 being previous mode 2 :

$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=1280,height=1024,pixelformat=RG12 --stream-mmap --verbose --set-ctrl sensor_mode=0,bypass_mode=0 
VIDIOC_QUERYCAP: ok
VIDIOC_S_EXT_CTRLS: ok
VIDIOC_G_FMT: ok
VIDIOC_S_FMT: failed: Invalid argument
		VIDIOC_REQBUFS returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_STREAMON returned -1 (Invalid argument)

Does this error indicate that the Raw12 format is not supported by driver ?

7

Any error message from the “sudo dmesg”