8GB Jetson Orin Nano with two IMX519 cameras

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin Nano
1

Hi folks,

I am doing a project which involves two IMX519 connected directly to a Jetson Orin Nano Super.

  • JP 6.2.1 - L4T 36.4.4.
  • CAM0 is ok (attached CAM0_trace.txt)
  • CAM1 (attached CAM1_trace.txt) → no valid stream of data coming from the camera.
  • Both CAM tested individually on CAM0 and worked.
  • Attempted to modify pix_clk_hz, and cil_settletime (attached IMX519 DTS). None worked.

I have turned on clock_boost and VI tracing. Obtained the opcode 0x00000088, which is an LP sequence error. I have no way to probe this, so I do not know what to fix in terms of timing.

Any help is extremely appreciated !

  • Command used:
    ++ CAM0: v4l2-ctl -d /dev/video1 --set-fmt-video=width=1920,height=1080,pixelformat=‘RG10’ --stream-mmap --stream-count=60 --set-ctrl=sensor_mode=2

++ CAM1: v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=‘RG10’ --stream-mmap --stream-count=60 --set-ctrl=sensor_mode=2

echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrg_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate

modprobe rtcpu_debug
echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 2 > /sys/kernel/debug/camrtc/log-level
echo > /sys/kernel/debug/tracing/trace

[  797.853856] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  797.853879] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  797.854487] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel

rtcpu_nvcsi_intr: tstamp:19536355146 class:GLOBAL type:PHY_INTR0 phy:1 cil:0 st:0 vc:0 status:0x00000088

rtcpu_nvcsi_intr: tstamp:19536355146 class:CORRECTABLE_ERR type:PHY_INTR phy:1 cil:0 st:0 vc:0 status:0x00000188

rtcpu_vinotify_event: tstamp:19536555362 cch:0 vi:1 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:625164788192 data:0x799e300010000000

rtcpu_vinotify_event: tstamp:19536555593 cch:0 vi:1 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:625164845120 data:0x0000000031000001

rtcpu_vinotify_event: tstamp:19536555840 cch:0 vi:1 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:625164863296 data:0x799e2d0010000000

rtcpu_vinotify_event: tstamp:19536556056 cch:0 vi:1 tag:VIFALC_ACTIONLST channel:0x23 frame:0 vi_tstamp:625164883648 data:0x0000000007020001

rtcpu_vinotify_event: tstamp:19536556297 cch:0 vi:1 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:625164926016 data:0x0000000031000002

rtcpu_vinotify_event: tstamp:19615223543 cch:0 vi:1 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:627672097856 data:0x799e300010000000

rtcpu_vinotify_event: tstamp:19615223797 cch:0 vi:1 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:627672154816 data:0x0000000031000001

rtcpu_vinotify_event: tstamp:19615224016 cch:0 vi:1 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:627672172960 data:0x799e2d0010000000

rtcpu_vinotify_event: tstamp:19615224256 cch:0 vi:1 tag:VIFALC_ACTIONLST channel:0x23 frame:0 vi_tstamp:627672193344 data:0x0000000007020001

rtcpu_vinotify_event: tstamp:19615224470 cch:0 vi:1 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:627672238784 data:0x0000000031000002

log.txt (2.4 KB)

cam1_trace.txt (2.9 KB)

cam0_trace.txt (207.7 KB)

IMX519.txt (17.3 KB)

5

Hi folks, Any suggestions ?

6

hello ikrat0s69,

CAM0 slot now streams on CSI-B. CAM1 slot now stream-on CSI-C/D.
please also note that, CSI0 D1 and CSI1 D0 P/N will always been swizzled for P/N. you may use device tree property, lane_polarity to configure a polarity swap on any lane.

7

Hi @JerryChang ,

Thank you for your response.

If I understand correctly for JP 6.2, then lane_polarity for CAM0 should be 6, while lane_polarity for CAM1 should be 0. It is reflected in the attached DTS. Also serial_b for CAM0 and serial_c for CAM1, which should be ok right?

8

hello ikrat0s69,

please also check $ v4l2-ctl -d /dev/video0 --list-formats-ext for available sensor modes.
you may also testing with the 1st mode, 4656x3496.

besides..
let’s also check sensor stream through nvarguscamerasrc plugin with preview disabled and shows frame-rate only.
for instance,
$ gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=4656, height=3496, framerate=30/1, format=NV12' ! nvvidconv ! fpsdisplaysink text-overlay=0 name=sink_0 video-sink=fakesink sync=0 -v

9

Hi @JerryChang ,

Thank you for your prompt reply.

The list format is fine. List formats log:

gunna@gunna-desktop:~$ v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture

	[0]: 'RG10' (10-bit Bayer RGRG/GBGB)
		Size: Discrete 4656x3496
			Interval: Discrete 0.111s (9.000 fps)
		Size: Discrete 3840x2160
			Interval: Discrete 0.059s (17.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.008s (120.000 fps)

gunna@gunna-desktop:~$ v4l2-ctl -d /dev/video1 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture

	[0]: 'RG10' (10-bit Bayer RGRG/GBGB)
		Size: Discrete 4656x3496
			Interval: Discrete 0.111s (9.000 fps)
		Size: Discrete 3840x2160
			Interval: Discrete 0.059s (17.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.008s (120.000 fps)

I have attempted your suggested command through nvarguscamerasrc:

CAM0 works.

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 0 
   Output Stream W = 4656 H = 3496 
   seconds to Run    = 0 
   Frame Rate = 9.000000 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
Redistribute latency...
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0/GstFakeSink:fakesink0: sync = false
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 7, dropped: 0, current: 12.53, average: 12.53
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 12, dropped: 0, current: 9.03, average: 10.79
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 17, dropped: 0, current: 9.00, average: 10.19
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 22, dropped: 0, current: 9.02, average: 9.90
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 27, dropped: 0, current: 9.01, average: 9.72
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 32, dropped: 0, current: 9.01, average: 9.60
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0: last-message = rendered: 37, dropped: 0, current: 8.96, average: 9.51
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:05.390106028
Setting pipeline to NULL ...
GST_ARGUS: Cleaning up
CONSUMER: Done Success
GST_ARGUS: Done Success
Freeing pipeline ...

CAM1 does not work.

GST_ARGUS: Running with following settings:
   Camera index = 1 
   Camera mode  = 0 
   Output Stream W = 4656 H = 3496 
   seconds to Run    = 0 
   Frame Rate = 9.000000 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadExecute:734 NvBufSurfaceFromFd Failed.
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadFunction:245 (propagating)
Redistribute latency...
/GstPipeline:pipeline0/GstFPSDisplaySink:sink_0/GstFakeSink:fakesink0: sync = false
Got EOS from element "pipeline0".
Execution ended after 0:00:04.012520447
Setting pipeline to NULL ...
GST_ARGUS: Cleaning up
GST_ARGUS: Done Success
Freeing pipeline ...

As observed, I am getting NvBufSurfaceFromFd , which means there is no valid stream of pixels coming in.

Trace log indicates

rtcpu_string: tstamp:27131234763 id:0x04010000 str:"VM0 activating."
rtcpu_nvcsi_intr: tstamp:27148179997 class:GLOBAL type:PHY_INTR0 phy:1 cil:0 st:0 vc:0 status:0x00000088
rtcpu_nvcsi_intr: tstamp:27148179997 class:CORRECTABLE_ERR type:PHY_INTR phy:1 cil:0 st:0 vc:0 status:0x00000088
rtcpu_vinotify_event: tstamp:27149926087 cch:1 vi:1 tag:VIFALC_TDSTATE channel:0x0b frame:0 vi_tstamp:868796604128 data:0x799da00010000000
rtcpu_vinotify_event: tstamp:27149926331 cch:1 vi:1 tag:VIFALC_TDSTATE channel:0x0b frame:0 vi_tstamp:868796613792 data:0x0000000031000001
rtcpu_vinotify_event: tstamp:27149926575 cch:1 vi:1 tag:VIFALC_ACTIONLST channel:0x0b frame:0 vi_tstamp:868796618432 data:0x0000000007020001
rtcpu_vinotify_event: tstamp:27150607298 cch:1 vi:1 tag:VIFALC_TDSTATE channel:0x0b frame:0 vi_tstamp:868814332864 data:0x799d9d0010000000
rtcpu_vinotify_event: tstamp:27150607551 cch:1 vi:1 tag:VIFALC_TDSTATE channel:0x0b frame:0 vi_tstamp:868814342656 data:0x0000000031000002
rtcpu_isp_falcon_task_start: tstamp:1528048078 ch:0 task:HANDLE_EVENT
rtcpu_isp_falcon_task_end: tstamp:1528048127 task:HANDLE_EVENT
rtcpu_string: tstamp:27493087468 id:0x04010000 str:"VM0 deactivating."

Which means LP error sequence.

dmesg log

tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 1, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 2, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 3, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 4, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 5, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 6, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 7, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 8, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 9, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 10, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 11, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 12, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 13, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 14, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 15, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 16, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 17, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 18, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 19, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 20, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 21, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 22, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 23, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 24, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 25, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 26, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 27, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 28, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 29, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 30, flags: 0, err_data 131072
...
10

hello ikrat0s69,

normally, it should follow by LP11->LP01->LP00->LP11 sequence.
did you tried to swap the camera hardware to confirm it’s not a hardware issue?

there’re discarding frame warnings according to your dmesg logs,
let’s give it a try to revise VI driver to ignore frame_err,
for instance,

static void vi5_capture_dequeue(...)
{
...
                               dev_warn(vi->dev,
                                        "corr_err: discarding frame %d, flags: %d, "
                                        "err_data %d\n",
                                        descr->status.frame_id, descr->status.flags,
                                        descr->status.err_data);
                                // frame_err = true;
11

Hi @JerryChang ,

I have attempted to swap the camera hardware back and forth many times. The CAM0 always works, meaning the camera hardware itself is fine.

I followed the instructions that you have provided regarding ignoring frame_err. It does not output anything regarding discarding frame anymore for dmesg. The rest of the output is identical to what I reported previously.

If possible, would you mind sharing how the protocol works through I2C ? I understand from PHY perspective but I do not understand about how the CSI controller interacts with Jetson RX through I2C. It would be best if there is a public documentation where I can make sense out of it.

12

hello ikrat0s69,

please check Sensor Pixel Clock section to review your pix_clk_hz settings, which must be set correctly to avoid potential issues.
just an FYI, I’m usually using sensor CSI lane output rate to obtain the value.
i.e. pix_clk_hz = sensor data rate per lane (Mbps) * number of lanes / bits per pixel

besides.. you may also check developer guide to tune THS settle time of the MIPI lane, cil_settletime for checking.

13

Hi @JerryChang ,

Thank you for your reply.

CAM0 works, meaning the math should work in theory…

Please follow my math below:

I did a quick math for my pix_clk_hz. Given 4656x3496 with guaranteed 9 fps. It results in pix_clk_hz ~ 150 MHz.

My current pix_clk_hz = 300 MHz, which is doubled and I think it should be ok. At 1.5 Gbps per link, I don’t think deskew should happen. What do you think?

If pix_clk_hz = 300 MHz, then per lane UI ~ 0.67. According to MIPI spec, then I should tune the cil_settletime to be in between 89 ns and 152 ns.

Assume lp_clock_period is 1/(408 MHz), my acceptable range for cil_settletime is from 14-22. I have attempted to extend cil_settletime to 20, so RX should have enough time to switch to HS.

However, It still does not work. I now get HS-Sync alignment between links (0x00000040)error for both CAM0 and on CAM1.

Edit:

I made a mistake in the calculation for cil_settletime , should be 37 to 61 for cil_settletime. And I get LP error sequence for even when max settle time, which is 61.

14

actually not, pix_clk_hz it should close to the sensor output date-rate.

15

Hi @JerryChang ,

I have attempted to fix pix_clk_hz to 170 MHz and cil_settletime to 51. It still does not work for CAM1. CAM0 is working just fine with the modifications.

16

hello ikrat0s69,

just curious, do you have other Jetson Orin Nano platform to have cross validation?

17

Hi @JerryChang ,

We do not have an extra Jetson Orin Nano for cross validation unfortunately. We would have done that if we had an extra.

18

hello ikrat0s69,

let me double check your device tree settings,
please dump the system settings as below, and please attach the dts file for reference.
for instance,
$ sudo dtc -I fs -O dts /sys/firmware/devicetree/base > /tmp/123.dts

19

Hi @JerryChang ,

Here is the system device tree settings, for some reason I can’t upload dts. So I modified to txt file.

For camera dts file, please consult the first post, at IMX519.txt

123.txt (325.3 KB)

20

hello ikrat0s69,

it may be minor, but it should have status property within sensor node.
for instance,

			i2c@1 {
				arducam_imx519_c@1a {
					ports {
							endpoint {
								bus-width = <0x02>;
								remote-endpoint = <0x380>;
								port-index = <0x02>;
								phandle = <0x37d>;
+								status = "okay";
							};
21

Hi @JerryChang ,

I have tried your suggestions as well, it does not work.

We have moved on with other design option choice and no longer need CAM1 anymore.

I am extremely appreciated for your help. Thank you !!!

23

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.