AGX Orin CSI E (Port 4) support

elitewompa · January 16, 2025, 7:28pm

Brief System Description

JetPack 5.1.4 L4T 35.6.0
NVidia Jetson AGX Orin Module 64GB - Non Industrial Version
Orin is connected to 3 MAX96712 deserializers
Each deserializer is connected to Orin via CSI 4 lanes
All cameras are identical
Cameras 0-7 work fine
Cameras 8-11 DO NOT WORK

To simplify this problem we have:

disabled deserializers 1 and 2
disabled all cameras except camera 8

The camera/csi/vi/module related portion of the device tree used is here
camDevTree.txt (5.6 KB)

Dmesg output on boot is here
dmesgBoot.txt (71.9 KB)

We enabled debug trace with the following command

echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate

echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 2 > /sys/kernel/debug/camrtc/log-level
echo 1 > /sys/kernel/debug/tracing/events/camera_common/enable
cat /sys/kernel/debug/tracing/trace

We used the following command to attempt to acquire an image:
gst-launch-1.0 v4l2src device=/dev/gmsl/cam0 num-buffers=1 ! jpegenc ! filesink location=/home/root/cam0.jpg

Dmesg output after image acquire attempt is here:
debugDmesg.txt (886 Bytes)

We captured debug trace information with the following command

cat /sys/kernel/debug/tracing/trace

Debug trace output after image acquire attempt is here:
debugTrace.txt (1.0 MB)

All of our MIPI clock and data pairs are matched length for each of the 3 deserializers.
The table below shows the length differences as well as the corresponding time delay for the length difference.

One can see that the delay is less than 1 picosecond for any of the buses .
So unless delay added in the Orin module, the signals should be arriving virtualy simultaneously.
Note that the buses for the derserializers that work are MIPI-0 and MIPI-1the bus for the deserializer that does not work is MIPI-2.

Could you suggest a next step in debugging what is wrong.

carolyuu · January 16, 2025, 7:30pm

Hi,

For the camera basic functionality first needs to check the device and driver configuration.
You can reference to below program guide for the detailed information of device tree and driver implementation.
https://docs.nvidia.com/jetson/archives/r36.3/DeveloperGuide/SD/CameraDevelopment/SensorSoftwareDriverProgramming.html?highlight=programing#sensor-software-driver-programming

Please refer to Applications Using V4L2 IOCTL Directly by using V4L2 IOCTL to verify basic camera functionality.
https://docs.nvidia.com/jetson/archives/r36.3/DeveloperGuide/SD/CameraDevelopment/SensorSoftwareDriverProgramming.html?highlight=programing#to-run-a-v4l2-ctl-test

Once confirm the configure and still failed below link help to get log and some information and some tips for debug.
https://elinux.org/Jetson/l4t/Camera_BringUp#Steps_to_enable_more_debug_messages

Thanks!

elitewompa · January 16, 2025, 7:39pm

Camera basic functionality has already been verified (see above … wait I will copy it back down here)

All cameras are identical
Cameras 0-7 work fine
Cameras 8-11 DO NOT WORK

We are familiar with capturing trace debug information. If you read our original post we already captured that information and included it in the original post.

elitewompa · January 16, 2025, 8:42pm

From the debugDmesg.txt above looks like there are a number of start frame faults

     kworker/7:2-148     [007] ....  1371.999910: rtcpu_vinotify_error: tstamp:43546674109 cch:0 vi:0 tag:CSIMUX_FRAME channel:0x00 frame:0 vi_tstamp:1393493517984 data:0x00000000000000a4
     kworker/7:2-148     [007] ....  1372.000003: rtcpu_vinotify_event: tstamp:43546970053 cch:0 vi:0 tag:CSIMUX_FRAME channel:0x00 frame:0 vi_tstamp:1393493517984 data:0x00000000000000a4
     kworker/7:2-148     [007] ....  1372.000195: rtcpu_vinotify_error: tstamp:43547715711 cch:0 vi:0 tag:CSIMUX_FRAME channel:0x00 frame:0 vi_tstamp:1393526851776 data:0x00000000000000a4
     kworker/7:2-148     [007] ....  1372.000196: rtcpu_vinotify_event: tstamp:43547716856 cch:0 vi:0 tag:CSIMUX_FRAME channel:0x00 frame:0 vi_tstamp:1393526851776 data:0x00000000000000a4
     kworker/7:2-148     [007] ....  1372.060206: rtcpu_vinotify_error: tstamp:43548756936 cch:0 vi:0 tag:CSIMUX_FRAME channel:0x00 frame:0 vi_tstamp:1393560185568 data:0x00000000000000a4
     kworker/7:2-148     [007] ....  1372.060408: rtcpu_vinotify_event: tstamp:43549065852 cch:0 vi:0 tag:CSIMUX_FRAME channel:0x00 frame:0 vi_tstamp:1393560185568 data:0x00000000000000a4
     kworker/7:2-148     [007] ....  1372.060811: rtcpu_vinotify_error: tstamp:43549799060 cch:0 vi:0 tag:CSIMUX_FRAME channel:0x00 frame:0 vi_tstamp:1393593519360 data:0x00000000000000a4
     kworker/7:2-148     [007] ....  1372.115853: rtcpu_vinotify_event: tstamp:43550079301 cch:-1 vi:0 tag:CSIMUX_FRAME channel:0x00 frame:0 vi_tstamp:1393593519360 data:0x00000000000000a4

based on the decoding referenced above.

JerryChang · January 17, 2025, 1:57am

hello elitewompa,

according to the messages…

[ 1371.911432] ov10635 0-0009: Write sensor in addr=0x9 reg=0x100 val=0x1
[ 1371.952430] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 131072
[ 1371.985611] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 131072
[ 1372.052482] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 131072

you may see-also Topic 318537 for reference to debug discarding frame corr_err messages

besides,
please also give it another try with below commands to boost all the clocks for testing

sudo su
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate

elitewompa · January 17, 2025, 7:35pm

Hello @JerryChang,

The original post contained result logs from a debug capture with clocks maximized. (See original post, data also included there)

However it will not hurt to run it again.

We ran the following commands:

echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate

echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 2 > /sys/kernel/debug/camrtc/log-level
echo 1 > /sys/kernel/debug/tracing/events/camera_common/enable
cat /sys/kernel/debug/tracing/trace

gst-launch-1.0 v4l2src device=/dev/gmsl/cam0 num-buffers=1 ! jpegenc ! filesink location=/home/root/cam0.jpg

cat /sys/kernel/debug/tracing/trace > maxClockDebugTrace.txt
dmesg > debugDmesg.txt

Results are here:
debugDmesg-01172025.txt (72.8 KB)
debugTrace-01172025.txt (1.0 MB)

We observed the same output found in debugDmesg.txt

[ 2424.616559] ov10635 0-0009: Setting stream to 1
[ 2424.617699] ov10635 0-0009: Write sensor in addr=0x9 reg=0x100 val=0x1
[ 2424.654855] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 131072
[ 2424.688153] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 131072
[ 2424.754872] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 131072

We will now take some time to look at Topic 318537 as a reference to debug discarding frame corr_err messages.

err_data 131072 is equivalent to CAPTURE_STATUS_NOTIFY_BIT_CSIMUX_FRAME_CSI_FAULT_PD_CRC_ERR MK_BIT64(17) which means /** CSI CRC Error in payload data */.

elitewompa · January 22, 2025, 5:37pm

Hello @JerryChang,

As we review the documentation for Device Properties we have a question.

The documentation states that the sensor device properties clocks, clock-names, and mclk are used to identify the MIPI clock that is driven from the Tegra SOC.

Does this mean that the mclk in this scenario is used as a reference clock by the camera to generate the camera’s output MIPI stream?
Since our system has a GMSL link in between the camera and the Tegra SOC we do not need any of these properties, sensor device properties clocks, clock-names, and mclk. Is this understanding correct?
The sensor node V4l2 name value pair mclk_khz should be set to the MIPI clock frequency of the deserializer output connected to the Tegra SOC CSI port … is this understanding correct?

JerryChang · January 23, 2025, 2:41am

hello elitewompa,

>> Q1
mclk is the input clock for the device;
by default the value is extperiph1, with the maximum frequency 37.09MHz.

>> Q2
but… you’ll need to define any required hardware resources needed by driver.
you may see-also reference driver, $public_sources/kernel_src/hardware/nvidia/platform/t19x/galen/kernel-dts/common/tegra194-p2822-0000-camera-imx390-a00.dtsi

>> Q3
in general, we’ll still setting a standard MIPI driving clock frequency, i.e. 24000 in sensor device tree.
please refer to reference driver as well.
for instance, $public_sources/kernel_src/hardware/nvidia/platform/t19x/common/kernel-dts/t19x-common-modules/tegra194-camera-imx390-a00.dtsi

elitewompa · February 4, 2025, 8:55pm

Hello @JerryChang,

We want to provide a status update and ask a few questions.

Earlier you referred us to [Topic 318537](Understanding err_data 0 in tegra-camrtc-capture-vi: corr_err Logs - #4 by JerryChang)

We have reviewed settings for Sensor Pixel Clock
We computed the value using frame size and frame rate: 1312 x 814 x 30fps = 32039040

1. Does our calculated value for pix_clk_hz look correct?

Since our system uses GMSL we also reviewed settings for SerDes Pixel Clock
Our deserializer output bitrate is 800 MHz
serdes_pix_clk_hz = (800,000,000) * (4) / (16) = 200000000

2. Does our calculated value for serdes_pix_clk_hz look correct?

One device tree setting of note that we changed was cil_settletime.
After looking at Topic 60565 and Topic 283225 we understood that the ui term wants the duration for a single bit period on the MIPI bus. Following are our calculations to determine the value for cil_settletime.

// cil_settletime: The settle time of the MIPI lane, in nanoseconds
// 85 ns + 6×ui < (cil_settletime+6) × (lp_clock_period) < 145 ns + 10×ui
// lp_clock_period is 1/(204 MHz)
// ui is the unit interval, the bit period based on the MIPI bus clock rate (800MHz)
// ui = 1/800,000,000 = 0.000000001 s = 1 ns
// 85 ns + 6×1 < (cil_settletime+6) × (lp_clock_period) < 145 ns + 10×1
// 91 ns < (cil_settletime+6) × (lp_clock_period) < 155 ns
// 91 ns < (cil_settletime+6) × (4.902 ns) < 155 ns
// cil_settletime [12.87 ns, 26.13 ns]
// we choose average = 19.5 ns > round up to 20 ns
//cil_settletime = "20";  <-- Did not work for Good Cam
cil_settletime = "0";

However when we used this value for cil_settletime we could not acquire an image on deserializer 0 MIPI port 0.
So we changed the value of cil_settletime to 0.
3. Why does a correctly computed cil_settletime have adverse affects? Or is there something wrong with our calculations?

We also reviewed all camera related device tree settings.
We removed several optional and non-applicable to our system entries.
camDevTree-04Feb2025.txt (6.7 KB)

4. Do our revised camera device tree nodes look correct?

We also reviewed Camera Driver Porting Tips and found no anomalies.

We reviewed the Debugging Tips page found no changes to our system are necessary.

We successfully tested these changes with deserializer 0 connected to Orin MIPI port 0.

However when we switched over to deserialzier 2 connected to Orin MIPI port 4 we were still unable to obtain an image. We still receive the 131072 error.

[  162.677780] ov10635 0-0009: Setting format
[  162.678581] ov10635 0-0009: Setting format
[  162.678745] ov10635 0-0009: Setting format
[  162.681610] ov10635 0-0009: Power on
[  162.720503] bwmgr API not supported
[  162.732195] ov10635 0-0009: Setting stream to 1
[  162.733343] ov10635 0-0009: Write sensor in addr=0x9 reg=0x100 val=0x1
[  162.791839] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 131072
[  162.824996] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 131072
[  162.891829] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 131072
[  162.892175] ov10635 0-0009: Setting stream to 0
[  162.893239] ov10635 0-0009: Write sensor in addr=0x9 reg=0x100 val=0x0
[  162.897259] bwmgr API not supported
[  162.906888] ov10635 0-0009: Power off

During the failed capture attempt on deserializer 2 connected to Orin MIPI port 4, we also captured the debug trace and attached it as well
camDevTree-04Feb2025.txt (6.7 KB)

In summary, even though we have learned more about a number of the device tree cammera properties as well as became more confident of the settings we are using the problem still persists.

Do you have any further suggestions to debug this problem with Orin MIPI port 4?

JerryChang · February 5, 2025, 2:26am

hello elitewompa,

here’s JetPack-5.1.4/l4t-r35.6.0 camera firmware with debug flag enabled. for instance, Topic320332_Feb05.7z (254.0 KB)
please apply the debug RCE firmware to gather more details.
you may also refer to Topic 226574 for sample commands to apply debug firmware with flash script.

elitewompa · February 5, 2025, 3:13pm

Hello @JerryChang,

Per your instructions we flashed the RCE debug firmware.

For each of the following tests we enabled the max clocks and debug trace with the following commands:


echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate

echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 2 > /sys/kernel/debug/camrtc/log-level
echo 1 > /sys/kernel/debug/tracing/events/camera_common/enable

dmesg > dmesg-boot.txt

dmesg -C

gst-launch-1.0 v4l2src device=/dev/gmsl/cam0 num-buffers=1 ! jpegenc ! filesink location=/home/root/cam0.jpg

dmesg > dmesg-acquire.txt

cat /sys/kernel/debug/tracing/trace >debugTrace.txt

We performed 1 successful acquisition with deserializer 0 on MIPI port 0 and observed the following results
dmesg-boot.txt (96.9 KB)
dmesg-acquire.txt (2.7 KB)
debugTrace.txt (17.7 KB)

We performed 1 unsuccessful acquisition with deserializer 2 on MIPI port 4 and observed the following results
dmesg-boot.txt (95.4 KB)
dmesg-acquire.txt (3.1 KB)
debugTrace.txt (1.5 MB)

Question:
1. Are there registers on the Orin that control lane polarity? Or lane swapping?

elitewompa · February 5, 2025, 7:20pm

FYI @JerryChang

We reviewed the following polarity flip registers. We found no polarity flips based on reset values.
(See Orin-TRM_DP10508002_v1.2p.pdf pg. 1494)
NVCSI_PHY_0_NVCSI_CIL_A_POLARITY_SWIZZLE_CTRL_0

(See Orin-TRM_DP10508002_v1.2p.pdf pg. 1530)
NVCSI_PHY_0_NVCSI_CIL_B_POLARITY_SWIZZLE_CTRL_0

(See Orin-TRM_DP10508002_v1.2p.pdf pg. 1597)
NVCSI_PHY_1_NVCSI_CIL_A_POLARITY_SWIZZLE_CTRL_0

(See Orin-TRM_DP10508002_v1.2p.pdf pg. 1633)
NVCSI_PHY_1_NVCSI_CIL_B_POLARITY_SWIZZLE_CTRL_0

(See Orin-TRM_DP10508002_v1.2p.pdf pg. 1700)
NVCSI_PHY_2_NVCSI_CIL_A_POLARITY_SWIZZLE_CTRL_0

(See Orin-TRM_DP10508002_v1.2p.pdf pg. 1736)
NVCSI_PHY_2_NVCSI_CIL_B_POLARITY_SWIZZLE_CTRL_0

(See Orin-TRM_DP10508002_v1.2p.pdf pg. 1803)
NVCSI_PHY_3_NVCSI_CIL_A_POLARITY_SWIZZLE_CTRL_0

(See Orin-TRM_DP10508002_v1.2p.pdf pg. 1839)
NVCSI_PHY_3_NVCSI_CIL_B_POLARITY_SWIZZLE_CTRL_0

elitewompa · February 5, 2025, 7:30pm

FYI @JerryChang
We reviewed all register reads and writes between deserializer 0 connected to MIPI port 0 and deserializer 2 connected to MIPI port 4.

We found only 2 differences.

Difference 1 was the lane swap register.
Difference 2 was the polarity flip register.

These differences were independently re-verified by multiple team members and were found to be correct.

So other than these 2 expected differences all other deserializer registers are identical between deserializer 0 and deserializer 2.

elitewompa · February 5, 2025, 9:13pm

Hello @JerryChang

Is there a way to disable CRC checking and just let the data arrive and so we could see what the rest of the acquisition path does when CRC errors are ignored?

JerryChang · February 6, 2025, 2:12am

hello elitewompa,

all right, I don’t see any suspicious failures reported by rce debug logs.
you’re working with AGX Orin, right? you don’t need to configure a polarity swap. please configure lane_polarity = "0"; for testing.

JerryChang · February 6, 2025, 2:25am

hello elitewompa,

it’ll need to rebuild the rce firmware to ignore CRC checks, let’s try setting a correct polarity swap for verification.

elitewompa · February 6, 2025, 7:02pm

Yes, we use the NVidia Jetson AGX Orin Module 64GB - Non Industrial Version

elitewompa · February 6, 2025, 7:33pm

Hello @JerryChang

We added lane_polarity = “0” to our device tree.
camDevTree-06Feb2025.txt (6.7 KB)

We performed 1 successful acquisition with deserializer 0 on MIPI port 0 and observed the following results
dmesg-boot.txt (98.4 KB)
dmesg-acquire.txt (2.7 KB)
debugTrace.txt (21.3 KB)

We performed 1 unsuccessful acquisition with deserializer 2 on MIPI port 4 and observed the following results
dmesg-boot.txt (101.3 KB)
dmesg-acquire.txt (3.1 KB)
debugTrace.txt (1.6 MB)

In summary, adding lane_polarity = “0” had no effect on the results.

JerryChang · February 7, 2025, 6:01am

hello elitewompa,

please also disassembler the dtb file into text file, I would like to check the final device tree system has loaded.
for instance, $ dtc -I dtb -O dts -o temp.txt tegra234-xxx.dtb

elitewompa · February 7, 2025, 1:06pm

Attached is the live device tree when configured for deserializer 2 MIPI port 4
devtree-20250207-0702.txt (434.8 KB)

Topic		Replies	Views
Camera fails to capture frames with VIFALC_TDSTATE error Jetson AGX Orin camera	25	1034	February 14, 2024
No frame from IMX415 on orin devkit Jetson AGX Orin camera	3	526	November 6, 2023
Ov01f10-max96701-max96712 orin capture error Jetson Orin NX camera , kernel	20	1484	August 25, 2023
AGX Orin camera issues Jetson AGX Orin camera , vi	26	930	July 1, 2024
Cameras not detected on orin devkit + jetpack6 (R36.x) Jetson AGX Orin camera	30	838	June 4, 2024
V4l2-ctl can't capture video data Jetson AGX Orin camera , board-design	44	3354	May 17, 2023
AGX Orin 100% CPU usage when multiple video devices are enabled at the same time, one of the camera devices stops streaming data unexpectedly Jetson AGX Orin camera	30	1775	October 3, 2023
Orin nx max9296 + max9295 + uyvy sensor driver ctrl Gain range update failed message Jetson Orin NX camera , kernel , jetson , chinese	34	106	January 16, 2025
Porting new camera sensor driver Jetson Orin NX camera	19	1315	February 15, 2024
Orin fails to capture frame - uncorr_err: request timed out after 2500 ms Jetson Orin Nano camera	32	238	February 10, 2025

AGX Orin CSI E (Port 4) support

Related topics