Enabling rtcpu debug on custom board & Resolving rtcpu_vinotify_event tag:VIFALC_TDSTATE error

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
,
1

Hi Mods,

For context I am porting e-con AR0230 cameras from AGX Orin development kit to a custom carrier board. Previous thread here, resolved most of the device tree issues.

Gstreamer command to launch camera:
gst-launch-1.0 -v v4l2src device="/dev/video0" ! video/x-raw,framerate=60/1,width=640,height=480,format=UYVY ! xvimagesink

Right now I’m trying to open the cameras and retrieve the video stream but I am face with this error in dmesg:

[  183.948740]  Index = 0x0000 , format = 0x59565955, width = 640, height = 480, frate num = 60 
[  183.952423] Skipping Previous mode set ... 
[  184.159980]  Index = 0x0000 , format = 0x59565955, width = 640, height = 480, frate num = 60 
[  184.159989] Skipping Previous mode set ... 
[  184.225705] bwmgr API not supported
[  186.872605] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  186.885608] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  186.896447] (NULL device *): vi_capture_control_message: NULL VI channel received
[  186.904308] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=0, csi_port=0
[  186.915123] (NULL device *): vi_capture_control_message: NULL VI channel received
[  186.922988] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 0 vc- 0
[  186.934081] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  189.464551] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  189.473859] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  189.484453] (NULL device *): vi_capture_control_message: NULL VI channel received
[  189.492268] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=0, csi_port=0
[  189.503058] (NULL device *): vi_capture_control_message: NULL VI channel received
[  189.510871] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 0 vc- 0
[  189.521904] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  189.533705]  mcu_cam_stream_off 269 CAM Get CMD Stream off Success !! 
[  189.533724] (NULL device *): vi_capture_control_message: NULL VI channel received
[  189.541601] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=0, csi_port=0

I tried following some of the threads which talks about debugging cameras through /sys/kernel/debug/tracing/trace. Below are the commands I ran:

echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
###
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
832000000
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee  /sys/kernel/debug/bpmp/debug/clk/isp/rate
1011200000
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
642900000
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate
3199000000
###
echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 2 > /sys/kernel/debug/camrtc/log-level
echo 1 > /sys/kernel/debug/tracing/events/camera_common/enable
echo > /sys/kernel/debug/tracing/trace

cat /sys/kernel/debug/tracing/trace returns below log after trying to launch gstreamer which has the error:
rtcpu_vinotify_event: tstamp:6526856137 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:208848999168 data:0x379d580010000000.

root@miivii-tegra:/home/nvidia# cat /sys/kernel/debug/tracing/trace
# tracer: nop
#
# entries-in-buffer/entries-written: 35/35   #P:8
#
#                                _-----=> irqs-off
#                               / _----=> need-resched
#                              | / _---=> hardirq/softirq
#                              || / _--=> preempt-depth
#                              ||| /     delay
#           TASK-PID     CPU#  ||||   TIMESTAMP  FUNCTION
#              | |         |   ||||      |         |
     kworker/2:4-669     [002] ....   176.192580: rtcpu_string: tstamp:6275481431 id:0x04010000 str:"VM0 deactivating."
  gst-launch-1.0-7245    [006] ....   183.675355: tegra_channel_open: vi-output, ar0230 2-0043
  gst-launch-1.0-7245    [006] ....   183.675424: tegra_channel_close: vi-output, ar0230 2-0043
  gst-launch-1.0-7245    [006] ....   183.675432: tegra_channel_open: vi-output, ar0230 2-0044
  gst-launch-1.0-7245    [006] ....   183.675442: tegra_channel_close: vi-output, ar0230 2-0044
  gst-launch-1.0-7245    [006] ....   183.681383: tegra_channel_open: vi-output, ar0230 2-0043
    v4l2src0:src-7247    [003] ....   184.215702: tegra_channel_set_power: ar0230 2-0043 : 0x1
    v4l2src0:src-7247    [003] ....   184.215722: camera_common_s_power: status : 0x1
    v4l2src0:src-7247    [003] ....   184.216239: tegra_channel_set_power: 13e40000.host1x:nvcsi@15a00000- : 0x1
    v4l2src0:src-7247    [003] ....   184.216244: csi_s_power: enable : 0x1
    v4l2src0:src-7247    [003] ....   184.216392: tegra_channel_capture_setup: vnc_id 0 W 640 H 480 fmt 13
    v4l2src0:src-7247    [003] ....   184.223210: tegra_channel_set_stream: enable : 0x1
    v4l2src0:src-7247    [003] ....   184.226133: tegra_channel_set_stream: 13e40000.host1x:nvcsi@15a00000- : 0x1
    v4l2src0:src-7247    [003] ....   184.226136: csi_s_stream: enable : 0x1
    v4l2src0:src-7247    [003] ....   184.226467: tegra_channel_set_stream: ar0230 2-0043 : 0x1
     kworker/2:4-669     [002] ....   184.248519: rtcpu_string: tstamp:6526388682 id:0x04010000 str:"VM0 activating."
     kworker/2:4-669     [002] ....   184.248523: rtcpu_vinotify_event: tstamp:6526856137 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:208848999168 data:0x379d580010000000
     kworker/2:4-669     [002] ....   184.248524: rtcpu_vinotify_event: tstamp:6526856384 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:208849005696 data:0x0000000031000001
     kworker/2:4-669     [002] ....   184.248524: rtcpu_vinotify_event: tstamp:6526856632 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:208849061888 data:0x379d550010000000
     kworker/2:4-669     [002] ....   184.248524: rtcpu_vinotify_event: tstamp:6526856838 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:208849068480 data:0x0000000031000002
 vi-output, ar02-7261    [005] ....   186.933923: tegra_channel_capture_setup: vnc_id 0 W 640 H 480 fmt 13
     kworker/2:4-669     [002] ....   186.980514: rtcpu_vinotify_event: tstamp:6611461763 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:211559909536 data:0x379d580010000000
     kworker/2:4-669     [002] ....   186.980518: rtcpu_vinotify_event: tstamp:6611461974 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:211559916032 data:0x0000000031000001
     kworker/2:4-669     [002] ....   186.980519: rtcpu_vinotify_event: tstamp:6611462226 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:211564003552 data:0x379d550010000000
     kworker/2:4-669     [002] ....   186.980520: rtcpu_vinotify_event: tstamp:6611462432 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:211564010144 data:0x0000000031000002
 vi-output, ar02-7261    [002] ....   189.521744: tegra_channel_capture_setup: vnc_id 0 W 640 H 480 fmt 13
  gst-launch-1.0-7245    [001] ....   189.532400: tegra_channel_set_stream: enable : 0x0
  gst-launch-1.0-7245    [001] ....   189.532402: tegra_channel_set_stream: ar0230 2-0043 : 0x0
  gst-launch-1.0-7245    [002] ....   189.533712: tegra_channel_set_stream: 13e40000.host1x:nvcsi@15a00000- : 0x0
  gst-launch-1.0-7245    [002] ....   189.533717: csi_s_stream: enable : 0x0
  gst-launch-1.0-7245    [002] ....   189.554469: tegra_channel_set_power: ar0230 2-0043 : 0x0
  gst-launch-1.0-7245    [002] ....   189.554486: camera_common_s_power: status : 0x0
  gst-launch-1.0-7245    [002] ....   189.555009: tegra_channel_set_power: 13e40000.host1x:nvcsi@15a00000- : 0x0
  gst-launch-1.0-7245    [002] ....   189.555013: csi_s_power: enable : 0x0
  gst-launch-1.0-7245    [002] ....   189.557756: tegra_channel_close: vi-output, ar0230 2-0043

Output of v4l2-ctl -d /dev/video1 --list-formats-ext

	Type: Video Capture

	[0]: 'UYVY' (UYVY 4:2:2)
		Size: Discrete 640x480
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.022s (45.000 fps)
		Size: Discrete 960x540
			Interval: Discrete 0.017s (58.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.022s (45.000 fps)
		Size: Discrete 1280x960
			Interval: Discrete 0.029s (34.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.033s (30.000 fps)
	[1]: 'NV16' (Y/CbCr 4:2:2)
		Size: Discrete 640x480
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.022s (45.000 fps)
		Size: Discrete 960x540
			Interval: Discrete 0.017s (58.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.022s (45.000 fps)
		Size: Discrete 1280x960
			Interval: Discrete 0.029s (34.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.033s (30.000 fps)

Unsure how I should enable rtcpu debug (tegra-rtcpu-trace) on dmesg to get further debug logs, since I am on a custom board and the .img file going around are for the development kit. Any advice would be appreciated.

.dtsi file and the video modes are attached for reference, hopefully they are helpful.
tegra234-camera-max9296-a-nilecam.dtsi.txt (9.0 KB)
miivii-nilecam20_cuxvr_mode.dtsi.txt (7.6 KB)

3

The trace log tell didn’t receive any validate data from the sensor.
Only the board design able to help to review the dts.

4

What exact schematics do you require to help debug this? Do you have a sample so that I can ask the manufacturer for it? Is something like this sufficient? - https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-325/Tegra%20Linux%20Driver%20Package%20Development%20Guide/images/gmsl_camera_framework.1.02.jpg
image

My current understanding of the connection on the custom board is:
Orin i2c-2 bus → MAX9296A deserializer → (MAX9295 serializer → Onsemi AR0230 sensor)

5

You can download the kernel source to reference to those sample driver and device tree. Also check the programing guide to configure those properties like port-index …

6

@ShaneCCC thanks for the advice, I have already did that. The provided .dtsi file is referenced from Jetson Virtual Channel with GMSL Camera Framework — Jetson Linux<br/>Developer Guide 34.1 documentation.

I paid attention to these sections in the above document:

  1. tegra-capture-vi { ... }
  2. tegra-camera-platform { ... }
  3. host1x@13e00000 { nvcsi@15a00000 { ... } }
  4. And the camera device tree is set in i2c@3180000 { ... }

In vi channel, we linked

  • gmslcomm_vi_in0 → gmslcomm_csi_out0

In nvcsi, we linked

  • gmslcomm_csi_in0 → gmslcomm_des_out0
  • gmslcomm_csi_out0 → gmslcomm_vi_in0

In the camere i2c tree, we linked

  • gmslcomm_des_out0 → gmslcomm_csi_in0

Also port_index is set to <0> for the first camera and second camera since they are on the same deserializer.

However, I didn’t write gmsl-link in the device tree since I am not sure if it is needed. The vc-id is set to <0> for the first camera and <1> for the second camera.

The below code (also in the attached .dtsi file above) shows the port binding, do you mind taking a look to see if it is correct, and let me know what is missing? Thank you in advance!

tegra-capture-vi {
    num-channels = <2>;
    #address-cells = <1>;
    #size-cells = <0>;
    status = "okay";
	ports {
		#address-cells = <1>;
		#size-cells = <0>;
		status = "okay";
		port@0 { // first camera
			reg = <0>;
			status = "okay";
			gmslcomm_vi_in0: endpoint {
			vc-id = <0>;
			port-index = <0>;
			bus-width = <2>;
			status = "okay";
			remote-endpoint = <&gmslcomm_csi_out0>;
		};
        port@1 { ... // second camera
        };
    };
};

host1x@13e00000 {
	nvcsi@15a00000 {
		num-channels = <2>;
		#address-cells = <1>;
		#size-cells = <0>;
		status = "okay";
		channel@0 { // first camera
			reg = <0>;
			status = "okay";
			ports {
				#address-cells = <1>;
				#size-cells = <0>;
				status = "okay";
				port@0 {
					reg = <0>;
					status = "okay";
					gmslcomm_csi_in0: endpoint@0 {
						port-index = <0>;
						bus-width = <2>;
						status = "okay";
						remote-endpoint = <&gmslcomm_des_out0>;
					};
				};
				port@1 {
					reg = <1>;
					status = "okay";
					gmslcomm_csi_out0: endpoint@1 {
					status = "okay";
					remote-endpoint = <&gmslcomm_vi_in0>;
					};
				};
			};
		};
    channel@1 { ... // second camera
    };
};

i2c@3180000 { ...
    ar0230_a@43 { ... // first camera
        mode0 { ...
        };
        ports {
            #address-cells = <1>;
            #size-cells = <0>;

            port@0 {
                reg = <0>;
                gmslcomm_des_out0: endpoint {
                vc-id = <0>;
                port-index = <0>;
                bus-width = <2>;
                remote-endpoint = <&gmslcomm_csi_in0>;
	        };
        };
    };
    ar0230_b@44 { ... // second camera
    };
};
7

That’s looks correct.

Thanks

8

@ShaneCCC thanks for the confirmation, however, I am still getting the same error and there is no video output. What else could be the problem here?

9

The trace log show the same? If yes you may need to probe the sensor output signal.

Thanks

10

@ShaneCCC Yes, unfortunately the trace log shows the same error with the above .dtsi :(
I am still getting NULL VI channel received and rtcpu logs still shows the same VIFALC_TDSTATE error.

What do you mean by probing the output signal of the sensor? Do you have a page that I can read / reference from?

11

I mean use scope to make sure the MIPI signal.

12

@ShaneCCC Ah that would be too involved for me to scope the sensor output signal.

I know that the cameras do work on the Orin AGX Dev Kit with another deserializer board, so the e-con cameras aren’t the issue. The MIIVII Apex AD10 deserializers do work since I am able to get it to work with another camera (IMX390) which already has the kernel driver in the L4T kernel.

Could it still be that my device tree has issues resulting in the e-con AR0230 cameras not working?

13

Maybe you can configure the deserializer output test pattern to narrow down it.

14

@ShaneCCC this might be out of my expertise, but I will give it a shot. Thank you for the help thus far.

15

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.