General error queue is out of sync with frame queue

[norman_xu]

Hi Nvidia,

Hardware: Jetson Orin NX
System version: Jetpack 5.1.3 ( R35.5.0)
Camera: 2M raw Camera and 8M raw camera

1.Apply this patch

When the camera driver was upgraded from r35.4.1 to r35.5.0, a kernel crash occurred when vi could not receive data

hello amanelli, here’re patches for r35.5.0 (and also r35.6.0) to fix the failure caused by tegra_channel_error_recover(). for instance, 0001-kernel-camera-remove-csi-event-sync-for-recover.patch (2.3 KB) 0002-vi5_fops-fix-mem-leak.patch (4.3 KB) just double confirm the release version, did you meant jetson-linux-r3560 or jetson-linux-r363? could you please check the release tag for confirmation, $ cat /etc/nv_tegra_release please also gather the complete error logs for reference. BTW, …

2.Connect two cameras with different resolutions to one mipi (4-lane), configure the same “pix_clk_hz” and “serdes_pix_clk_hz” in the device tree.

pix_clk_hz = "200000000";
serdes_pix_clk_hz = "500000000";

4.Increase the rate

echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_lock ed echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate

5.There is no problem bringing up the 2M raw camera and the 8M raw camera separately. But when switching cameras, an exception will occur.

Test as follows:
Use the following command to bring up the 2M raw camera, it is ok.
v4l2-ctl --set-ctrl bypass_mode=0 --stream-mmap -d /dev/video0

Use the following command to turn on the 8M raw camera, and an error will be reported.

After Ctl-C stops, execute the command again, and then it returns to normal.

6. Question
How to fix the following error：

[ 1916.008451] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: flags 2, err_data 0
[ 1916.023491] [RCE] ERROR: camera-ip/vi5/vi5.c:745 [vi5_handle_eof] "General error queue is out of sync with frame queue. ts=1934051999520 sof_ts=1934052013792 gerror_code=2 gerror_data=a0 notify_bits=4"
[ 1921.151568] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 5000 ms
[ 1921.160715] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[ 1921.171623] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel

[carolyuu]

Hi,

For the camera basic functionality first needs to check the device and driver configuration.
You can reference to below program guide for the detailed information of device tree and driver implementation.
https://docs.nvidia.com/jetson/archives/r36.3/DeveloperGuide/SD/CameraDevelopment/SensorSoftwareDriverProgramming.html?highlight=programing#sensor-software-driver-programming

Please refer to Applications Using V4L2 IOCTL Directly by using V4L2 IOCTL to verify basic camera functionality.
https://docs.nvidia.com/jetson/archives/r36.3/DeveloperGuide/SD/CameraDevelopment/SensorSoftwareDriverProgramming.html?highlight=programing#to-run-a-v4l2-ctl-test

Once confirm the configure and still failed below link help to get log and some information and some tips for debug.
https://elinux.org/Jetson/l4t/Camera_BringUp#Steps_to_enable_more_debug_messages

Thanks!

[JerryChang]

@norman_xu

System version: Jetpack 5.1.3 (R35.5.1)

there’re several camera bug within earlier Jetpack release version.
is it possible for moving to the latest JetPack 5.1.4 for verification?

[norman_xu]

Hi jerry,

The customer requested to use this version. Can you help analyze and troubleshoot it?

[JerryChang]

hello norman_xu,

let’s try adding format settings into v4l pipeline for launching camera streams.
for instance,
please examine your sensor format dumps, $ v4l2-ctl -d /dev/video0 --list-formats-ext
after that, please specify the supported format into v4l pipeline for testing.
$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=100

[norman_xu]

Hi Jerry,

The problem still occurs.

[JerryChang]

hello norman_xu,

it looks weird, is it actually sending MIPI signaling to CSI channel while issue happened?
BTW, had you refer to Topic 293597 to apply camera firmware update to improve CSI stability.

[norman_xu]

Hi Jerry,

1.Yes, it actually sending MIPI signaling.
2.How to flash it on Jetson Orin NX?

I flash with the follow command
sudo ./flash.sh -r -k A_rce-fw jetson-orin-nano-devkit nvme0n1p1

then got stuck

[JerryChang]

hello norman_xu,

you should execute with l4t_initrd_flash.sh since you’re working with Orin NX,
please refer to Topic 260583 for demonstration steps to replace the camera debug firmware.

[norman_xu]

Hi Jerry，

We have verified that there is no such issue on Jetpack 5.1.4 (R35.6.0).

Because our product has been tested on Jetpack 5.1.3 (R35.5.0) for a long time and is about to be mass-produced, we do not want to upgrade to Jetpack 5.1.4 (R35.6.0). Can you provide a patch to fix this issue?

[norman_xu]

Hi Jerry，

1.I failed to flash the camera debug firmware by executing l4t_initrd_flash.sh

2.Then I ran a full flash by execute the following command

sudo ./apply_binaries.sh
sudo ./tools/kernel_flash/l4t_initrd_flash.sh --external-device nvme0n1p1   -c tools/kernel_flash/flash_l4t_external.xml -p "-c bootloader/t186ref/cfg/flash_t234_qspi.xml"   --showlogs --network usb0 jetson-orin-nano-devkit internal

Verified again, the issue is solved, thanks.

[JerryChang]

hello norman_xu,

you may try adding --network usb0 to flash pipeline.
or… partition update has disabled if those were fused (i.e. secureboot enabled)

anyways,

@norman_xu

Verified again, the issue is solved, thanks.

got it, thanks for status update.