We are migrating our design from the TX2 to the TX2i. Our application uses four cameras, a color with Bayer mosaic on csi 0/1, a second Bayer camera on csi 2/3, a monochrome camera on csi 4 and a second monochrome camera on csi 5. The first two cameras are processed through the ISP and are accessed using the argus library. The other two cameras are accessed using the ioctl interface. Our board and application work consistently with the TX2. Our camera drivers and camera application have all been thoroughly tested on the TX2.
However, when we substitute in the TX2i, the fourth camera, the one on csi 5 stops working. The other three cameras continue to work without error. The system reports no errors on power-up: we have video 0,1,2,3, pass v4ctl-compliance tests, and all ports bind as expected.
When we try to access the fourth camera, we get PXL_SOF syncpt time! Err=-11. When we run a trace, we get the rtcpu_start message, the CSIMUX_STREAM message, and then a sequence of rtos_queue_peek_from_isr_failed messages. Occasionally, when we start the camera, we get a single, corrupted, frame, but usually the frame is black.
We tried changing the MIPI Calibration timeout, as mentioned in https://devtalk.nvidia.com/default/topic/1042163/?comment=5286455
That had no effect.
While this is clearly due to some subtle difference between the TX2 and TX2i, we do not know where to look next to find the problem. Could you provide guidance? Thanks!
Update: Further tracing shows that the error we get on csi port 5 is a CHANSEL pixel runaway. It is reporting an error when we go one line beyond the 512 frame height. Any thoughts on why the TX2i would throw this error when the TX2 has no problem?
hello carcher18a4s,
- may I know which JetPack release you’re working with
- could you please share the VI tracing logs of your 4th camera.
steps to enable the VI tracing from debugfs
echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 2 > /sys/kernel/debug/camrtc/log-level
echo > /sys/kernel/debug/tracing/trace
cat /sys/kernel/debug/tracing/trace
Hi Jerry,
We are running JetPack 3.2.1.
I am attaching the trace from two runs. The first run is using the csi 4 camera that works correctly. The second run (starting at time stamp 2210.026629) is from the csi 5 camera that does not work. Note that the problem is tied to csi 5, not the particular camera. When I swap cameras, the fault still occurs on csi 5.
trace.txt (113 KB)
hello carcher18a4s,
according to the tracing logs, it seems VI engine did not receive sensor streaming.
kworker/0:2-2724 [000] ...1 2210.026635: rtcpu_vinotify_handle_msg: tstamp:69846501288 tag:CSIMUX_STREAM channel:0xff frame:0 vi_tstamp:1127024169 data:0x00100000
kworker/0:2-2724 [000] ...1 2210.182596: rtos_queue_peek_from_isr_failed: tstamp:69851463710 queue:0x0b4a3c58
kworker/0:2-2724 [000] ...1 2210.338680: rtos_queue_peek_from_isr_failed: tstamp:69856464295 queue:0x0b4a3c58
kworker/0:2-2724 [000] ...1 2210.494689: rtos_queue_peek_from_isr_failed: tstamp:69861464909 queue:0x0b4a3c58
kworker/0:2-2724 [000] ...1 2210.650668: rtos_queue_peek_from_isr_failed: tstamp:69866465295 queue:0x0b4a3c58
...
couple of suggestions as below,
- we had fix some known issue for l4t-r28.2.1 kernel drivers, please apply the patches from Topic 1038067.
- are you able to probe the hardware signaling? please check the MIPI signaling is work as expect.
Thanks, Jerry.
We tried the patch today and it made no difference. Attaching the trace from running the camera on the csi5 port.
The MIPI signaling is working as expected. In addition, camera works with no problem on the TX2.
Any additional data that would be helpful? We can even stick printk statements in the driver code, if you tell us where.
tx2icsi5trace.txt (6.61 KB)
As you can see from the last trace dump, we are not getting any useful error messages. And it sounds like this is not a problem you have seen before. So we need to find a way to look deeper into what is happening.
Can you tell us where there are differences between TX2 and TX2i in the CSI → VI area of the SOM (hardware or driver), so we have some idea of where to start looking for a solution?
Thanks.
hello carcher18a4s,
- please enable debug prints for TX2’s VI driver, steps to enable dynamic debug flag, dev_dbg(…)
cd /sys/kernel/debug/dynamic_debug/
echo file vi4_fops.c +p > control
-
could you please share the commands of accessing monochrome camera on csi-5.
-
you might also look into below kernel sources,
here are the control ops you might check
$TOP/kernel_src/kernel/kernel-4.4/drivers/media/platform/tegra/camera/vi/channel.c
static const struct vb2_ops tegra_channel_queue_qops = {
.queue_setup = tegra_channel_queue_setup,
.buf_prepare = tegra_channel_buffer_prepare,
.buf_queue = tegra_channel_buffer_queue,
.wait_prepare = vb2_ops_wait_prepare,
.wait_finish = vb2_ops_wait_finish,
.start_streaming = tegra_channel_start_streaming,
.stop_streaming = tegra_channel_stop_streaming,
};
you might also check below functions about frame capture. also, vi_notify_wait() is waiting for sensor start of frame hardware signaling.
$TOP/kernel_src/kernel/kernel-4.4/drivers/media/platform/tegra/camera/vi/vi4_fops.c
static int tegra_channel_capture_frame(struct tegra_channel *chan,
struct tegra_channel_buffer *buf)
{
...
/* wait for vi notifier events */
vi_notify_wait(chan, &ts);
}
Thanks, Jerry. We will see what we can learn using your suggestions today. We are also having our electrical engineer double check trace lengths of the csi lines on our board.
We have attached our application for running the camera on CSI 4 and CSI 5. This is based off one of the examples Nvidia provides with Jetpack. We run it from the command line with
./camera_v4l2 -d /dev/video# -s 640x512 -r 30
where # is 2 for csi4 and 3 for csi5, but you would use what matches your test set-up and camera
FFv4l2camera.tar.gz (197 KB)
We have new debug files for you to look at as well. Not sure they are telling us anything new, but maybe its something you have seen before? The TX2 log is the one that works, the TX2i is the one where we get the black image.
video3csi5TX2i.txt (5.49 KB)
video3csi5TX2.txt (26.7 KB)
hello carcher18a4s,
there are two difference between your TX2 and TX2i results,
- it seems you had TC358746 bridge to transmit CSI signaling.
I saw your configurations for TX2 and TX2i were different.
TX2i
[ 9015.268933] -----TC358746 CSI-TX status-----
[ 9015.273258] TC358746 Lanes in use: 2
[ 9015.277144] TC358746 Waiting for particular sync signal: yes
[ 9015.282856] TC358746 Transmit mode: no
[ 9015.286641] TC358746 Stopped: yes
TX2
[ 3839.850369] -----TC358746 CSI-TX status-----
[ 3839.854699] TC358746 Lanes in use: 2
[ 3839.858530] TC358746 Waiting for particular sync signal: no
[ 3839.864127] TC358746 Transmit mode: no
[ 3839.867922] TC358746 Stopped: no
- also, your bridge driver sending “TWO” SOF (start-of-frame) packets in the sensor streaming,
please check below snippet,
[ 9016.304020] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
[ 9016.310410] video4linux video3: tegra_channel_capture_frame: vi4 got SOF syncpt buf[ffffffc17c600000]
[ 9017.316054] tegra-vi4 15700000.vi: ATOMP_FE syncpt timeout! err = -11
[ 9017.316056] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
<b>[ 9017.316064] video4linux video3: tegra_channel_capture_frame: vi4 got SOF syncpt buf[ffffffc17c603400]</b> <=== additional SOF
[ 9017.338071] video4linux video3: tegra_channel_release_frame: vi4 got EOF syncpt buf[ffffffc17c600000]
VI driver is expecting hardware signaling coming with SOF/EOF as one complete frame.
could you please check with bridge driver vendor for the configuration.
thanks
We wanted to give Nvidia some final feedback on this issue. We went back to both the bridge vendor and our camera electronics vendor. The problem turned out to be with our camera electronics board. The material used in the board, coupled with the way the traces were routed, caused the camera to not meet the impedance requirements for MIPI CSI. A new careful board layout with a different board manufacturer fixed the problem. We now have cameras working with the TC358746 in the CSI4 and CSI5 slots on both the TX2 and TX2i.
Thanks for the help.