[Camera Sensor Driver ] - device is not able to capture frames

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
1

Our hardware is based on an FPGA device that takes SDI input and outputs CSI data.
The device is self-driven, so there is no I²C interface required for configuration.
We are developing a driver on the AGX Orin platform to support it.
In the driver, the I²C part is ignored, and we are able to successfully create the /dev/video0 node.
Required modifications in the Device Tree have also been completed.
Now, we are trying to receive CSI-2 data without I2C from an FPGA on AGX Orian (36.4.3). the device is still unable to capture frames. Although /dev/video0 appears successfully. need help to debug these issues
We are using v4l2-ctl to try to capture the data.

#v4l2-ctl -d /dev/video0 --stream-mmap --stream-count=5 --verbose
VIDIOC_QUERYCAP: ok
VIDIOC_REQBUFS returned 0 (Success)
VIDIOC_QUERYBUF returned 0 (Success)
VIDIOC_QUERYBUF returned 0 (Success)
VIDIOC_QUERYBUF returned 0 (Success)
VIDIOC_QUERYBUF returned 0 (Success)
VIDIOC_QBUF returned 0 (Success)
VIDIOC_QBUF returned 0 (Success)
VIDIOC_QBUF returned 0 (Success)
VIDIOC_QBUF returned 0 (Success)
VIDIOC_STREAMON returned 0 (Success)
cap dqbuf: 0 seq: 0 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 0 seq: 0 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 1 seq: 1 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 2 seq: 2 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 3 seq: 3 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 0 seq: 0 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 0 seq: 0 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 1 seq: 1 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 2 seq: 2 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)
cap dqbuf: 3 seq: 3 bytesused: 4147200 ts: 0.000000 (error, ts-monotonic, ts-src-eof)

#dmesg log-
fpga_dev_open
[75462.407171] fpga-media 2-0036: fpga_dev_open:
[75542.177243] power_on
[75542.188172] _set_mode
[75542.188178] fpga_dev_start_streaming
[75542.188179] fpga_dev_start_streaming -done
[75544.849309] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[75544.849333] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[75544.850583] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[75547.409428] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[75547.409448] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[75547.410694] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[75549.937575] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[75549.937591] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[75549.938819] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel

#media-ctl -p
Media controller API version 5.15.148

Media device information

driver tegra-camrtc-ca
model NVIDIA Tegra Video Input Device
serial
bus info
hw revision 0x3
driver version 5.15.148

Device topology

  • entity 1: 13e00000.host1x:nvcsi@15a00000- (2 pads, 2 links)
    type V4L2 subdev subtype Unknown flags 0
    device node name /dev/v4l-subdev0
    pad0: Sink
    ← “fpga-media 2-0036”:0 [ENABLED]
    pad1: Source
    → “vi-output, fpga-media 2-0036”:0 [ENABLED]

  • entity 4: fpga-media 2-0036 (1 pad, 1 link)
    type V4L2 subdev subtype Sensor flags 0
    device node name /dev/v4l-subdev1
    pad0: Source
    [fmt:UYVY8_1X16/1920x1080@1/60 field:none colorspace:srgb]
    → “13e00000.host1x:nvcsi@15a00000-”:0 [ENABLED]

  • entity 6: vi-output, fpga-media 2-0036 (1 pad, 1 link)
    type Node subtype V4L flags 0
    device node name /dev/video0
    pad0: Sink
    ← “13e00000.host1x:nvcsi@15a00000-”:1 [ENABLED]

We are using following way to trace log to see NVCSI/VI activity but not receive any frame
echo 1 > /sys/kernel/debug/tracing/events/camera_common/enable
cat /sys/kernel/debug/tracing/trace_pipe
<…>-5274 [003] … 79509.386783: tegra_channel_open: vi-output, fpga-media 2-0036
v4l2-ctl-5274 [003] … 79509.414745: tegra_channel_set_power: fpga-media 2-0036 : 0x1
v4l2-ctl-5274 [003] … 79509.414761: camera_common_s_power: status : 0x1
v4l2-ctl-5274 [003] … 79509.414776: tegra_channel_set_power: 13e00000.host1x:nvcsi@15a00000- : 0x1
v4l2-ctl-5274 [003] … 79509.414778: csi_s_power: enable : 0x1
v4l2-ctl-5274 [003] … 79509.415531: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1080 fmt 13
<…>-5275 [007] … 79509.425123: vi_task_submit: class_id:48 ch:0 syncpt_id:11 syncpt_thresh:0 pid:5275 tid:5275
vi-output, fpga-5275 [007] … 79509.425142: vi_task_submit: class_id:48 ch:0 syncpt_id:11 syncpt_thresh:0 pid:5275 tid:5275
vi-output, fpga-5275 [007] … 79509.425147: vi_task_submit: class_id:48 ch:0 syncpt_id:11 syncpt_thresh:0 pid:5275 tid:5275
vi-output, fpga-5275 [007] … 79509.425149: vi_task_submit: class_id:48 ch:0 syncpt_id:11 syncpt_thresh:0 pid:5275 tid:5275
v4l2-ctl-5274 [000] … 79509.425313: tegra_channel_set_stream: enable : 0x1
v4l2-ctl-5274 [000] … 79509.425914: tegra_channel_set_stream: 13e00000.host1x:nvcsi@15a00000- : 0x1
v4l2-ctl-5274 [000] … 79509.425917: csi_s_stream: enable : 0x1
v4l2-ctl-5274 [000] … 79509.426224: tegra_channel_set_stream: fpga-media 2-0036 : 0x1
vi-output, fpga-5276 [002] … 79512.162137: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1080 fmt 13
vi-output, fpga-5275 [007] … 79512.162504: vi_task_submit: class_id:48 ch:0 syncpt_id:11 syncpt_thresh:0 pid:5275 tid:5275
vi-output, fpga-5275 [007] … 79512.162520: vi_task_submit: class_id:48 ch:0 syncpt_id:11 syncpt_thresh:0 pid:5275 tid:5275
vi-output, fpga-5275 [007] … 79512.162521: vi_task_submit: class_id:48 ch:0 syncpt_id:11 syncpt_thresh:0 pid:5275 tid:5275
vi-output, fpga-5275 [007] … 79512.162523: vi_task_submit: class_id:48 ch:0 syncpt_id:11 syncpt_thresh:0 pid:5275 tid:5275
vi-output, fpga-5276 [002] … 79514.722351: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1080 fmt 13
vi-output, fpga-5275 [000] … 79514.722565: vi_task_submit: class_id:48 ch:0 syncpt_id:11 syncpt_thresh:0 pid:5275 tid:5275
vi-output, fpga-5275 [000] … 79514.722599: vi_task_submit: class_id:48 ch:0 syncpt_id:11 syncpt_thresh:0 pid:5275 tid:5275
vi-output, fpga-5275 [000] … 79514.722609: vi_task_submit: class_id:48 ch:0 syncpt_id:11 syncpt_thresh:0 pid:5275 tid:5275
vi-output, fpga-5275 [000] … 79514.722611: vi_task_submit: class_id:48 ch:0 syncpt_id:11 syncpt_thresh:0 pid:5275 tid:5275

Sharing dts mode0 value define for FPGA device

mode0 {
mclk_khz = “24000”; //as a dummy
status = “okay”;
num_lanes = “4”;
tegra_sinterface = “serial_a”; //CSI2 port A
phy_mode = “DPHY”;
discontinuous_clk = “no”;
//discontinuous_clk = “yes”;
dpcm_enable = “false”;
csi_pixel_bit_depth = “16”;
mode_type = “yuv”;
pixel_phase = “uyvy”;
cil_settletime = “0”;
active_w = “1920”;
active_h = “1080”;
line_length = “3840”;
inherent_gain = “1”;
mclk_multiplier = “6.67”;
pix_clk_hz = “124416000”; //
//pix_clk_hz = “148500000”; //
gain_factor = “10”;
min_gain_val = “10”;
max_gain_val = “160”;
step_gain_val = “1”;
default_gain = “10”;
min_hdr_ratio = “1”;
max_hdr_ratio = “1”;
framerate_factor = “1000000”;
min_framerate = “1816577”;
max_framerate = “30000000”;
step_framerate = “1”;
default_framerate = “30000000”;
exposure_factor = “1000000”;
min_exp_time = “34”;
max_exp_time = “550385”;
step_exp_time = “1”;
default_exp_time = “33334”;
// embedded_metadata_height = “0”;
//test_pattern_en = “1”;
};

5

Get the trace log by below command.

modprobe rtcpu_debug

echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 3 > /sys/kernel/debug/camrtc/log-level
echo 1 > /sys/kernel/debug/tracing/events/camera_common/enable
echo > /sys/kernel/debug/tracing/trace

v4l2-ctl --stream-mmap -c bypass_mode=0

cat /sys/kernel/debug/tracing/trace
6

result 0f cat /sys/kernel/debug/tracing/trace - (Details log is attached)
tracer_log.txt (25.1 KB)

tegra_channel_set_stream: enable : 0x1
v4l2-ctl-2522 [010] … 610.233013: tegra_channel_set_stream: 13e00000.host1x:nvcsi@15a00000- : 0x1
v4l2-ctl-2522 [010] … 610.233015: csi_s_stream: enable : 0x1
v4l2-ctl-2522 [010] … 610.233323: tegra_channel_set_stream: fpga-media 2-0036 : 0x1
kworker/5:1-83 [005] … 610.273668: rtcpu_string: tstamp:19888315884 id:0x04010000 str:“VM0 activating.”
kworker/5:1-83 [005] … 610.273674: rtcpu_vinotify_event: tstamp:19888793508 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:636430934880 data:0x359d580010000000
kworker/5:1-83 [005] … 610.273675: rtcpu_vinotify_event: tstamp:19888793648 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:636430941280 data:0x0000000031000001
kworker/5:1-83 [005] … 610.273676: rtcpu_vinotify_event: tstamp:19888793805 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:0 vi_tstamp:636430944288 data:0x0000000007020001
kworker/5:1-83 [005] … 610.273677: rtcpu_vinotify_event: tstamp:19888793938 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:636430991424 data:0x359d550010000000
kworker/5:1-83 [005] … 610.273678: rtcpu_vinotify_event: tstamp:19888794094 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:636430997888 data:0x0000000031000002
vi-output, fpga-2524 [010] … 612.962384: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1080 fmt 13
vi-output, fpga-2523 [001] … 612.962783: vi_task_submit: class_id:48 ch:0 syncpt_id:11 syncpt_thresh:0 pid:2523 tid:2523
vi-output, fpga-2523 [001] … 612.962795: vi_task_submit: class_id:48 ch:0 syncpt_id:11 syncpt_thresh:0 pid:2523 tid:2523
vi-output, fpga-2523 [001] … 612.962797: vi_task_submit: class_id:48 ch:0 syncpt_id:11 syncpt_thresh:0 pid:2523 tid:2523
vi-output, fpga-2523 [001] … 612.962800: vi_task_submit: class_id:48 ch:0 syncpt_id:11 syncpt_thresh:0 pid:2523 tid:2523

7

Hi,
Could you please help review what might cause this or guide how to debug further?

8

Looks like NVCSI/VI didn’t receive any validate data from the sensor.

Check the output signal if the port-index/bus-width without problem.

9

After few changes on FPGA side, we are getting below log (Details log is attached), it looks trace is getting sof and eof but it is terminated and issues with payload. please review and guide us how to proceed.
log -
543.916131: tegra_channel_capture_frame: sof:571.999480448
vi-output, fpga-2454 [001] … 543.916132: tegra_channel_capture_frame: eof:572.184024576
vi-output, fpga-2453 [009] … 543.916375: vi_task_submit: class_id:48 ch:0 syncpt_id:11 syncpt_thresh:0 pid:2453 tid:2453
kworker/4:2-180 [004] … 543.969408: rtcpu_nvcsi_intr: tstamp:17880685949 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
kworker/4:2-180 [004] … 543.969410: rtcpu_nvcsi_intr: tstamp:17880685949 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004

543.969469: rtcpu_nvcsi_intr: tstamp:17881038695 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
kworker/4:2-180 [004] … 543.969469: rtcpu_nvcsi_intr: tstamp:17881038695 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
kworker/4:2-180 [004] … 543.969472: rtcpu_vinotify_event: tstamp:17881041707 cch:0 vi:0 tag:CHANSEL_PXL_EOF channel:0x23 frame:0 vi_tstamp:572183928000 data:0x0000000004370002
kworker/4:2-180 [004] … 543.969473: rtcpu_vinotify_event: tstamp:17881041861 cch:0 vi:0 tag:ATOMP_FRAME_DONE channel:0x23 frame:0 vi_tstamp:572183928608 data:0x0000000000000000

543.969475: rtcpu_vinotify_event: tstamp:17881042717 cch:0 vi:0 tag:CHANSEL_FAULT channel:0x23 frame:0 vi_tstamp:572184019296 data:0x0000000004380040
kworker/4:2-180 [004] … 543.969475: rtcpu_vinotify_event: tstamp:17881042851 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:0 vi_tstamp:572184024800 data:0x0000000001020005

kworker/4:2-180 [004] … 544.137387: rtcpu_vinotify_event: tstamp:17886822300 cch:0 vi:0 tag:CHANSEL_SHORT_FRAME channel:0x01 frame:0 vi_tstamp:572367498976 data:0x0000200001000000
kworker/4:2-180 [004] … 544.137388: rtcpu_vinotify_event: tstamp:17886822435 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:0 vi_tstamp:572367525632 data:0x0000000001020006
kworker/4:2-180 [004] … 544.137388: rtcpu_vinotify_event: tstamp:17886822589 cch:0 vi:0 tag:FS channel:0x00 frame:0 vi_tstamp:572367498976 data:0x0000000000000010
kworker/4:2-180 [004] … 544.137389: rtcpu_vinotify_event: tstamp:17886822722 cch:0 vi:0 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:572367498976 data:0x0000000800000000
kworker/4:2-180 [004] … 544.137390: rtcpu_vinotify_event: tstamp:17886822907 cch:0 vi:0 tag:CHANSEL_NOMATCH channel:0x01 frame:0 vi_tstamp:572367500064 data:0x00000000000003c9
kworker/4:2-180 [004] … 544.137390: rtcpu_vinotify_event: tstamp:17886823043 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:572367575744 data:0x359d580010000000
trace-log_pixel-16_13-10.log (8.3 MB)

10

Below tell the CRC error, could be output signal problem.

     kworker/4:2-180     [004] .......   543.129720: rtcpu_nvcsi_intr: tstamp:17856177162 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:0 vc:0 status:0x00000004
11

We are still facing same issues, Here, sharing screen output of signal analyzer . please review —(CSI D0 lane signals without Jetson module connected.)

image
image734×536 190 KB

—CSI D0 lane signals with Jetson module connected

image
image737×538 164 KB

–Ideal CSI signals

image
image777×285 103 KB

13

Hi ,
Is the above output helpful to you?
Could you please share more information about the CRC error — what could be wrong with the signal and how to fix it?
Do you think the issue is coming from the FPGA device/Hardware or something incorrectly defined in the Device Tree?

I’m sharing the mode0 definition data for your review — please check if anything seems incorrect.
Your help and guidance are very important and will be greatly appreciated.

mode0 {
mclk_khz = “24000”; //as a dummy
status = “okay”;
num_lanes = “4”;
tegra_sinterface = “serial_a”; //CSI2 port A
phy_mode = “DPHY”;
discontinuous_clk = “no”;
dpcm_enable = “false”;
csi_pixel_bit_depth = “16”;
mode_type = “yuv”;
pixel_phase = “uyvy”;
cil_settletime = “0”;
active_w = “1920”;
active_h = “1080”;
readout_orientation = “0”;
line_length = “2200”;
inherent_gain = “1”;
mclk_multiplier = “6.18”;
pix_clk_hz = “148500000”;
gain_factor = “10”;
min_gain_val = “10”;
max_gain_val = “160”;
step_gain_val = “1”;
default_gain = “10”;
min_hdr_ratio = “1”;
max_hdr_ratio = “1”;
framerate_factor = “1000000”;
min_framerate = “60000000”;
max_framerate = “60000000”;
step_framerate = “1”;
default_framerate = “60000000”;
//device doesn’t support exposure changes,so use dummy values
exposure_factor = “67475”;
min_exp_time = “15”;
max_exp_time = “16666”;
step_exp_time = “1”;
default_exp_time = “8333”;
embedded_metadata_height = “0”;

};

14

Looks like the CRC may not the key.

Below tell the output size less than expected.

Maybe boost the clocks to try. If still have this message that could be the sensor config problem.

     kworker/4:2-180     [004] .......   544.865650: rtcpu_vinotify_event: tstamp:17909877319 cch:0 vi:0 tag:CHANSEL_SHORT_FRAME channel:0x01 frame:0 vi_tstamp:573105658240 data:0x0000200001000000
15

Below command to boost the clocks.

sudo su
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/nvcsi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/emc/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate |tee /sys/kernel/debug/bpmp/debug/clk/vi/rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate | tee  /sys/kernel/debug/bpmp/debug/clk/isp/rate
cat /sys/kernel/debug/bpmp/debug/clk/nvcsi/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/nvcsi/rate
cat /sys/kernel/debug/bpmp/debug/clk/emc/max_rate | tee /sys/kernel/debug/bpmp/debug/clk/emc/rate
16

We are still seeing the same log messages even after boosting the clock with the command mentioned earlier. Given the mode0 configuration shown above, what configuration fields are most critical to check next? We’ve already tried every relevant hardware change.
dmesg is below -
6657.686044] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 131072
tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 64

17

Confirm if still see CHANSEL_SHORT_FRAME, you can modify the sensor driver report less lines(heigh) one by one to figure out the problem.

18

As per my understanding, we are changing the value of active_h = “1080” as mentioned in the mode0 section of the sensor node in the DTS file.
Do we also need to decrease line_length = “2200” accordingly?

Additionally, is it required to update the driver file as well, where the following structure is defined?

static const struct camera_common_frmfmt fpga_dev_frmfmt = {
{ {1920, 1080}, fpga_dev_60fps, 1, 0, 1 },
};

Here we are define less_lines on active_h like 1076,1072…980 etc but result is same.

19

Doesn’t matter with line_length.

Do you confirm the size by v4l2-ctl after modification?

v4l2-ctl --list-formats-ext
20

Yes, it is confirmed .
Any change in the driver code will be reflected in the v4l2-ctl output.-
static const struct camera_common_frmfmt fpga_dev_frmfmt = {
{ {1920, 1080}, fpga_dev_60fps, 1, 0, 0 },
};

For testing purposes, we changed the FPGA device configuration to use a single CSI lane. After updating the DTS for single-lane operation, we are seeing log messages indicating that some frames are being dropped.

What would be your suggestion for addressing these frame drops?

#v4l2-ctl --stream-mmap -c bypass_mode=0
<<<<<<<<< 22.20 fps, dropped buffers: 16
<<<<<<< 22.20 fps, dropped buffers: 14
<<<<<<<< 22.20 fps, dropped buffers: 16
<<<<<<< 22.20 fps, dropped buffers: 14
<<<<<<<< 22.20 fps, dropped buffers: 16
<<<<<<< 22.20 fps, dropped buffers: 14

21

Any message in dmesg?