Orin nano crashes when calling VIDIOC_STREAMON

I have been working on developing a camera driver and working to get an image on screen. The device tree and driver create the /dev/video0 node but whenever I use any v4l2 command line function to attempt streaming, jetson freezes and then crashes shortly after. I have narrowed it down to whenever the VIDIOC_STREAMON ioctl is called. Is there a direct mapping between this ioctl and a function in my device driver for further debugging?

Nothing is output through the serial log before and after crashing so I didn’t include it.

Hello @richardcusolito1

Could you please share your driver.c and the device tree files?

Regards!
Eduardo Salazar
Embedded SW Engineer at RidgeRun

Contact us: support@ridgerun.com
Developers wiki: https://developer.ridgerun.com/
Website: www.ridgerun.com

Thank you for your response.
nv_oah0428_driver.txt (26.0 KB)
oah0428_dt.txt (10.4 KB)

Hello @richardcusolito1

Thanks for sharing your files.

Here you said that there is no output from the serial log. Does dmesg command out print the following line?

dev_info(dev, "Detected oah0428 sensor\n");

Regards!

Yes, dmesg prints

Detected oah0428 sensor

it also prints

tegra-camrtc-capture-vi tegra-capture-vi: subdev oah0428 9-0036 bound

right before the detected print statement.

Hello @richardcusolito1

Could you please share the output of the commands?

v4l2-ctl --list-devices
v4l2-ctl --list-formats-ext -d /dev/video0
media-ctl -d /dev/media0 -p

Furthermore, what command do you use to try capturing video?

Regards!

Calling

v4l2-ctl --list-devices

produces

NVIDIA Tegra Video Input Device (platform:tegra-camrtc-ca):
	/dev/media0

vi-output, oah0428 10-0036 (platform:tegra-capture-vi:1):
	/dev/video1

vi-output, oah0428 9-0036 (platform:tegra-capture-vi:2):
	/dev/video0

calling

v4l2-ctl --list-formats-ext -d /dev/video1

produces

ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture

	[0]: 'RG10' (10-bit Bayer RGRG/GBGB)
		Size: Discrete 400x400
			Interval: Discrete 0.033s (30.000 fps)

calling

media-ctl -d /dev/media0 -p

produces

Media controller API version 5.15.148

Media device information
------------------------
driver          tegra-camrtc-ca
model           NVIDIA Tegra Video Input Device
serial          
bus info        
hw revision     0x3
driver version  5.15.148

Device topology
- entity 1: 13e00000.host1x:nvcsi@15a00000- (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
	pad0: Sink
		<- "oah0428 10-0036":0 [ENABLED]
	pad1: Source
		-> "vi-output, oah0428 10-0036":0 [ENABLED]

- entity 4: 13e00000.host1x:nvcsi@15a00000- (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev1
	pad0: Sink
		<- "oah0428 9-0036":0 [ENABLED]
	pad1: Source
		-> "vi-output, oah0428 9-0036":0 [ENABLED]

- entity 7: oah0428 9-0036 (1 pad, 1 link)
            type V4L2 subdev subtype Sensor flags 0
            device node name /dev/v4l-subdev2
	pad0: Source
		[fmt:SRGGB10_1X10/400x400 field:none colorspace:srgb]
		-> "13e00000.host1x:nvcsi@15a00000-":0 [ENABLED]

- entity 9: vi-output, oah0428 9-0036 (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video0
	pad0: Sink
		<- "13e00000.host1x:nvcsi@15a00000-":1 [ENABLED]

- entity 23: oah0428 10-0036 (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev3
	pad0: Source
		[fmt:SRGGB10_1X10/400x400 field:none colorspace:srgb]
		-> "13e00000.host1x:nvcsi@15a00000-":0 [ENABLED]

- entity 25: vi-output, oah0428 10-0036 (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video1
	pad0: Sink
		<- "13e00000.host1x:nvcsi@15a00000-":1 [ENABLED]

The command I use to try and capture video that causes a crash is

v4l2-ctl --set-fmt-video=width=400,height=400,pixelformat=RG10 --stream-mmap --stream-count=1 -d /dev/video1 --stream-to=ov6946.raw

I will also add that attempting to start a gstreamer pipeline with the following command

gst-launch-1.0 v4l2src device=/dev/video1 num-buffers=10 ! ‘video/x-bayer,width=400,height=400,format=rggb,framerate=30/1’ ! identity silent=false ! filesink location=test.bayer

It doesn’t cause a crash but it generates this output if its relevant

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
../libs/gst/base/gstbasesrc.c(3127): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:00.000367333
Setting pipeline to NULL ...
Freeing pipeline ...

Hello @richardcusolito1

Thanks for the information provided.

I was checking your device tree. I see more than one node related to tegra-capture-viand host1x@13e00000. Also, in the tegra-camera-platform the compatible field is incorrect, it should be compatible = "nvidia, tegra-camera-platform";. In general, I suggest to check your device tree. You can take tegra234-p3767-camera-p3768-imx477-dual.dts file as a reference. That file is in the following path:

hardware/nvidia/t23x/nv-public/overlay/tegra234-p3767-camera-p3768-imx477-dual.dts

You can download the sources from Jetson Linux | NVIDIA Developer if you are using JetPack 6.1

Regards!

Thank you very much! after implementing your changes it no longer crashes but instead hanges when running the v4l2-ctl command from before.

dmesg produces these errors as it hangs.

[  132.914549] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  132.914579] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  132.915440] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel

I think this error is occuring because of the device tree but I don’t know how to further narrow down the cause of the problem.

Hello @richardcusolito1

Could you please share the new device tree?

Regards!

Hi @richardcusolito1

Did you copy the device tree twice? Or why is there more than one overlay in the attached device tree?

Please, create your custom overlay only for your sensor and only load that overlay!

Also, keep the port-index to <1> for the sensor connected to the channel, port and i2c 0.

Regards!

Sorry I mistakenly copied the tree twice in this forum.

Are you saying I should only define one i2c@X instead of doing i2c@0 and i2c@1?
I was just mimicking the example driver.

Thank you @EduardoSalazar96 !

No. What I am saying is to revert those port-indexes that you changed. Those port-indexes are associated with i2c@0. Let me know if this is clear enough.

Regards!

Yes I have reverted them but again still the same dmesg error that repeats as long as I let the command run for. I probed the CSI data lines going to the CAM#1 port and there is data on them so I am trying to figure out why it times out.

@richardcusolito1

By checking the device tree, some changes need to be applied.

  • for bus@0i2c@0oah0428_a@36 change the tegra_sinterface value from serial_b to serial_a in both modes: mode0 and mode1. Also, for the i2c@0 change the port-index to <0>.
  • Also do the same for port-index to in the following node: oah0428_vi_in0 in tegra-capture-vi and oah0428_csi_in0 in nvcsi@15a00000.

I am assuming that you have two sensors connected to the board. Is this correct?

Share the device tree after applying the above changes so we can check.

Regards!

hello richardcusolito1,

this doesn’t looks correct,
you cannot setup two camera node for sharing same CSI port without virtual channel support.
for instance,

besides, please check Applications Using V4L2 IOCTL Directly for sample pipeline to use V4L2 IOCTL to verify basic camera functionality.

Yes I was just testing some different values. If you look at the most recent device tree I sent you will see the port-index values are different.

I have used the command you linked to verify basic camera functionality and that is generating the uncorr_err dmesg that I sent in the most recent message

hello richardcusolito1,

let me double check which Jetapck release version you’re working with.
you may examine release tag, i.e. $ cat /etc/nv_tegra_release for confirmation.

please refer to Topic 69753 for the mappings of the NVCSI and VI port-index.
you may also narrow down by leaving only single camera connected for driver development.

hello richardcusolito1,

please use V4L2 IOCTL to verify the stream,
for instance,
$ v4l2-ctl -d /dev/video0 --list-formats-ext
$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=400,height=400,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=100

you may also gather the tracing logs for more details.
please load the debug node, i.e. modprobe rtcpu_debug
here’re the steps to enable VI tracing logs.

echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 30720 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/tegra_rtcpu/enable
echo 1 > /sys/kernel/debug/tracing/events/freertos/enable
echo 2 > /sys/kernel/debug/camrtc/log-level
echo > /sys/kernel/debug/tracing/trace
cat /sys/kernel/debug/tracing/trace

$ v4l2-ctl -d /dev/video0 --list-formats-ext

ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture

	[0]: 'RG10' (10-bit Bayer RGRG/GBGB)
		Size: Discrete 400x400
			Interval: Discrete 0.033s (30.000 fps)

$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=400,height=400,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=100
as stated previously this hangs and repeatedly prints the following on dmesg until I ctrl+C

[667022.293523] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[667022.293551] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[667022.294411] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel

Below is what is the trace

^Croot@rcuso-desktop:/home/rcuso/GStreamerTests# cat /sys/kernel/debug/tracing/trace
# tracer: nop
#
# entries-in-buffer/entries-written: 34/34   #P:6
#
#                                _-------=> irqs-off
#                               / _------=> need-resched
#                              | / _-----=> need-resched-lazy
#                              || / _----=> hardirq/softirq
#                              ||| / _---=> preempt-depth
#                              |||| / _--=> preempt-lazy-depth
#                              ||||| / _-=> migrate-disable
#                              |||||| /     delay
#           TASK-PID     CPU#  |||||||  TIMESTAMP  FUNCTION
#              | |         |   |||||||      |         |
        v4l2-ctl-43488   [002] ....... 667230.180914: tegra_channel_open: vi-output, oah0428 10-0036
        v4l2-ctl-43488   [002] ....... 667230.192929: tegra_channel_set_power: oah0428 10-0036 : 0x1
        v4l2-ctl-43488   [002] ....... 667230.193069: camera_common_s_power: status : 0x1
        v4l2-ctl-43488   [002] ....... 667230.199368: tegra_channel_set_power: 13e00000.host1x:nvcsi@15a00000- : 0x1
        v4l2-ctl-43488   [002] ....... 667230.199374: csi_s_power: enable : 0x1
        v4l2-ctl-43488   [002] ....... 667230.200141: tegra_channel_capture_setup: vnc_id 0 W 400 H 400 fmt c4
 vi-output, oah0-43489   [001] ....... 667230.211534: vi_task_submit: class_id:48 ch:0 syncpt_id:28 syncpt_thresh:0 pid:43489 tid:43489
        v4l2-ctl-43488   [002] ....... 667230.211542: tegra_channel_set_stream: enable : 0x1
 vi-output, oah0-43489   [001] ....... 667230.211544: vi_task_submit: class_id:48 ch:0 syncpt_id:28 syncpt_thresh:0 pid:43489 tid:43489
 vi-output, oah0-43489   [001] ....... 667230.211546: vi_task_submit: class_id:48 ch:0 syncpt_id:28 syncpt_thresh:0 pid:43489 tid:43489
 vi-output, oah0-43489   [001] ....... 667230.211547: vi_task_submit: class_id:48 ch:0 syncpt_id:28 syncpt_thresh:0 pid:43489 tid:43489
        v4l2-ctl-43488   [002] ....... 667230.212974: tegra_channel_set_stream: 13e00000.host1x:nvcsi@15a00000- : 0x1
        v4l2-ctl-43488   [002] ....... 667230.212977: csi_s_stream: enable : 0x1
        v4l2-ctl-43488   [002] ....... 667230.213194: tegra_channel_set_stream: oah0428 10-0036 : 0x1
 vi-output, oah0-43490   [005] ....... 667232.751047: tegra_channel_capture_setup: vnc_id 0 W 400 H 400 fmt c4
 vi-output, oah0-43489   [001] ....... 667232.751244: vi_task_submit: class_id:48 ch:0 syncpt_id:28 syncpt_thresh:0 pid:43489 tid:43489
 vi-output, oah0-43489   [001] ....... 667232.751258: vi_task_submit: class_id:48 ch:0 syncpt_id:28 syncpt_thresh:0 pid:43489 tid:43489
 vi-output, oah0-43489   [001] ....... 667232.751260: vi_task_submit: class_id:48 ch:0 syncpt_id:28 syncpt_thresh:0 pid:43489 tid:43489
 vi-output, oah0-43489   [001] ....... 667232.751262: vi_task_submit: class_id:48 ch:0 syncpt_id:28 syncpt_thresh:0 pid:43489 tid:43489
 vi-output, oah0-43490   [005] ....... 667235.311366: tegra_channel_capture_setup: vnc_id 0 W 400 H 400 fmt c4
 vi-output, oah0-43489   [001] ....... 667235.311683: vi_task_submit: class_id:48 ch:0 syncpt_id:28 syncpt_thresh:0 pid:43489 tid:43489
 vi-output, oah0-43489   [001] ....... 667235.311696: vi_task_submit: class_id:48 ch:0 syncpt_id:28 syncpt_thresh:0 pid:43489 tid:43489
 vi-output, oah0-43489   [001] ....... 667235.311698: vi_task_submit: class_id:48 ch:0 syncpt_id:28 syncpt_thresh:0 pid:43489 tid:43489
 vi-output, oah0-43489   [001] ....... 667235.311699: vi_task_submit: class_id:48 ch:0 syncpt_id:28 syncpt_thresh:0 pid:43489 tid:43489
        v4l2-ctl-43488   [004] ....... 667236.158000: tegra_channel_close: vi-output, oah0428 10-0036
 vi-output, oah0-43490   [005] ....... 667237.839141: tegra_channel_capture_setup: vnc_id 0 W 400 H 400 fmt c4
        v4l2-ctl-43488   [004] ....... 667237.839934: tegra_channel_set_stream: enable : 0x0
        v4l2-ctl-43488   [004] ....... 667237.839936: tegra_channel_set_stream: oah0428 10-0036 : 0x0
        v4l2-ctl-43488   [004] ....... 667237.843099: tegra_channel_set_stream: 13e00000.host1x:nvcsi@15a00000- : 0x0
        v4l2-ctl-43488   [004] ....... 667237.843106: csi_s_stream: enable : 0x0
        v4l2-ctl-43488   [003] ....... 667237.845026: tegra_channel_set_power: oah0428 10-0036 : 0x0
        v4l2-ctl-43488   [003] ....... 667237.845036: camera_common_s_power: status : 0x0
        v4l2-ctl-43488   [003] ....... 667237.845070: tegra_channel_set_power: 13e00000.host1x:nvcsi@15a00000- : 0x0
        v4l2-ctl-43488   [003] ....... 667237.845074: csi_s_power: enable : 0x0