Unable to stream 1080p and 720p simultaneously

Hi,
On our custom board, Orin NX, JP5.1.2
We have two cameras connected to CSI -1 and CSI -3 respectively.
video0 - CSI 1 - 1280x720 (50 fps)
video2 - CSI 2 - 1920x1080 (25fps)

We are unable to stream these simultaneously.
In ov5693_mode_tbls.h :
When i define only 1280x720 under enum , i am able to see the video0 in 720p and also video2 in 720p (while video2 is configured for 1080p in the device tree)
When i define only 1920x1080 under enum , i am able to see video2 in 1080p and also video0 in 1080p (while video0 is configured for 720p in the device tree)

Independently both 720p and 1080p configurations work completely fine.
but if both are define in the mode tables, any v4l2 command for 1080p fails, cause kernel to crash even though there is video dump for 1080p.

While defining both modes:


enum {

        OV5693_MODE_1920X1080,
        OV5693_MODE_1280X720_120FPS,

        OV5693_MODE_START_STREAM,
        OV5693_MODE_STOP_STREAM,
        OV5693_MODE_TEST_PATTERN
};

#ifdef I2C_ENB

static const ov5693_reg *mode_table[] = {
      [OV5693_MODE_2592X1944]                 = mode_2592x1944,
      [OV5693_MODE_2592X1458]                 = mode_2592x1458,
      [OV5693_MODE_1280X720_120FPS]           = mode_1280x720_120fps,
      [OV5693_MODE_640X480]                   = mode_640x480,
      [OV5693_MODE_1920X1080]                 = mode_1920x1080,
      [OV5693_MODE_2592X1944_HDR]             = mode_2592x1944_HDR_24fps,
      [OV5693_MODE_1920X1080_HDR]             = mode_1920x1080_HDR_30fps,
      [OV5693_MODE_2592x1944_15FPS]           = mode_2592x1944_one_lane_15fps,

        [OV5693_MODE_START_STREAM]              = ov5693_start,
        [OV5693_MODE_STOP_STREAM]               = ov5693_stop,
        [OV5693_MODE_TEST_PATTERN]              = tp_colorbars,
};
#endif

static const int ov5693_15fps[] = {
        15,
};

static const int ov5693_24fps[] = {
        24,
};

static const int ov5693_30fps[] = {
        30,
};

static const int ov5693_120fps[] = {
        120,
};

static const struct camera_common_frmfmt ov5693_frmfmt[] = {
        {{1280, 720},   ov5693_120fps,  1, 0,   OV5693_MODE_1280X720_120FPS},
        {{1920, 1080},  ov5693_30fps,   1, 0,   OV5693_MODE_1920X1080},
};

With the above settings, 720p works completely fine on video0, but while trying to stream 1080p on video2 the kernel crashes.
There is video dump on cat /dev/video2.
But gstreamer does not show any video and the kernel crashes.

we tried the below pipelines:

GST_DEBUG=3 gst-launch-1.0 v4l2src device=/dev/video2 ! video/x-raw, width=1920, height=1080 ! videoconvert ! autovideosink -v

GST_DEBUG=3 gst-launch-1.0 v4l2src device=/dev/video2 ! video/x-raw, width=1920, height=1080 ! videoconvert ! ximagesink -v

The above pipeline gives kernel panic and we have to hard reset the Orin.

$ v4l2-ctl -d /dev/video2 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture

	[0]: 'YUYV' (YUYV 4:2:2)
		Size: Discrete 1280x720
			Interval: Discrete 0.008s (120.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.033s (30.000 fps)
	[1]: 'YUYV' (YUYV 4:2:2)
		Size: Discrete 1280x720
			Interval: Discrete 0.008s (120.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.033s (30.000 fps)

We are able to have 1280x720 and 640x480 simultaneously but not 720p and 1080p.

Device tree:
compiled.dts.txt (419.3 KB)

Please assist.
Thanks.

Why report duplicated modes?

@ShaneCCC

Even video0 shows duplicate modes like this and video0 works fine with 720p. Attached is the mode tables settings:

ov5693_mode_tbls.h.txt (41.8 KB)

$ v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture

	[0]: 'YUYV' (YUYV 4:2:2)
		Size: Discrete 1280x720
			Interval: Discrete 0.008s (120.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.033s (30.000 fps)
	[1]: 'YUYV' (YUYV 4:2:2)
		Size: Discrete 1280x720
			Interval: Discrete 0.008s (120.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.033s (30.000 fps)

@ShaneCCC

I observed that the Width/Height : 1280/720 is still 720p for video2 while we have configured for 1080p in the device tree and added the mode for 1920x1080 in mode tables.

$ v4l2-ctl -d /dev/video2 --all
Driver Info:
	Driver name      : tegra-video
	Card type        : vi-output, ov5693 2-0012
	Bus info         : platform:tegra-capture-vi:3
	Driver version   : 5.10.120
	Capabilities     : 0x84200001
		Video Capture
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04200001
		Video Capture
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : tegra-camrtc-ca
	Model            : NVIDIA Tegra Video Input Device
	Serial           : 
	Bus info         : 
	Media version    : 5.10.120
	Hardware revision: 0x00000003 (3)
	Driver version   : 5.10.120
Interface Info:
	ID               : 0x0300002c
	Type             : V4L Video
Entity Info:
	ID               : 0x0000002a (42)
	Name             : vi-output, ov5693 2-0012
	Function         : V4L2 I/O
	Pad 0x0100002b   : 0: Sink
	  Link 0x02000030: from remote pad 0x1000009 of entity '13e40000.host1x:nvcsi@15a00000-': Data, Enabled
Priority: 2
Video input : 0 (Camera 3: ok)
Format Video Capture:
	Width/Height      : 1280/720
	Pixel Format      : 'YUYV' (YUYV 4:2:2)
	Field             : None
	Bytes per Line    : 2560
	Size Image        : 1843200
	Colorspace        : sRGB
	Transfer Function : Default (maps to sRGB)
	YCbCr/HSV Encoding: Default (maps to ITU-R 601)
	Quantization      : Default (maps to Limited Range)
	Flags             : 

Camera Controls

                     group_hold 0x009a2003 (bool)   : default=0 value=0 flags=execute-on-write
                     hdr_enable 0x009a2004 (intmenu): min=0 max=1 default=0 value=0
				0: 0 (0x0)
				1: 1 (0x1)
                    eeprom_data 0x009a2005 (str)    : min=0 max=2048 step=2 value='' flags=read-only, has-payload
                       otp_data 0x009a2006 (str)    : min=0 max=1024 step=2 value='' flags=read-only, has-payload
                        fuse_id 0x009a2007 (str)    : min=0 max=16 step=2 value='' flags=read-only, has-payload
                           gain 0x009a2009 (int64)  : min=9 max=160 step=1 default=10 value=9 flags=slider
                       exposure 0x009a200a (int64)  : min=33 max=550385 step=1 default=33334 value=33 flags=slider
                     frame_rate 0x009a200b (int64)  : min=1816577 max=30000000 step=1 default=30000000 value=1816577 flags=slider
                 exposure_short 0x009a200c (int64)  : min=34 max=550385 step=1 default=33334 value=34 flags=slider
           sensor_configuration 0x009a2032 (u32)    : min=0 max=4294967295 step=1 default=0 [22] flags=read-only, volatile, has-payload
         sensor_mode_i2c_packet 0x009a2033 (u32)    : min=0 max=4294967295 step=1 default=0 [1026] flags=read-only, volatile, has-payload
      sensor_control_i2c_packet 0x009a2034 (u32)    : min=0 max=4294967295 step=1 default=0 [1026] flags=read-only, volatile, has-payload
                    bypass_mode 0x009a2064 (intmenu): min=0 max=1 default=0 value=0
				0: 0 (0x0)
				1: 1 (0x1)
                override_enable 0x009a2065 (intmenu): min=0 max=1 default=0 value=0
				0: 0 (0x0)
				1: 1 (0x1)
                   height_align 0x009a2066 (int)    : min=1 max=16 step=1 default=1 value=1
                     size_align 0x009a2067 (intmenu): min=0 max=2 default=0 value=0
				0: 1 (0x1)
				1: 65536 (0x10000)
				2: 131072 (0x20000)
               write_isp_format 0x009a2068 (int)    : min=1 max=1 step=1 default=1 value=1
       sensor_signal_properties 0x009a2069 (u32)    : min=0 max=4294967295 step=1 default=0 [30][18] flags=read-only, has-payload
        sensor_image_properties 0x009a206a (u32)    : min=0 max=4294967295 step=1 default=0 [30][16] flags=read-only, has-payload
      sensor_control_properties 0x009a206b (u32)    : min=0 max=4294967295 step=1 default=0 [30][36] flags=read-only, has-payload
              sensor_dv_timings 0x009a206c (u32)    : min=0 max=4294967295 step=1 default=0 [30][16] flags=read-only, has-payload
               low_latency_mode 0x009a206d (bool)   : default=0 value=1
               preferred_stride 0x009a206e (int)    : min=0 max=65535 step=1 default=0 value=0
                   sensor_modes 0x009a2082 (int)    : min=0 max=30 step=1 default=30 value=1 flags=read-only

You also need modify the sensor driver but not just modify the device tree.

Thanks

@ShaneCCC

We have included 1080p resolution even the driver. The issue is when both 720p and 1080p are defined in the driver, 720p works but 1080p does not.

I would suspect it could be the sensor driver problem.

@ShaneCCC

We enabled dev_dbg() messages and the 1920x1080 is being taken by the driver on CSI3.

[  948.993638] ov5693 2-0012: camera_common_try_fmt: size 1920 x 1080
[  949.000033] ov5693 2-0012: camera_common_try_fmt: Step 08: current resolution 1920 x 1080
[  949.029509] ov5693 2-0012: v4l2sd_set_fmt(8209) Step6. size 1920 x 1080
[  949.036344] ov5693 2-0012: camera_common_try_fmt: size 1920 x 1080
[  949.042725] ov5693 2-0012: camera_common_try_fmt: Step 08: current resolution 1920 x 1080
[  949.051151] ov5693 2-0012: v4l2sd_set_fmt(8209) Step6. size 1920 x 1080
[  949.057984] ov5693 2-0012: camera_common_s_fmt(8209) size 1920 x 1080
[  949.064612] ov5693 2-0012: camera_common_try_fmt: size 1920 x 1080
[  949.070980] ov5693 2-0012: camera_common_try_fmt: Step 08: current resolution 1920 x 1080
[  949.084057] ov5693 2-0012: camera_common_mclk_enable: enable MCLK with 24000000 Hz
[  949.092301] ov5693 2-0012: camera_common_dpd_disable: csi 3
[  949.110078] bwmgr API not supported
[  949.118863] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000008
[  949.131499] Mem abort info:
[  949.134391]   ESR = 0x96000004
[  949.137541]   EC = 0x25: DABT (current EL), IL = 32 bits
[  949.143002]   SET = 0, FnV = 0
[  949.146148]   EA = 0, S1PTW = 0
[  949.149381] Data abort info:
[  949.152336]   ISV = 0, ISS = 0x00000004
[  949.156287]   CM = 0, WnR = 0
[  949.159346] user pgtable: 4k pages, 48-bit VAs, pgdp=000000013c116000
[  949.165972] [0000000000000008] pgd=0000000000000000, p4d=0000000000000000
client_loop: send disconnect: Broken pipe

The gstreamer pipeline is running in the background and a blank screen appears for 1080p alone . It goes into kernel panic mode and it returns:

Unable to handle kernel NULL pointer dereference at virtual address 0000000000000008

Please assist.

Could you verify by v4l2-ctl

v4l2-ctl --set-fmt-video=width=1920,height=1080 --set-ctrl bypass_mode=0 --stream-mmap

@ShaneCCC

We enabled extra print messages:

On running:

v4l2-ctl -d /dev/video2 --set-fmt-video=width=1920,height=1080,pixelformat=YUYV --set-ctrl bypass_mode=0 --stream-mmap --stream-count=1000

We observe that mode=1 and mode_prop_idx=1
def_width and def_height is 1280x720 instead of 1920x1080

[  563.089765] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_start_streaming: csi_pt=3, st_id=3, vc_id=0, pg_mode=0x0
[  563.093366] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: phy mode unavailable in props, use default
[  563.093371] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_set_config: stream_id=3, csi_port=3
[  563.094165] ov5693 2-0012: csi5_stream_set_config: 
               ops=000000002395ee10, ctrl_handler=0000000080e2c841, dev=00000000d368d46f, frmfmt=00000000125c6a01, colorfmt=00000000326b8b25, debugdir=000000008cc3ed10, power=00000000b32b2421
               subdev.name=ov5693 2-0012, ctrls=0000000003a981c4, owner=0000000000000000
               sensor_props=00000000df48b8d7, tegracam_ctrl_hdl=0000000080e2c841, regmap=000000003a372ab3, pdata=000000004b02aa02, priv=00000000634eb490
               numctrls=8, csi_port=3, numlanes=2, mode=1, mode_prop_idx=1
               numfmts=2, def_mode=0, def_width=1280, def_height=720
               def_clk_freq=24000000, fmt_width=1920, fmt_height=1080
               sensor_mode_id=0, use_sensor_mode_id=0, override_enable=0, version=131078
[  563.094170] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: cil_settingtime is pulled from device
[  563.094173] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: mode not listed in DT, use default
[  563.094179] tegra194-vi5 13e40000.host1x:vi1@14c00000: vi_capture_ivc_send_control: sending chan_id 0 msg_id 64
[  563.094235] tegra194-vi5 13e40000.host1x:vi1@14c00000: vi_capture_ivc_send_control: response chan_id 0 msg_id 65
[  563.094239] tegra194-vi5 13e40000.host1x:vi1@14c00000: vi_capture_ivc_send_control: sending chan_id 0 msg_id 54
[  563.094501] tegra194-vi5 13e40000.host1x:vi1@14c00000: vi_capture_ivc_send_control: response chan_id 0 msg_id 55
[  563.094565] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000008
[  563.103631] Mem abort info:
[  563.106518]   ESR = 0x96000004
[  563.109673]   EC = 0x25: DABT (current EL), IL = 32 bits
[  563.115135]   SET = 0, FnV = 0
[  563.118281]   EA = 0, S1PTW = 0
[  563.121513] Data abort info:
[  563.124468]   ISV = 0, ISS = 0x00000004
[  563.128415]   CM = 0, WnR = 0
[  563.131478] user pgtable: 4k pages, 48-bit VAs, pgdp=000000010e8d6000
[  563.138109] [0000000000000008] pgd=0000000000000000, p4d=0000000000000000

Message from syslogd@TREX2DEV at May 20 06:29:37 ...
 kernel:[  563.145104] Internal error: Oops: 96000004 [#1] PREEMPT SMP

What’s the num_modes? Suppose the numfmts should be 1

@ShaneCCC

num_modes=1
yes numfmts should be 1 has we have defined only one format for CSI 3 but shows 2.

Well, should num_modes =2? 720P and 1080P should be 2

@ShaneCCC

I defined 720p under mode0 and 1080p under mode1 in the dts
We are able to stream the videos at their resolutions.

Thanks.

@ShaneCCC

We get a clear video out for 1080p @ 25 fps
but for 50 fps the video is not as expected:

Please check the sensor REG configuration.

Thanks