Different camera source architecture

vikas.wfh · November 3, 2020, 3:48pm

Hello,

We are developing a 4-lane CSI-2 video card as follows:

a) Platform : TX2
b) Card Architecture: VGA Video Decoder followed by a FPGA converting VGA to CSI-2. Hence, the output of FPGA is connected to TX2 CSI-2 lanes
c) Driver Architecture : The driver has been built for VGA video Decoder which has I2C bus. The FPGA working is fixed and requires no software programming hence no driver built for it. The FPGA has been tested through simulation and is working fine.

Our VGA video decoder driver is configured as follows :

Media Bus format : MEDIA_BUS_FMT_RGB888_1X24
Pixel Format : RGB888

We referred this document to make our board software architecture. As per our understanding we developed our video decoder driver based on V4l2, We have not ported our driver as per your reference for imx185 camera as our driver does not support all the parameters of IMX185 camera_common and sensor_common driver.

In our system video decoder output is parallel RGB888 which is input to the FPGA (Non-configurable). FPGA output is RGB888 which goes on the 4 lanes of the CSI-2 interface.

As in our software architecture, we are not using camera_common and sensor_common and hence we are following V4l2 architecture and not the nvarguscamera source. Please let us know if our software architecture understanding is correct. Is there any necessity to port our driver for Tegra camera platform using nvarguscamera as the reference given for IMX185. If we are using V4l2 architecture for our application where ISP is not required, would there be any other limitation in using V4L2 architecture for our requirement of dual channel Audio/video recording -1600x1200 @60Hz using H.264 HP/H.265 ?

Please suggest if any workarounds possible to make our V4L2 architecture work for our requirement as porting as per nvarguscamera architecture requires considerable time which our project schedule does not allow.

Regards,
Vikas Dwivedi

ShaneCCC · November 4, 2020, 5:57am

For your case if only need support v4l2 you can reference to tc358840.c

vikas.wfh · November 7, 2020, 2:31pm

Hello,

Thank you for suggestion.

We referred tc358840.c and found that some changes is required for pixel format and and frame interval so we made changes accordingly as below:

static int adv7604_enum_framesizes(struct v4l2_subdev *sd,
				struct v4l2_subdev_pad_config *cfg,
				struct v4l2_subdev_frame_size_enum *fse)
{
	const struct camera_common_frmfmt *frmfmt = adv7604_frmfmt;
	int num_frmfmt = ARRAY_SIZE(adv7604_frmfmt);

	v4l2_dbg(2, debug, sd, "%s()\n", __func__);

	if (fse->code != V4L2_PIX_FMT_UYVY &&
		fse->code != V4L2_PIX_FMT_BGR24 &&
		fse->code != V4L2_PIX_FMT_RGB24 &&
		fse->code != V4L2_PIX_FMT_BGR666 &&
		fse->code != V4L2_PIX_FMT_ARGB32 &&
		fse->code != V4L2_PIX_FMT_ABGR32 &&
	    	fse->code != V4L2_PIX_FMT_XBGR32 &&
		fse->code != V4L2_PIX_FMT_XRGB32)
		return -EINVAL;

	if (fse->index >= num_frmfmt)
		return -EINVAL;
	fse->index = array_index_nospec(fse->index, num_frmfmt);

	fse->min_width = fse->max_width = frmfmt[fse->index].size.width;
	fse->min_height = fse->max_height = frmfmt[fse->index].size.height;

	return 0;
}


static int adv7604_enum_frameintervals(struct v4l2_subdev *sd,
				struct v4l2_subdev_pad_config *cfg,
				struct v4l2_subdev_frame_interval_enum *fie)
{
	const struct camera_common_frmfmt *frmfmt = adv7604_frmfmt;
	int num_frmfmt = ARRAY_SIZE(adv7604_frmfmt);
	int i;

	v4l2_dbg(2, debug, sd, "%s()\n", __func__);

	if (fie->code != V4L2_PIX_FMT_UYVY &&
	    fie->code != V4L2_PIX_FMT_BGR24 &&
	    fie->code != V4L2_PIX_FMT_RGB24 &&
	    fie->code != V4L2_PIX_FMT_BGR666 &&
	    fie->code != V4L2_PIX_FMT_ARGB32 &&
	    fie->code != V4L2_PIX_FMT_ABGR32 &&
	    fie->code != V4L2_PIX_FMT_XBGR32 &&
	    fie->code != V4L2_PIX_FMT_XRGB32)
		return -EINVAL;

	for (i = 0; i < num_frmfmt; i++) {
		if (frmfmt[i].size.width == fie->width &&
		    frmfmt[i].size.height == fie->height)
			break;
	}
	if (i >= num_frmfmt)
		return -EINVAL;

	if (fie->index >= frmfmt[i].num_framerates)
		return -EINVAL;

	fie->index = array_index_nospec(fie->index, frmfmt[i].num_framerates);

	fie->interval.numerator = 1;
	fie->interval.denominator = frmfmt[i].framerates[fie->index];

	return 0;
}

Accordingly made changes in header file as follows:

enum {
	ADV7604_MODE_1600X1200,
	ADV7604_MODE_1024X768,
	ADV7604_MODE_800X600,
};

static const int adv7604_30fps[] = {
	30,
};

static const int adv7604_60fps[] = {
	60,
};

static const int adv7604_30_60fps[] = {
	30,
	60,
};

static const struct camera_common_frmfmt adv7604_frmfmt[] = {
	{{1600, 1200},	adv7604_60fps, 1, 1, ADV7604_MODE_1600X1200},
	{{1024, 768},	adv7604_60fps, 1, 1, ADV7604_MODE_1024X768},
	{{800, 600},	adv7604_60fps, 1, 1, ADV7604_MODE_800X600},
};

To add support in camera_common.c I updated:

	{
		MEDIA_BUS_FMT_RGB888_1X24,
		V4L2_COLORSPACE_SRGB,
		V4L2_PIX_FMT_RGB24,
	},
	{
		MEDIA_BUS_FMT_RGB888_1X24,
		V4L2_COLORSPACE_SRGB,
		V4L2_PIX_FMT_BGR24,
	},
	{
		MEDIA_BUS_FMT_RGB888_1X24,
		V4L2_COLORSPACE_SRGB,
		V4L2_PIX_FMT_BGR666,
	},
	{
		MEDIA_BUS_FMT_RGB888_1X24,
		V4L2_COLORSPACE_SRGB,
		V4L2_PIX_FMT_ARGB32,
	},
	{
		MEDIA_BUS_FMT_RGB888_1X24,
		V4L2_COLORSPACE_SRGB,
		V4L2_PIX_FMT_ABGR32,
	},
	{
		MEDIA_BUS_FMT_RGB888_1X24,
		V4L2_COLORSPACE_SRGB,
		V4L2_PIX_FMT_XBGR32,
	},
	{
		MEDIA_BUS_FMT_RGB888_1X24,
		V4L2_COLORSPACE_SRGB,
		V4L2_PIX_FMT_XRGB32,
	},

Accordingly changes in sensor_common.c :

	else if (strncmp(pixel_t, "rgb_rgb24", size) == 0)
		*format = V4L2_PIX_FMT_RGB24;
	else if (strncmp(pixel_t, "rgb_bgr24", size) == 0)
		*format = V4L2_PIX_FMT_BGR24;
	else if (strncmp(pixel_t, "rgb_bgrh", size) == 0)
		*format = V4L2_PIX_FMT_BGR666;
	else if (strncmp(pixel_t, "rgb_argb32", size) == 0)
		*format = V4L2_PIX_FMT_ARGB32;
	else if (strncmp(pixel_t, "rgb_abgr32", size) == 0)
		*format = V4L2_PIX_FMT_ABGR32;
	else if (strncmp(pixel_t, "rgb_xbgr32", size) == 0)
		*format = V4L2_PIX_FMT_XBGR32;
	else if (strncmp(pixel_t, "rgb_xrgb32", size) == 0)
		*format = V4L2_PIX_FMT_XRGB32;
	else {
		pr_err("%s: Need to extend format%s\n", __func__, pixel_t);
		return -EINVAL;
	}

After all this changes I am running following pipeline:

sds@sds-desktop:~$ gst-launch-1.0 v4l2src device="/dev/video0" ! 'video/x-raw,format=BGRA,width=1600,height=1200' ! videoconvert ! glimagesink -e
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Got context from element 'sink': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayX11\)\ gldisplayx11-0";
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Device '/dev/video0' has no supported format
Additional debug info:
gstv4l2object.c(3760): gst_v4l2_object_set_format_full (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
Call to S_FMT failed for AR24 @ 1600x1200: Invalid argument
EOS on shutdown enabled -- waiting for EOS after Error
Waiting for EOS...
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Interrupt while waiting for EOS - stopping pipeline...
Execution ended after 0:00:01.372031872
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
sds@sds-desktop:~$

It is giving pixel format is not supported error. As we know VI engine does not support RGB888 directly. some conversion is required for the same but where this conversion happens and what else we need to make changes for RGB888.

Here is log for list of formats supported:

sds@sds-desktop:~$ v4l2-ctl -d /dev/video0 --all --list-formats-ext
Driver Info (not using libv4l2):
	Driver name   : tegra-video
	Card type     : vi-output, adv7604 30-0020
	Bus info      : platform:15700000.vi:0
	Driver version: 4.9.140
	Capabilities  : 0x84200001
		Video Capture
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps   : 0x04200001
		Video Capture
		Streaming
		Extended Pix Format
Priority: 2
Video input : 0 (HDMI 0: ok)
DV timings:
	Active width: 1600
	Active height: 1200
	Total width: 2160
	Total height: 1250
	Frame format: progressive
	Polarities: +vsync +hsync
	Pixelclock: 162000000 Hz (60.00 frames per second)
	Horizontal frontporch: 64
	Horizontal sync: 192
	Horizontal backporch: 304
	Vertical frontporch: 1
	Vertical sync: 3
	Vertical backporch: 46
	Standards: DMT
	Flags: 
DV timings capabilities:
	Minimum Width: 0
	Maximum Width: 1920
	Minimum Height: 0
	Maximum Height: 1200
	Minimum PClock: 25000000
	Maximum PClock: 225000000
	Standards: CTA-861, DMT, CVT, GTF
	Capabilities: Progressive, Reduced Blanking, Custom Formats
Format Video Capture:
	Width/Height      : 1600/1200
	Pixel Format      : 'RGB3'
	Field             : Any
	Bytes per Line    : 6400
	Size Image        : 7680000
	Colorspace        : Default
	Transfer Function : Default (maps to Rec. 709)
	YCbCr/HSV Encoding: Default (maps to ITU-R 601)
	Quantization      : Default (maps to Full Range)
	Flags             : 

User Controls

                 brightness 0x00980900 (int)    : min=-128 max=127 step=1 default=0 value=0 flags=slider
                   contrast 0x00980901 (int)    : min=0 max=255 step=1 default=128 value=128 flags=slider
                 saturation 0x00980902 (int)    : min=0 max=255 step=1 default=128 value=128 flags=slider
                        hue 0x00980903 (int)    : min=0 max=128 step=1 default=0 value=0 flags=slider

Camera Controls

                bypass_mode 0x009a2064 (intmenu): min=0 max=1 default=0 value=0
            override_enable 0x009a2065 (intmenu): min=0 max=1 default=0 value=0
               height_align 0x009a2066 (int)    : min=1 max=16 step=1 default=1 value=1
                 size_align 0x009a2067 (intmenu): min=0 max=2 default=0 value=0
           write_isp_format 0x009a2068 (int)    : min=1 max=1 step=1 default=1 value=1
   sensor_signal_properties 0x009a2069 (u32)    : min=0 max=0 step=0 default=0 flags=read-only, has-payload
    sensor_image_properties 0x009a206a (u32)    : min=0 max=0 step=0 default=0 flags=read-only, has-payload
  sensor_control_properties 0x009a206b (u32)    : min=0 max=0 step=0 default=0 flags=read-only, has-payload
          sensor_dv_timings 0x009a206c (u32)    : min=0 max=0 step=0 default=0 flags=read-only, has-payload
           low_latency_mode 0x009a206d (bool)   : default=0 value=0
               sensor_modes 0x009a2082 (int)    : min=0 max=30 step=1 default=30 value=30 flags=read-only

Digital Video Controls

              power_present 0x00a00964 (bitmask): max=0x0000000f default=0x00000000 value=0x00000000 flags=read-only
  rx_rgb_quantization_range 0x00a00965 (menu)   : min=0 max=2 default=0 value=0
         rx_it_content_type 0x00a00966 (menu)   : min=0 max=4 default=4 value=4 flags=read-only, volatile
      analog_sampling_phase 0x00a01900 (int)    : min=0 max=31 step=1 default=0 value=0
  free_running_color_manual 0x00a01901 (bool)   : default=0 value=0
         free_running_color 0x00a01902 (int)    : min=0 max=16777215 step=1 default=0 value=0
ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'AR24'
	Name        : 32-bit BGRA 8-8-8-8

	Index       : 1
	Type        : Video Capture
	Pixel Format: 'RGB3'
	Name        : 24-bit RGB 8-8-8

	Index       : 2
	Type        : Video Capture
	Pixel Format: 'UYVY'
	Name        : UYVY 4:2:2

	Index       : 3
	Type        : Video Capture
	Pixel Format: 'VYUY'
	Name        : VYUY 4:2:2

	Index       : 4
	Type        : Video Capture
	Pixel Format: 'YUYV'
	Name        : YUYV 4:2:2

	Index       : 5
	Type        : Video Capture
	Pixel Format: 'YVYU'
	Name        : YVYU 4:2:2

	Index       : 6
	Type        : Video Capture
	Pixel Format: 'NV16'
	Name        : Y/CbCr 4:2:2

	Index       : 7
	Type        : Video Capture
	Pixel Format: 'YUYV'
	Name        : YUYV 4:2:2

	Index       : 8
	Type        : Video Capture
	Pixel Format: 'YVYU'
	Name        : YVYU 4:2:2

I am not able to understand why VIDIOC_ENUM_FMT is showing 8 indexes and AR24 pixel format. VIDIOC_ENUM_FMT ioctl call is also not showing my frame format table.

Looking forward for your guidance.

Thanks and Regards,
Vikas Dwivedi

ShaneCCC · November 8, 2020, 3:45pm

Could you check with v4l2-ctl command first.

v4l2-ctl -d /dev/video0 --set-fmt-video=width=1600,height=1200–set-ctrl bypass_mode=0 --stream-mmap --stream-count=1 --stream-to=ov1080.raw

Topic		Replies	Views
Tegra VI platform understanding Jetson TX2 camera	2	1277	October 18, 2021
TX2 video input that is not an i2c device. Jetson TX2	15	2546	October 18, 2021
Can not render camera preview with L4T R28.2.1 Jetson TX2	21	2350	October 18, 2021
Camera aggregator driver NVP6324 Jetson TX2 camera	4	2123	October 18, 2021
How could I use nvarguscamerasrc plugin with raw12 ? Jetson TX2	20	2734	October 18, 2021
Video driver port from TX2 NX (JP4.6.4) to Orin NX (JP5.1.3) Jetson Orin NX camera , kernel	16	337	June 27, 2024
Help to understand NVP6324 integration error at Jetson TX2 NX Jetson TX2 camera	2	1004	October 18, 2021
How to capture 4 chans video data through one csi interface based ov5693_i2c driver on jetson tx1 board Jetson TX1	12	2135	October 18, 2021
Why get video data error through ov4689 sensor（2 lane mipi）？？ Jetson AGX Xavier	21	2175	October 18, 2021
Jetson Xavier yuv422 camera driver [one camera] no reply from camera processor Jetson AGX Xavier camera	35	3512	October 17, 2021

Different camera source architecture

Related topics