2 lane CSI custom FPGA with RGB888 output for L4T R28.2.1

Hello all,
I have an FPGA, that outputs 1920x1080@30 2-lane CSI stream in RGB888 format, I am able to recieve this stream, however, the system misinterprets the settings I gave and treats this signal as BGRA (32-bit). Moreover, the output has only 15 fps instead of 30 and has 200ms delay (this is the shortest delay time I was able to achieve by tuning the cil_settletime). My case is similar to this one: https://devtalk.nvidia.com/default/topic/1051967/jetson-tx1/v4l-reports-ar24-instead-of-rgb888/
I browsed and read all relevant forum threads I could find with no success. More details below.

The FPGA device functionality was verified on raspberry Pi, (with slightly modified raspiraw program - i2c operations were bypassed , that’s all AFAIK) since pinout and lane configuration matches raspicam 2.1. Notably, on the RPI the latency was measured at ~130ms, the same monitor was used.

This is the tracing I get on my TX2, note the LOAD_FRAMED it isnt right, is it?:

# tracer: nop
#
# entries-in-buffer/entries-written: 253/253   #P:6
#
#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |
     kworker/0:1-58    [000] ...1    60.156235: rtos_queue_peek_from_isr_failed: tstamp:2232766306 queue:0x0b4a3c58
     kworker/0:1-58    [000] ...1    60.156242: rtcpu_start: tstamp:2232803798
     kworker/0:1-58    [000] ...1    60.156246: rtcpu_vinotify_handle_msg: tstamp:2232911295 tag:CSIMUX_STREAM channel:0xff frame:0 vi_tstamp:2232910875 data:0x00000001
     kworker/0:1-58    [000] ...1    60.205837: rtcpu_vinotify_handle_msg: tstamp:2234243891 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:2234243290 data:0x00000001
     kworker/0:1-58    [000] ...1    60.205841: rtcpu_vinotify_handle_msg: tstamp:2234244111 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:2234243297 data:0x00000000
     kworker/0:1-58    [000] ...1    60.205842: rtcpu_vinotify_handle_msg: tstamp:2234246393 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:0 vi_tstamp:2234245968 data:0x08000000
     kworker/0:1-58    [000] ...1    60.257827: rtcpu_vinotify_handle_msg: tstamp:2235243471 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:2235243061 data:0x04370002
     kworker/0:1-58    [000] ...1    60.257831: rtcpu_vinotify_handle_msg: tstamp:2235247237 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2235246793 data:0x00000000
     kworker/0:1-58    [000] ...1    60.257832: rtcpu_vinotify_handle_msg: tstamp:2236327172 tag:CHANSEL_PXL_SOF channel:0x00 frame:0 vi_tstamp:2236326636 data:0x00000001
     kworker/0:1-58    [000] ...1    60.257833: rtcpu_vinotify_handle_msg: tstamp:2236327321 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:2236326643 data:0x00000000
     kworker/0:1-58    [000] ...1    60.257833: rtcpu_vinotify_handle_msg: tstamp:2236329384 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:0 vi_tstamp:2236328972 data:0x08000000
     kworker/0:1-58    [000] ...1    60.310345: rtcpu_vinotify_handle_msg: tstamp:2237326876 tag:CHANSEL_PXL_EOF channel:0x00 frame:0 vi_tstamp:2237326407 data:0x04370002
     kworker/0:1-58    [000] ...1    60.310348: rtcpu_vinotify_handle_msg: tstamp:2237330611 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:2237330139 data:0x00000000
     kworker/0:1-58    [000] ...1    60.310350: rtos_queue_peek_from_isr_failed: tstamp:2237767178 queue:0x0b4a3c58

This is my relevant device tree setting (you can disregard the comments):

mode0 { 
            mclk_khz = "24000";
            mclk_multiplier = "25";//JG 10 was ok
            //pix_clk_hz = "182400000";// 
            pix_clk_hz = "74250000";            //ok
            //pix_clk_hz = "37125000";
            //pix_clk_hz = "148500000";
            num_lanes = "2";                    //ok
            tegra_sinterface = "serial_a";
            discontinuous_clk = "no";           //ok, clk and lanes run at all times
            cil_settletime = "150";             //not sure, "1" starts @ 60ns and ends @ ~300ns
                                                //according to my calculations must be:
                                                // 127ns < cil_settletime + 5 < 215ns
            pixel_t = "rgb3";                   //ok
            //pixel_t = "bayer_rggb";
            //------------------------------ 
            //mode_type = "Rgb";//jg
            //csi_pixel_bit_depth = "24";//jg
            //pixel_phase = "rgb888";//jg rggb
            //------------------------------
            readout_orientation = "90";         //not important
            inherent_gain = "1";                //not important

            active_w = "1920";                  //ok
            active_h = "1080";                  //ok

            line_length = "2200";               //teoretycznie ok =2200

            dpcm_enable = "false";

            min_gain_val = "1.0";
            max_gain_val = "16";
            min_hdr_ratio = "1";
            max_hdr_ratio = "64";
            min_framerate = "30";//1
            max_framerate = "30";
            //min_exp_time = "33";
            min_exp_time = "11";//11
            max_exp_time = "683709";
            embedded_metadata_height = "0";

          };

The drivers, as well as the device tree settings are based on notes made by Cospandesign (http://cospandesign.github.io/linux,tx2,kernel,driver/2017/12/15/tx2-rpi-camera-port.html) modified in order to match the R28.2.1 version (with driver update guide) and to match the output color format.

My sensor_common.c modifications (based on what I found on the NVIDIA forums):

static int extract_pixel_format(
	const char *pixel_t, u32 *format)
{
	size_t size = strnlen(pixel_t, OF_MAX_STR_LEN);

	if (strncmp(pixel_t, "bayer_bggr10", size) == 0)
		*format = V4L2_PIX_FMT_SBGGR10;
	else if (strncmp(pixel_t, "bayer_rggb10", size) == 0)
		*format = V4L2_PIX_FMT_SRGGB10;
	else if (strncmp(pixel_t, "bayer_bggr12", size) == 0)
		*format = V4L2_PIX_FMT_SBGGR12;
	else if (strncmp(pixel_t, "bayer_rggb12", size) == 0)
		*format = V4L2_PIX_FMT_SRGGB12;
	else if (strncmp(pixel_t, "bayer_wdr_pwl_rggb12", size) == 0)
		*format = V4L2_PIX_FMT_SRGGB12;
	else if (strncmp(pixel_t, "bayer_wdr_dol_rggb10", size) == 0)
		*format = V4L2_PIX_FMT_SRGGB10;
	else if (strncmp(pixel_t, "bayer_xbggr10p", size) == 0)
		*format = V4L2_PIX_FMT_XBGGR10P;
	else if (strncmp(pixel_t, "bayer_xrggb10p", size) == 0)
		*format = V4L2_PIX_FMT_XRGGB10P;
        else if (strncmp(pixel_t, "rgb3", size) == 0)//jg dodano
		*format = V4L2_PIX_FMT_RGB24;
	else {
		pr_err("%s: Need to extend format%s\n", __func__, pixel_t);
		return -EINVAL;
	}

	return 0;
}

My camera_common.c (based on what I found on the NVIDIA forums):

static const struct camera_common_colorfmt camera_common_color_fmts[] = {
	{
		MEDIA_BUS_FMT_SRGGB12_1X12,
		V4L2_COLORSPACE_SRGB,
		V4L2_PIX_FMT_SRGGB12,
	},
	{
		MEDIA_BUS_FMT_SRGGB10_1X10,
		V4L2_COLORSPACE_SRGB,
		V4L2_PIX_FMT_SRGGB10,
	},
	{
		MEDIA_BUS_FMT_SBGGR10_1X10,
		V4L2_COLORSPACE_SRGB,
		V4L2_PIX_FMT_SBGGR10,
	},
	{
		MEDIA_BUS_FMT_SRGGB8_1X8,
		V4L2_COLORSPACE_SRGB,
		V4L2_PIX_FMT_SRGGB8,
	},
	{//JG, DODANO obsluge rgb24
		MEDIA_BUS_FMT_RGB888_1X24,
		V4L2_COLORSPACE_SRGB,
		V4L2_PIX_FMT_RGB24,		
	},
	/*
	 * The below two formats are not supported by VI4,
	 * keep them at the last to ensure they get discarded
	 */
	{
		MEDIA_BUS_FMT_XRGGB10P_3X10,
		V4L2_COLORSPACE_SRGB,
		V4L2_PIX_FMT_XRGGB10P,
	},
	{
		MEDIA_BUS_FMT_XBGGR10P_3X10,
		V4L2_COLORSPACE_SRGB,
		V4L2_PIX_FMT_XRGGB10P,
	},


};

Sample hexdump of a frame captured with v4l2-ctl (note all the the 0xFF as in abovementioned forum thread):

07e8f60 6955 ff92 6854 ff91 6a51 ff92 684f ff90
07e8f70 684f ff92 674e ff91 684f ff90 674e ff8f
07e8f80 674e ff8f 684f ff90 684e ff90 674d ff8f
07e8f90 684b ff8f 684b ff8f 664c ff8e 664c ff8e
07e8fa0 664d ff90 654c ff8f 664c ff8e 664c ff8e
07e8fb0 664d ff8e 654c ff8d 634d ff8c 644e ff8d
07e8fc0 634d ff8c 644e ff8d 634d ff8c 634d ff8c
07e8fd0 644e ff8d 634d ff8c 634d ff8c 624c ff8b
07e8fe0 634a ff8b 634a ff8b 6349 ff8b 6349 ff8b
07e8ff0 634a ff8b 634a ff8b 6249 ff8a 6047 ff88

Here are my questions:

  1. Did anyone manage to get rid of this problem or at least pinpoint the cause of it?
  2. Will this problem disappear if I transfer to newest Jetpack/L4T?

Best regards,
JG

hello jaglaz,

there’re no failures shown in your VI tracing logs.
CHANSEL_LOAD_FRAMED=0x08000000 which bit-27 means the frame is actually loaded by VI engine.
also,
these driver implementation seems correct, did you access with v4l2src for verification.
for example,

$ gst-launch-1.0 v4l2src ! videoconvert ! ximagesink

you may also check similar discussion threads, such as Topic 1046071, and Topic 1015837 for reference.
thanks

Hello JerryChang,
and thanks for the clarification, I thought this was an error.

When I do the pipeline you mentioned, that is:

$ gst-launch-1.0 v4l2src ! videoconvert ! ximagesink

The pipeline seems to assume BGRA color format, and the video looks as if Jetson tried to load 2 frames at once - it loads one frame up to a point, then fills the rest of the screen with another frame.
When I demand BGRA color format explicitly, the effect remains.

However, when I run gst with RGB, that is:

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=RGB,width=1920,height=1080,framerate=30/1 ! timeoverlay ! glimagesink

The abovementioned effect dissappears.

Also, below pipeline verified that video in both cases (format=RGB, format=BGRA) is at 15 fps instead 30.

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=RGB,width=1920,height=1080,framerate=30/1 ! fpsdisplaysink video-sink=glimagesink

It is worth to mention this output of --list-formats:

$ v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'AR24'
	Name        : 32-bit BGRA 8-8-8-8
		Size: Discrete 1920x1080
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 640x480
			Interval: Discrete 0.011s (90.000 fps)

Thanks for the links I will refer to them and double check what I’ve done so far.