Bayer Video Feed on OV2740

Dear forum,

Recently, I have been trying to get the Omnivision OV2740 to work with the Jetson Xavier NX through the nvarguscamerasrc for ISP demosaicing. It is in RAW10, grbg format.

I manage to get a frame of working image using this:
v4l2-ctl --stream-mmap --stream-skip=100 --stream-count=1 -d /dev/video0 --stream-to=new-6.raw

However, attempting to use the nvarguscamerasrc for ISP demosaicing is where it fails. Running the following command does not seem to get any response from the camera, giving the following error:

bbd-desktop:~$ gst-launch-1.0 nvarguscamerasrc num-buffers=20 ! fakesink -e -v
[   96.522203] ov2740 10-0036: ov2740_open:
[   96.524043] ov2740 10-0036: ov2740_power_on: power on
[   96.525774] ov2740 10-0036: ov2740_power_off: power off
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstFakeSink:fakesink0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1
[   97.165467] ov2740 10-0036: ov2740_power_on: power on
[   97.165899] ov2740 10-0036: ov2740_power_off: power off
[   97.169739] ov2740 10-0036: ov2740_power_on: power on
[   97.171207] ov2740 10-0036: ov2740_power_off: power off
[   97.205490] ov2740 10-0036: ov2740_power_on: power on
[   97.205787] ov2740 10-0036: ov2740_power_off: power off
[   97.207882] ov2740 10-0036: ov2740_power_on: power on
[   97.208163] ov2740 10-0036: ov2740_power_off: power off
[   97.209516] ov2740 10-0036: ov2740_power_on: power on
[   97.209929] ov2740 10-0036: ov2740_power_off: power off
[   97.211196] ov2740 10-0036: ov2740_power_on: power on
[   97.211541] ov2740 10-0036: ov2740_power_off: power off
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 1920 x 1080 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 0.000000, max 48.000000; Exposure Range min 1000, max 430000000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 0 
   Output Stream W = 1920 H = 1080 
   seconds to Run    = 0 
   Frame Rate = 59.999999 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
[   97.607277] ov2740 10-0036: ov2740_power_on: power on
[   97.616327] ov2740 10-0036: ov2740_set_gain:  gain reg: 160
[   97.863132] ov2740 10-0036: ov2740_set_gain:  gain reg: 160
[   97.863905] ov2740 10-0036: Stream Beginning in Function ov2740_start_streaming
[   99.382081] tegra194-vi5 15c10000.vi: no reply from camera processor
[   99.382275] tegra194-vi5 15c10000.vi: vi capture get status failed
[  102.454035] tegra194-vi5 15c10000.vi: no reply from camera processor
[  102.454212] tegra194-vi5 15c10000.vi: vi capture get status failed
[  105.462074] tegra194-vi5 15c10000.vi: no reply from camera processor
[  105.462241] tegra194-vi5 15c10000.vi: vi capture get status failed
[  106.966078] tegra194-vi5 15c10000.vi: no reply from camera processor
[  106.966253] tegra194-vi5 15c10000.vi: vi capture get status failed
^Chandling interrupt.
Interrupt: Stopping pipeline ...
EOS on shutdown enabled -- Forcing EOS on the pipeline

Here is my dts file (9.9 KB) for the camera that works with v4l2-ctl capture, but not with nvarguscamerasrc.

        vi@15c10000  {
            num-channels = <2>;
            ports {
                #address-cells = <1>;
                #size-cells = <0>;
                vi_port1: port@1 {
                    reg = <1>;
                    ov2740_vi_in1: endpoint {
                        port-index = <2>;
                        bus-width = <2>;
                        remote-endpoint = <&ov2740_csi_out1>;
                    };
                };
            };
        };

        nvcsi@15a00000 {
            num-channels = <2>;
            #address-cells = <1>;
            #size-cells = <0>;
            csi_chan1: channel@1 {
                reg = <1>;
                ports {
                    #address-cells = <1>;
                    #size-cells = <0>;
                    csi_chan1_port0: port@0 {
                        reg = <0>;
                        ov2740_csi_in1: endpoint@2 {
                            port-index = <2>;
                            bus-width = <2>;
                            remote-endpoint = <&ov2740_out1>;
                        };
                    };
                    csi_chan1_port1: port@1 {
                        reg = <1>;
                        ov2740_csi_out1: endpoint@3 {
                            remote-endpoint = <&ov2740_vi_in1>;
                        };
                    };
                };
            };
        };

    cam_i2cmux {
        i2c_1: i2c@1 {
            ov2740_cam1: ov2740@36 {
            
             ...
                ports {
                    #address-cells = <1>;
                    #size-cells = <0>;
                    port@0 {
                        reg = <0>;
                        ov2740_out1: endpoint {
                            status = "okay";
                            port-index = <2>;
                            bus-width = <2>;
                            remote-endpoint = <&ov2740_csi_in1>;
                        };
                    };
                };
            };
        };
    };

};

I ran media_ctl and got this as a response:

bbd@bbd-desktop:~$ media-ctl -p
Media controller API version 0.1.0

Media device information
------------------------
driver          tegra194-vi5
model           NVIDIA Tegra Video Input Device
serial          
bus info        
hw revision     0x3
driver version  0.0.0

Device topology
- entity 1: 15a00000.nvcsi--2 (2 pads, 0 link)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
        pad0: Sink
        pad1: Source

- entity 4: 15a00000.nvcsi--1 (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev1
        pad0: Sink
                <- "ov2740 10-0036":0 [ENABLED]
        pad1: Source
                -> "vi-output, ov2740 10-0036":0 [ENABLED]

- entity 7: ov2740 10-0036 (1 pad, 1 link)
            type V4L2 subdev subtype Sensor flags 0
            device node name /dev/v4l-subdev2
        pad0: Source
[   40.165799] ov2740 10-0036: ov2740_open:
                [fmt:SGRBG10_1X10/1920x1080 field:none colorspace:srgb]
                -> "15a00000.nvcsi--1":0 [ENABLED]

- entity 9: vi-output, ov2740 10-0036 (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video0
        pad0: Sink
                <- "15a00000.nvcsi--1":1 [ENABLED]

Currently, this camera is connected to J9 of the Jetson NX development board - listed as CSI 2 in Figure 2.8 of the Camera Module Hardware Design Guide. This CSI is denoted as CSI-C on the Port Index.
I have updated my dts file (5.4 KB) so that the camera points from CSI-C to Port 2 of CSI to Stream 2 of VI (within this diagram) with the following dts and the same error persists (and now although probe works, /dev/video0 does not show up anymore):

    vi@15c10000  {
        num-channels = <2>;
        ports {
            #address-cells = <1>;
            #size-cells = <0>;
            vi_port_2: port@2 {
                reg = <2>;
                vi_stream_2: endpoint {
                    port-index = <2>;
                    bus-width = <2>;
                    remote-endpoint = <&csi_C_out>;
                };
            };
        };
    };

    nvcsi@15a00000 {
        num-channels = <2>;
        #address-cells = <1>;
        #size-cells = <0>;
        csi_C_channel: channel@2 {
            reg = <2>;	// Port 2(CSI) is connected to CSI-C
            ports {
                #address-cells = <1>;
                #size-cells = <0>;
                csi_C_channel_port0: port@0 {
                    reg = <0>;
                    csi_C_in: endpoint@4 {
                        port-index = <2>;
                        bus-width = <2>;
                        remote-endpoint = <&ov2740_out1>;
                    };
                };
                csi_C_channel_port1: port@1 {
                    reg = <1>;
                    csi_C_out: endpoint@5 {
                        remote-endpoint = <&vi_stream_2>;
                    };
                };
            };
        };
    };

cam_i2cmux {
    i2c_1: i2c@1 {
        ov2740_cam1: ov2740@36 {
            ...

            ports {
                #address-cells = <1>;
                #size-cells = <0>;
                port@0 {
                    reg = <0>;
                    ov2740_out1: endpoint {
                        status = "okay";
                        port-index = <2>;
                        bus-width = <2>;
                        remote-endpoint = <&csi_C_in>;
                    };
                };
            };
        };
    };
};

I might have misunderstood a couple parameter, especially the endpoint indexing. Is there any obvious mistake I am making?

Have a check the exposure/gain/frame rate setting for the v4l2-ctl command and nvarguscamerasrc.
You can make those control function to dummy function to check if those function cause the problem.

So with all of these control functions set to always return true:

static struct tegracam_ctrl_ops ov2740_ctrl_ops = {
	.numctrls = ARRAY_SIZE(ctrl_cid_list),
	.ctrl_cid_list = ctrl_cid_list,
	.string_ctrl_size = {0, OV2740_FUSE_ID_STR_SIZE},
	.set_gain = ov2740_set_gain,
	.set_exposure = ov2740_set_exposure,
	.set_frame_rate = ov2740_set_frame_rate,
	.set_group_hold = ov2740_set_group_hold,
	.fill_string_ctrl = ov2740_fill_string_ctrl,
};
static int ov2740_set_gain(struct tegracam_device *tc_dev, s64 val)
{
	return 0;
}
static int ov2740_set_exposure(struct tegracam_device *tc_dev, s64 val)
{
	return 0;
}
static int ov2740_set_frame_rate(struct tegracam_device *tc_dev, s64 val)
{

	return 0;
}
static int ov2740_set_group_hold(struct tegracam_device *tc_dev, bool val)
{
	return 0;
}
static int ov2740_fill_string_ctrl(struct tegracam_device *tc_dev,
				struct v4l2_ctrl *ctrl)
{
	return 0;
}

I am still receiving the same error:

bbd-desktop:~$ gst-launch-1.0 nvarguscamerasrc num-buffers=20 ! fakesink -e -v
[  125.069314] ov2740 10-0036: ov2740_open:
[  125.072292] ov2740 10-0036: ov2740_power_on: power on
[  125.074706] ov2740 10-0036: ov2740_power_off: power off
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstFakeSink:fakesink0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, format=(string)NV12, framerate=(fraction)30/1
[  125.722243] ov2740 10-0036: ov2740_power_on: power on
[  125.722731] ov2740 10-0036: ov2740_power_off: power off
[  125.726694] ov2740 10-0036: ov2740_power_on: power on
[  125.727054] ov2740 10-0036: ov2740_power_off: power off
[  125.759434] ov2740 10-0036: ov2740_power_on: power on
[  125.759737] ov2740 10-0036: ov2740_power_off: power off
[  125.761930] ov2740 10-0036: ov2740_power_on: power on
[  125.762376] ov2740 10-0036: ov2740_power_off: power off
[  125.763821] ov2740 10-0036: ov2740_power_on: power on
[  125.764152] ov2740 10-0036: ov2740_power_off: power off
[  125.765315] ov2740 10-0036: ov2740_power_on: power on
[  125.765652] ov2740 10-0036: ov2740_power_off: power off
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 1920 x 1080 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 0.000000, max 48.000000; Exposure Range min 1000, max 430000000;

GST_ARGUS: Running with following settings:
Camera index = 0 
Camera mode  = 0 
Output Stream W = 1920 H = 1080 
seconds to Run    = 0 
Frame Rate = 59.999999 
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
[  126.134982] ov2740 10-0036: ov2740_power_on: power on
[  126.384181] ov2740 10-0036: Stream Beginning in Function ov2740_start_streaming
[  127.890039] tegra194-vi5 15c10000.vi: no reply from camera processor
[  127.890226] tegra194-vi5 15c10000.vi: vi capture get status failed
[  129.426088] tegra194-vi5 15c10000.vi: no reply from camera processor
[  129.426281] tegra194-vi5 15c10000.vi: vi capture get status failed
[  132.466031] tegra194-vi5 15c10000.vi: no reply from camera processor
[  132.466044] tegra194-vi5 15c10000.vi: vi capture get status failed
[  133.970106] tegra194-vi5 15c10000.vi: no reply from camera processor
[  133.970266] tegra194-vi5 15c10000.vi: vi capture get status failed

Here are the error.log (40.7 KB) produced by nvargus-daemon, with the following error, maybe this points towards something?

NvPclSettingsApply: Reading PCL settings
(NvCapture) Error Timeout: capture waiting for status timeout (in /dvs/git/dirty/git-master_linux/camera/capture/nvcapture/capture_hw_linux.c, function NvCaptureLinuxGetCaptureStatus(), line 362)
(NvCapture) Error Timeout:  (propagating from /dvs/git/dirty/git-master_linux/camera/capture/nvcapture/capture.c, function NvCaptureDequeue(), line 1055)
SCF: Error Timeout:  (propagating from src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 851)
SCF: Error InvalidState: Timeout waiting on frame end sensor guid 0, capture sequence ID = 1, channel = 1/1 (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 858)
SCF: Error InvalidState: Something went wrong with waiting on frame end (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 939)
SCF: Error InvalidState: Something went wrong with waiting on frame end (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 939)
SCF: Error InvalidState: Timeout waiting on frame start sensor guid 0, capture sequence ID = 1 (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 966)
SCF: Error InvalidState: Something went wrong with waiting on frame start (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 1021)
SCF: Error InvalidState: Something went wrong with waiting on frame start (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 1021)
FiberScheduler: cc 101, session 0, fiber 0x7f48000b20 aborted in async operation

It seems to be waiting for or is having issues with the frame.

After modify the CID function does v4l2-ctl still can capture well?

When running my sample pipeline: nvarguscamerasrc num-buffers=20 ! fakesink, I get this from debugging where it misses the ending of a frame. I have also attached the full log (30.8 KB) for your convenience too.

writeFrameRate: INPUT frameRate:30.000002
updateOutputSettings:   OUTPUT frameRate:30.000000
apply:-----------------------
NvPclSettingsApply: Reading PCL settings
NvPclSettingsUpdate: Sending Updated Settings through PCL
NvPclSettingsApply: Applying last settings through PCL
apply:+++++++++++++++++++++++
apply:-----------------------
NvPclSettingsApply: Reading PCL settings
(NvCapture) Error Timeout: capture waiting for status timeout (in /dvs/git/dirty/git-master_linux/camera/capture/nvcapture/capture_hw_linux.c, function NvCaptureLinuxGetCaptureStatus(), line 362)
(NvCapture) Error Timeout:  (propagating from /dvs/git/dirty/git-master_linux/camera/capture/nvcapture/capture.c, function NvCaptureDequeue(), line 1055)
SCF: Error Timeout:  (propagating from src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 851)
SCF: Error InvalidState: Timeout waiting on frame end sensor guid 0, capture sequence ID = 1, channel = 1/1 (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 858)
SCF: Error InvalidState: Something went wrong with waiting on frame end (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 939)
SCF: Error InvalidState: Something went wrong with waiting on frame end (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 939)
SCF: Error InvalidState: Timeout waiting on frame start sensor guid 0, capture sequence ID = 1 (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 966)
SCF: Error InvalidState: Something went wrong with waiting on frame start (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 1021)
SCF: Error InvalidState: Something went wrong with waiting on frame start (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 1021)
FiberScheduler: cc 101, session 0, fiber 0x7f30000b20 aborted in async operation

And here, I can’t seem to adjust the SensorMode, as it seem to list BayerS16GRBG, when I am attempting to do 10-bit GRBG RAW. I don’t see anywhere obvious to change that too.

InstructionList:
+ GraphSettings
    | SensorMode: 1920x1080 BayerS16GRBG 60.0fps
    | output 0: 1920x1080 BL U8_V8_ER 420SP
    | inputClipRect 0: (0.00,0.00, 1.00,1.00)
    | sharedIsp

+ Instruction List
    | id: 0
    +  0: CCDataSetupStage
        | EstimatedIspOutLatencyFrames: 6
        | NumConcurrentCaptures: 1
        | UnprocessedYuvBufferMask: 0
    +  1: ACSynchronizeStage
    +  2: AeAfApplyStage
    +  3: AcPluginStage
        | operation: opApply
    +  4: AcMergeStage
        | IspIn: [1920, 1080]
        | IspOut0: [1920, 1080]
        | IspOut1: [0, 0]
        | IspOut2: [640, 360]
    +  5: TempBufferAcquireStage
        | Buffer Index: 3
        | BufferRequirements: 1920x1080 Pitch BayerS16GRBG

Something that might be helpful is maybe this debugging trace from /sys/kernel/debug/tracing/trace

root@bbd-desktop:/home/bbd# cat /sys/kernel/debug/tracing/trace
# tracer: nop
#
# entries-in-buffer/entries-written: 5/5   #P:4
#
#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |
    kworker/0:0-8478  [000] ....   861.188191: rtos_queue_peek_from_isr_failed: tstamp:27278143275 queue:0x0bcbcf78
    kworker/0:0-8478  [000] ....   861.356211: rtos_queue_peek_from_isr_failed: tstamp:27283143272 queue:0x0bcbcf78
    kworker/0:0-8478  [000] ....   861.524181: rtos_queue_peek_from_isr_failed: tstamp:27288143273 queue:0x0bcbcf78
    kworker/0:1-8412  [000] ....   861.692196: rtos_queue_peek_from_isr_failed: tstamp:27293143272 queue:0x0bcbcf78
    kworker/0:1-8412  [000] ....   861.860295: rtos_queue_peek_from_isr_failed: tstamp:27298143452 queue:0x0bcbcf78

Attached is the only mode section of my camera.

			mode0 { /* OV2740_MODE_1920x1080_67FPS */
				mclk_khz = "24000";
				num_lanes = "2";
				tegra_sinterface = "serial_c";
				phy_mode = "DPHY";
				discontinuous_clk = "yes";
				dpcm_enable = "false";
				cil_settletime = "0";

				active_w = "1920";
				active_h = "1080";
				pixel_phase = "grbg";
				mode_type = "bayer";
				readout_orientation = "0";
				line_length = "1920"; //To Update
				inherent_gain = "1";
				mclk_multiplier = "18";
				pix_clk_hz = "124416000";

				dynamic_pixel_bit_depth = "10";
				csi_pixel_bit_depth = "10";
				
				gain_factor = "16";
				framerate_factor = "1";
				exposure_factor = "1";
				min_gain_val = "0";  
				max_gain_val = "48"; 
				step_gain_val = "1";
				default_gain = "16"; /* 1.00x */
				min_hdr_ratio = "10";
				max_hdr_ratio = "10";
				min_framerate = "1"; 
				max_framerate = "60"; 
				step_framerate = "1";
				default_framerate = "21000000"; /* 21.0 fps */
				min_exp_time = "1";
				max_exp_time = "430000";
				step_exp_time = "1";
				default_exp_time = "2495"; /* us */	
				
				serdes_pix_clk_hz = "160000000";

				embedded_metadata_height = "0";
			};

Finally. Yes, I ran v4l2src too and it gives me negotiation errors. Plus, I think that v4l2src does not support RAW10.

bbd@bbd-desktop:~$ GST_DEBUG=3 gst-launch-1.0 -v v4l2src ! fakesink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
0:00:00.078124346  9349   0x557bd40630 WARN                 basesrc gstbasesrc.c:3055:gst_base_src_loop:<v4l2src0> error: Internal data stream error.
0:00:00.078198521  9349   0x557bd40630 WARN                 basesrc gstbasesrc.c:3055:gst_base_src_loop:<v4l2src0> error: streaming stopped, reason not-negotiated (-4)
Setting pipeline to PLAYING ...
0:00:00.078782776  9349   0x557bdb1d80 WARN                 v4l2src gstv4l2src.c:692:gst_v4l2src_query:<v4l2src0> Can't give latency since framerate isn't fixated !
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
Execution ended after 0:00:00.000201151
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

Using v4l2-ctl command would be better.

v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=50

Hey, I figured out what’s going on. I had my line_length configured wrongly (too large) so it missed the end-frame of the MIPI transaction. After changing that and it just worked.

Thanks @ShaneCCC for helping out too!