Orin RGB888 Stream

Autonomous Machines Jetson & Embedded Systems Jetson Orin NX
,
1

Hello,

I have an orin nx dev kit that is getting a MIPI stream from an FPGA.

My FPGA is outputting 8 bits per component RGB888_24 stream. (2 Lanes)

However when I try to receive on the orin and save to a file using V4L2 as follows I get:

My device tree:

 rbpcv2_imx219_c@10 {
                                compatible = "sony,imx219";
                                reg = <0x10>;
                                devnode = "video1";
                                physical_w = "3.680";
                                physical_h = "2.760";
                                sensor_model = "imx219";
                                use_sensor_mode_id = "true";
                                reset-gpios = <0x50 0xa0 0x00>;
                                phandle = <0x494>;
                                sensor-bpp = "24";

                                mode0 {
                                        mclk_khz = "400000";
                                        //mclk_khz = "24000";
                                        num_lanes = "2";
                                        tegra_sinterface = "serial_c";
                                        phy_mode = "DPHY";
                                        discontinuous_clk = "no";
                                        dpcm_enable = "false";
                                        cil_settletime = "0";
                                        lane_polarity = "0";
                                        active_w = "1920";
                                        active_h = "1080";
                                        pixel_t = "rgb_rgb88824";
                                        // mode_type = "rgb";
                                        //pixel_phase = "uyvy";
                                        csi_pixel_bit_depth = "24";
                                        //dynamic_pixel_bit_depth = "24";
                                        readout_orientation = "0";
                                        line_length = "1920";
                                        inherent_gain = [31 00];
                                        mclk_multiplier = "9.33";
                                        //pix_clk_hz = "182400000";
                                        //pix_clk_hz = "200000000";
                                        pix_clk_hz = "66666667";
                                        gain_factor = "16";
                                        framerate_factor = "1000000";
                                        exposure_factor = "1000000";
                                        min_gain_val = "16";
                                        max_gain_val = "170";
                                        step_gain_val = [31 00];
                                        default_gain = "16";
                                        min_hdr_ratio = [31 00];
                                        max_hdr_ratio = [31 00];
                                        min_framerate = "2000000";
                                        max_framerate = "30000000";
                                        step_framerate = "1";
                                        default_framerate = "30000000";
                                        min_exp_time = "13";
                                        max_exp_time = "683709";
                                        step_exp_time = [31 00];
                                        default_exp_time = "2495";
                                        embedded_metadata_height = "0";
                                };

                                ports {
                                        #address-cells = <0x01>;
                                        #size-cells = <0x00>;

                                        port@0 {
                                                reg = <0x00>;

                                                endpoint {
                                                        status = "okay";
                                                        port-index = <0x02>;
                                                        bus-width = <0x02>;
                                                        remote-endpoint = <0x2c8>;
                                                        phandle = <0x289>;
                                                };
                                        };
                                };
                        };
                };
        };

I calculated the following with 800 Mhz lane rate * 2 lanes / 24 bits per pixel = 66Mhz pixel clock
800 Mhz lane rate / 2 lane = 400 Mhz Mclk

However I get this error:

[  142.077702] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 4194402
[  142.110172] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 4194402
[  142.142653] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 4194402
[  142.175130] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 4194402
[  142.207619] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 4194402
[  142.240109] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 4194402
[  142.272566] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 4194402
[  142.305049] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 4194402
[  142.337523] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 4194402
[  142.370000] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 4194402
[  142.383740] bwmgr API not supported
[  142.481226] arm-smmu 10000000.iommu: Unhandled context fault: fsr=0x80000402, iova=0x7ffb2a4400, fsynr=0x600013, cbfrsynra=0xc04, cb=1
[  142.497350] mc-err: unknown mcerr fault, int_status=0x00101000, ch_int_status=0x00000000, hubc_int_status=0x00000000 sbs_int_status=0x00000000, hub_int_status=0x00000000

And here is the output of tracing:

     kworker/2:9-162     [002] ....   522.256687: rtcpu_nvcsi_intr: tstamp:16907611281 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256687: rtcpu_nvcsi_intr: tstamp:16907612221 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256687: rtcpu_nvcsi_intr: tstamp:16907612221 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256688: rtcpu_nvcsi_intr: tstamp:16907613161 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256688: rtcpu_nvcsi_intr: tstamp:16907613161 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256688: rtcpu_nvcsi_intr: tstamp:16907614101 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256688: rtcpu_nvcsi_intr: tstamp:16907614101 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256688: rtcpu_nvcsi_intr: tstamp:16907615040 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256689: rtcpu_nvcsi_intr: tstamp:16907615040 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256689: rtcpu_nvcsi_intr: tstamp:16907615980 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256689: rtcpu_nvcsi_intr: tstamp:16907615980 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256689: rtcpu_nvcsi_intr: tstamp:16907616919 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256689: rtcpu_nvcsi_intr: tstamp:16907616919 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256689: rtcpu_nvcsi_intr: tstamp:16907617859 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256690: rtcpu_nvcsi_intr: tstamp:16907617859 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256690: rtcpu_nvcsi_intr: tstamp:16907618799 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256690: rtcpu_nvcsi_intr: tstamp:16907618799 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256690: rtcpu_nvcsi_intr: tstamp:16907619739 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256690: rtcpu_nvcsi_intr: tstamp:16907619739 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256691: rtcpu_nvcsi_intr: tstamp:16907620679 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256691: rtcpu_nvcsi_intr: tstamp:16907620679 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256691: rtcpu_nvcsi_intr: tstamp:16907621618 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256691: rtcpu_nvcsi_intr: tstamp:16907621618 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256691: rtcpu_nvcsi_intr: tstamp:16907622558 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256692: rtcpu_nvcsi_intr: tstamp:16907622558 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256692: rtcpu_nvcsi_intr: tstamp:16907623497 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256692: rtcpu_nvcsi_intr: tstamp:16907623497 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256692: rtcpu_nvcsi_intr: tstamp:16907624437 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256692: rtcpu_nvcsi_intr: tstamp:16907624437 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256693: rtcpu_nvcsi_intr: tstamp:16907625377 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256693: rtcpu_nvcsi_intr: tstamp:16907625377 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256693: rtcpu_nvcsi_intr: tstamp:16907626317 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256693: rtcpu_nvcsi_intr: tstamp:16907626317 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256693: rtcpu_nvcsi_intr: tstamp:16907627256 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256694: rtcpu_nvcsi_intr: tstamp:16907627256 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256694: rtcpu_nvcsi_intr: tstamp:16907628196 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256694: rtcpu_nvcsi_intr: tstamp:16907628196 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256694: rtcpu_nvcsi_intr: tstamp:16907629135 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256694: rtcpu_nvcsi_intr: tstamp:16907629135 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256694: rtcpu_nvcsi_intr: tstamp:16907630075 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256695: rtcpu_nvcsi_intr: tstamp:16907630075 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256695: rtcpu_nvcsi_intr: tstamp:16907631015 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256695: rtcpu_nvcsi_intr: tstamp:16907631015 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256695: rtcpu_nvcsi_intr: tstamp:16907631955 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256695: rtcpu_nvcsi_intr: tstamp:16907631955 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256696: rtcpu_nvcsi_intr: tstamp:16907632894 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256696: rtcpu_nvcsi_intr: tstamp:16907632894 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256696: rtcpu_nvcsi_intr: tstamp:16907633834 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256696: rtcpu_nvcsi_intr: tstamp:16907633834 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256696: rtcpu_nvcsi_intr: tstamp:16907634774 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256697: rtcpu_nvcsi_intr: tstamp:16907634774 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256697: rtcpu_nvcsi_intr: tstamp:16907635713 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256697: rtcpu_nvcsi_intr: tstamp:16907635713 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256697: rtcpu_nvcsi_intr: tstamp:16907636648 class:GLOBAL type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
     kworker/2:9-162     [002] ....   522.256697: rtcpu_nvcsi_intr: tstamp:16907636648 class:CORRECTABLE_ERR type:STREAM_VC phy:0 cil:0 st:2 vc:0 status:0x00000004
        v4l2-ctl-3158    [001] ....   522.256975: tegra_channel_set_power: 13e40000.host1x:nvcsi@15a00000- : 0x0
        v4l2-ctl-3158    [001] ....   522.256979: csi_s_power: enable : 0x0
     kworker/2:9-162     [002] ....   522.375894: rtcpu_vinotify_event: tstamp:16910791085 cch:-1 vi:1 tag:FE channel:0x00 frame:0 vi_tstamp:541145251776 data:0x0000000000000022
     kworker/2:9-162     [002] ....   522.375897: rtcpu_vinotify_event: tstamp:16910791286 cch:-1 vi:1 tag:CHANSEL_SHORT_FRAME channel:0x04 frame:0 vi_tstamp:541145251776 data:0x0173200001000000
     kworker/2:9-162     [002] ....   522.375897: rtcpu_vinotify_event: tstamp:16910791465 cch:-1 vi:1 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:541145251776 data:0x0000000800000000

The command I am trying to use to capture output is:

sudo v4l2-ctl --stream-mmap --stream-count=1 -d /dev/video1 --stream-to=test.raw

Any help here would be appreciated.

3

hello vikram.bhat,

VI did not support RGB888 memory formats due to it’ll add luminance format.
please see-also similar forum threads, such as Topic 289673, and Topic 108194 for reference.

4

Thank you Jerry for your swift reply.

I see, I thought it did because its present in sensor_common.c: linux-tegra-5.10/nvidia/drivers/media/platform/tegra/camera/sensor_common.c at oe4t-patches-l4t-35.6.0 · OE4T/linux-tegra-5.10 · GitHub

	else if (strncmp(pixel_t, "rgb_rgb88824", size) == 0)
		*format = V4L2_PIX_FMT_RGB24;

You’re saying that elsewhere in the pipeline it maps it to add luminence (ARGB?) [Is that what’s happening below?]

nvidia/drivers/media/platform/tegra/camera/vi/vi5_formats.h

static const struct tegra_video_format vi5_video_formats[] = {
...
        /* RGB888 */
        TEGRA_VIDEO_FORMAT(RGB888, 24, RGB888_1X24, 4, 1, T_A8R8G8B8,
                                RGB888, RGBA32, "RGBA-8-8-8-8"),

I found in the orin TRM that it does seem to support RGB888 image format, but I assume that’s the hardware and not kernel 5.10 specifically?

image
image1688×1609 390 KB

Would you recommend we switch the FPGA to a different format, like YUV 422?

5

Also, @JerryChang would you be able to comment on mclk_khz/pixel clock?

I am uncertain if my calculations for them are correct based on 800Mhz line rate.

I see for all sensors that value seems to be 24MHz , but the sensor programming guide says it should be the MIPI driving clock frequency, which I believe is a lot larger than 24MHz (usually closer to GHz).

6

hello vikram.bhat,

(1) it’s TRM to list hardware capability, however, VIC output format only support RGB/BGR with alpha channel. that’s why RGBA is used.
(2) please refer to Sensor Pixel Clock section to review your clock settings.
(3) Here’s another example to check CSI date rate, for instance, 4K(3840x2160),90-fps,10-bit.
CSI data-rate = 3840 * 2160 * 10 * 90 * 1.15 ~= 8.584 Gbps whereas, pix_clk_hz = 8600000000;

7

Thanks @JerryChang .

That makes sense for pixel clock.

Would you say that the value of mclk_khz matters for the orin to obtain and track the MIPI CSI signal?

My understanding is that the receiver will “negoatiate” with the transmitter to get a lock on the clock.

Otherwise it seems like the 24Mhz “Standard MIPI driving clock frequency” would need to be changed to be half the sensor data rate per lane.

8

hello vikram.bhat,

you may leave it as default if you use an internal clock source.
FYI,
the maximum frequency for extperiph1 is 37.09 MHz,
other input clocks, extperiph2, extperiph3, and extperiph4 are limited to 24 MHz.

9

Hi @JerryChang.

Thank you we will leave it as default as this is driven by the FPGA. Thank you.

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.