How to receive the Embedded data from a YUV camera

Robotics & Edge Computing Jetson & Embedded Systems Jetson Thor
1

Hi,

I have a camera module with oax4000 and max96717, and it can send YUV422(YUYV) 8-bit data(Data Type 0x1e) and embedded data(Data type 0x12) to the thor main board, and the max96724 can receive two type data in one pipeline with mutiple source and destination config. But when I config serdes driver and device tree, the thor VI can’t capture YUV data and embedded data, please help me to analyse, than you very much.dtsi has config:

embedded_metadata_height = “4”; 
mode_type = “yuv”; 
pixel_phase = “yuyv”; 
gmsl-link { 								src-csi-port = "b"; 								dst-csi-port = "a"; 								serdes-csi-link = "b"; 								csi-mode = "1x4"; 								st-vc = <0>; 								vc-id = <1>; 								num-lanes = <4>; 								streams = "yuv16", "embed"; 							};
3

For YUV sensor need to modify the vi5_fops.c to get the embedded data then send to user space.

4

How can modify the vi5_fops.c? Or can you supply the patch? Thank you!

5

Have check below link.

6

This case’s EMD data is the first line of the frame, but mine EMD data’s Data Type is 0x12 which follows the MIPI protocol. And the chan->emb_buf how can userspace get?

7

You need to modify the vi5_fops.c to archive what you want.

Thanks

8

Hi,

Can VI receive different MIPI data type in one nvCSI vc? such as YUV422 8bit (data type: 0x1e) and EMBED 8bit (data type: 0x12).

And I print the embedded data in vi5_capture_dequeue function, but it’s zero.

		wake_up_interruptible(&chan->start_wait);
	/* Read SOF from capture descriptor */
	ts = ns_to_timespec64((s64)descr->status.sof_timestamp);
	trace_tegra_channel_capture_frame("sof", &ts);
	vb->vb2_buf.timestamp = descr->status.sof_timestamp;

	/* mi-0919 debug */
	if (chan->emb_buf_addr) {
    // 方法1: 十六进制转储
    // printk(KERN_INFO "DMA buffer allocated at %p, size %zu\n", 
    //        chan->emb_buf_addr, chan->emb_buf_size);
    print_hex_dump(KERN_INFO, "emb_buf: ", DUMP_PREFIX_OFFSET, 
                   16, 1, chan->emb_buf_addr, 
                   min(chan->emb_buf_size, 128), true);

	}

	if (frame_err)
		buf->vb2_state = VB2_BUF_STATE_ERROR;
	else
		buf->vb2_state = VB2_BUF_STATE_DONE;
	/* Read EOF from capture descriptor */
	ts = ns_to_timespec64((s64)descr->status.eof_timestamp);
	trace_tegra_channel_capture_frame("eof", &ts);

	goto rel_buf;

kern.log:

2025-09-19T17:48:56.137152+08:00 tegra-ubuntu-pos kernel: tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
2025-09-19T17:48:56.175817+08:00 tegra-ubuntu-pos kernel: [RCE] VI ch[35] frame configuration: 2592x1944 
2025-09-19T17:48:56.175824+08:00 tegra-ubuntu-pos kernel: [RCE]     left skip pixels=0 top skip lines=0
2025-09-19T17:48:56.175826+08:00 tegra-ubuntu-pos kernel: [RCE]     right crop pixels=2592 bottom crop lines=1944
2025-09-19T17:48:56.175828+08:00 tegra-ubuntu-pos kernel: [RCE]     pixel format=VI_PIXFMT_FORMAT_T_U8_Y8__V8_Y8 fmt=19
2025-09-19T17:48:56.179820+08:00 tegra-ubuntu-pos kernel: tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 262144
2025-09-19T17:48:56.179826+08:00 tegra-ubuntu-pos kernel: emb_buf: 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
2025-09-19T17:48:56.179828+08:00 tegra-ubuntu-pos kernel: emb_buf: 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
2025-09-19T17:48:56.179829+08:00 tegra-ubuntu-pos kernel: emb_buf: 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
2025-09-19T17:48:56.179831+08:00 tegra-ubuntu-pos kernel: emb_buf: 00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
2025-09-19T17:48:56.179832+08:00 tegra-ubuntu-pos kernel: emb_buf: 00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
2025-09-19T17:48:56.179833+08:00 tegra-ubuntu-pos kernel: emb_buf: 00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
2025-09-19T17:48:56.179835+08:00 tegra-ubuntu-pos kernel: emb_buf: 00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
2025-09-19T17:48:56.179837+08:00 tegra-ubuntu-pos kernel: emb_buf: 00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
2025-09-19T17:48:58.688145+08:00 tegra-ubuntu-pos kernel: tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
2025-09-19T17:48:58.688158+08:00 tegra-ubuntu-pos kernel: tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
2025-09-19T17:48:58.697146+08:00 tegra-ubuntu-pos kernel: tegra194-vi5 8181200000.host1x:vi0@8188400000: vi5_get_syncpt_gos_backing: id=110 addr=0x7ffc6e0000 gos_idx=255 gos_offset=0
2025-09-19T17:48:58.697155+08:00 tegra-ubuntu-pos kernel: tegra194-vi5 8181200000.host1x:vi0@8188400000: vi5_get_syncpt_gos_backing: id=111 addr=0x7ffc6f0000 gos_idx=255 gos_offset=0
2025-09-19T17:48:58.697158+08:00 tegra-ubuntu-pos kernel: tegra194-vi5 8181200000.host1x:vi0@8188400000: vi5_get_syncpt_gos_backing: id=112 addr=0x7ffc700000 gos_idx=255 gos_offset=0
2025-09-19T17:48:58.697160+08:00 tegra-ubuntu-pos kernel: tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
2025-09-19T17:48:58.697162+08:00 tegra-ubuntu-pos kernel: [RCE] VI ch[35] frame configuration: 2592x1944 
2025-09-19T17:48:58.697163+08:00 tegra-ubuntu-pos kernel: [RCE]     left skip pixels=0 top skip lines=0
2025-09-19T17:48:58.697165+08:00 tegra-ubuntu-pos kernel: [RCE]     right crop pixels=2592 bottom crop lines=1944
2025-09-19T17:48:58.697167+08:00 tegra-ubuntu-pos kernel: [RCE]     pixel format=VI_PIXFMT_FORMAT_T_U8_Y8__V8_Y8 fmt=19
2025-09-19T17:48:58.747821+08:00 tegra-ubuntu-pos kernel: tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 262144
2025-09-19T17:48:58.747829+08:00 tegra-ubuntu-pos kernel: emb_buf: 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
2025-09-19T17:48:58.747831+08:00 tegra-ubuntu-pos kernel: emb_buf: 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
2025-09-19T17:48:58.747833+08:00 tegra-ubuntu-pos kernel: emb_buf: 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
2025-09-19T17:48:58.747834+08:00 tegra-ubuntu-pos kernel: emb_buf: 00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
2025-09-19T17:48:58.747836+08:00 tegra-ubuntu-pos kernel: emb_buf: 00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
2025-09-19T17:48:58.747837+08:00 tegra-ubuntu-pos kernel: emb_buf: 00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
2025-09-19T17:48:58.747839+08:00 tegra-ubuntu-pos kernel: emb_buf: 00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
2025-09-19T17:48:58.747841+08:00 tegra-ubuntu-pos kernel: emb_buf: 00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
2025-09-19T17:49:01.504071+08:00 tegra-ubuntu-pos kernel: tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
2025-09-19T17:49:01.504087+08:00 tegra-ubuntu-pos kernel: tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
2025-09-19T17:49:01.513188+08:00 tegra-ubuntu-pos kernel: tegra194-vi5 8181200000.host1x:vi0@8188400000: vi5_get_syncpt_gos_backing: id=110 addr=0x7ffc6e0000 gos_idx=255 gos_offset=0
2025-09-19T17:49:01.513200+08:00 tegra-ubuntu-pos kernel: tegra194-vi5 8181200000.host1x:vi0@8188400000: vi5_get_syncpt_gos_backing: id=111 addr=0x7ffc6f0000 gos_idx=255 gos_offset=0
2025-09-19T17:49:01.513203+08:00 tegra-ubuntu-pos kernel: tegra194-vi5 8181200000.host1x:vi0@8188400000: vi5_get_syncpt_gos_backing: id=112 addr=0x7ffc700000 gos_idx=255 gos_offset=0
2025-09-19T17:49:01.513205+08:00 tegra-ubuntu-pos kernel: tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
9

Yes, it is.

10

When I test the video capture, I found that the VI only can receive one frame, and the VI will report timed out, please help me to analysis it, thanks!

And I config the deserializer register’s like this, is that right?

// debug for embedded data
	max96724_write_reg(dev, 0x933, 0x10); // Un-double EMB8 on ctrl 1
	max96724_write_reg(dev, 0x973, 0x10); // Un-double EMB8 on ctrl 1 (master for port A)
	//Pipe 0 for GMSL-A to CSI-A
	max96724_write_reg(dev, 0x90B, 0xF); // Enable 4 mappings
	max96724_write_reg(dev, 0x90C, 0x0);
	max96724_write_reg(dev, 0x90D, 0x1E); // Src is YUV422, VC0
	max96724_write_reg(dev, 0x90E, 0x1E); // Dst is YUV422, VC0
	max96724_write_reg(dev, 0x90F, 0x0); // Src is FS,     VC0
	max96724_write_reg(dev, 0x910, 0x0); // Dst is FS,     VC0
	max96724_write_reg(dev, 0x911, 0x1); // Src is FE,     VC0
	max96724_write_reg(dev, 0x912, 0x1); // Dst is FE,     VC0
	max96724_write_reg(dev, 0x913, 0x12); // Src is EMB8,      VC0
	max96724_write_reg(dev, 0x914, 0x12); // Dst is EMB8,      VC0
	max96724_write_reg(dev, 0x92D, 0x55); // All mappings to controller 1 (port A)

serializer register like this,

	// test for embedded data
	max96717_write_reg(dev, 0x312, 0x04); // Double EMB8 on pipe Z
	max96717_write_reg(dev, 0x31e, 0x30); // Min BPP = 16 on pipe Z
	max96717_write_reg(dev, 0x111, 0x50); // Max BPP = 16
	max96717_write_reg(dev, 0x110, 0x60); // Disable auto BPP
11

I can’t tell the REG config is correct or not, please check with vendor.

What’s the error after first frame and the trace log.

12

What trace log did you mean? I have supply the kernel log. And how can I to catch the log that you want? Please in detail, I’m a newer, thanks!

13

I have catch the VI trace, is that enough? Please help me to analysis it, thanks!

vi_trace_0922.txt (327.0 KB)

14

The log tell the output size doesn’t as expected.

CHANSEL_SHORT_FRAME
15

What did the output size represent？embed data or yuv image data?

16

The frame size. It could be the height less than driver reported.

17

You mean the yuv422 data’s height in mipi signal is less than the dtsi config? or the emba data’s heigh?

18

The YUV422 data height less than driver reported.

You can confirm the size by “v4l2-ctl –list-formats-ext”

19

Hi,

I execute the command, and the result is:

$ v4l2-ctl -d /dev/side_right_camera -list-formats-ext
Video input set to 0 (Camera 0: Camera, ok)

Camera Controls

                     group_hold 0x009a2003 (bool)   : default=0 value=0 flags=execute-on-write
                     hdr_enable 0x009a2004 (intmenu): min=0 max=1 default=0 value=0 (0 0x0)
                    eeprom_data 0x009a2005 (str)    : min=0 max=1198 step=2 value='efb29defa7c6478f4083c0caa145408f40022b8716d93794404e621058393b8e4011beb43f9097d03fded044328542b4bfbbc54ddbb03ea7bff699d44ccebb8d3f00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000fca9f1d24d004000d9cef75363124000000000000000005bb1bfec9e3cc43f1ff46c567daece3f0000000000000000022e0000001907060e36180000000000000000000000000000000000000000004a5537373030333037303532570000000000000000000000000000004100000000423530584f000000003230413130353043560000000047373137363958414d01195c132c8263000101010000000000000030413230313530313172746c5500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003030464437303931353138343230363444333237394443313130423035303132000000000100000091ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff' flags=read-only, has-payload
                           gain 0x009a2009 (int64)  : min=0 max=301 step=3 default=0 value=0 flags=slider
                       exposure 0x009a200a (int64)  : min=59 max=33334 step=1 default=33333 value=59 flags=slider
                     frame_rate 0x009a200b (int64)  : min=30000000 max=30000000 step=1 default=30000000 value=30000000 flags=slider
                 exposure_short 0x009a200c (int64)  : min=59 max=33333 step=1 default=33333 value=59 flags=slider
           sensor_configuration 0x009a2032 (u32)    : min=0 max=4294967295 step=1 default=0 dims=[22] flags=read-only, volatile, has-payload
         sensor_mode_i2c_packet 0x009a2033 (u32)    : min=0 max=4294967295 step=1 default=0 dims=[1026] flags=read-only, volatile, has-payload
      sensor_control_i2c_packet 0x009a2034 (u32)    : min=0 max=4294967295 step=1 default=0 dims=[1026] flags=read-only, volatile, has-payload
                    bypass_mode 0x009a2064 (intmenu): min=0 max=1 default=0 value=0 (0 0x0)
                override_enable 0x009a2065 (intmenu): min=0 max=1 default=0 value=0 (0 0x0)
                   height_align 0x009a2066 (int)    : min=1 max=16 step=1 default=1 value=1
                     size_align 0x009a2067 (intmenu): min=0 max=2 default=0 value=0 (1 0x1)
               write_isp_format 0x009a2068 (int)    : min=1 max=1 step=1 default=1 value=1
       sensor_signal_properties 0x009a2069 (u32)    : min=0 max=4294967295 step=1 default=0 dims=[30][18] flags=read-only, has-payload
        sensor_image_properties 0x009a206a (u32)    : min=0 max=4294967295 step=1 default=0 dims=[30][16] flags=read-only, has-payload
      sensor_control_properties 0x009a206b (u32)    : min=0 max=4294967295 step=1 default=0 dims=[30][36] flags=read-only, has-payload
              sensor_dv_timings 0x009a206c (u32)    : min=0 max=4294967295 step=1 default=0 dims=[30][16] flags=read-only, has-payload
               low_latency_mode 0x009a206d (bool)   : default=0 value=0
               preferred_stride 0x009a206e (int)    : min=0 max=65535 step=1 default=0 value=0
    override_capture_timeout_ms 0x009a206f (int)    : min=-1 max=2147483647 step=1 default=2500 value=2500
                   sensor_modes 0x009a2082 (int)    : min=0 max=30 step=1 default=30 value=1 flags=read-only

The YUV422 data resolution is:

active_w = "2592";
active_h = "1944";
embedded_metadata_height = "4";

the driver is

static const struct camera_common_frmfmt ox05b1s_frmfmt[] = {
	//{{1920, 1080}, ox05b1s_30fps, 1, 0, OX05B1S_MODE_1920X1080_CROP_30FPS},
	{{2592, 1944}, ox05b1s_30fps, 1, 0, OX05B1S_MODE_2592X1944_CROP_30FPS},
};

Is that right?

20

Have below command to confirm the reported size. (Width/Height)

nvidia@tegra-ubuntu:~$ v4l2-ctl --all
Driver Info:
	Driver name      : tegra-video
	Card type        : vi-output, ov5693 30-0036
	Bus info         : platform:tegra-capture-vi:0
	Driver version   : 5.10.216
	Capabilities     : 0x84200001
		Video Capture
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04200001
		Video Capture
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : tegra-camrtc-ca
	Model            : NVIDIA Tegra Video Input Device
	Serial           :
	Bus info         :
	Media version    : 5.10.216
	Hardware revision: 0x00000003 (3)
	Driver version   : 5.10.216
Interface Info:
	ID               : 0x03000017
	Type             : V4L Video
Entity Info:
	ID               : 0x00000015 (21)
	Name             : vi-output, ov5693 30-0036
	Function         : V4L2 I/O
	Pad 0x01000016   : 0: Sink
	  Link 0x0200001b: from remote pad 0x1000003 of entity '13e10000.host1x:nvcsi@15a00000-': Data, Enabled
Priority: 2
Video input : 0 (Camera 0: ok)
Format Video Capture:
	Width/Height      : 2592/1458
	Pixel Format      : 'BG10' (10-bit Bayer BGBG/GRGR)
21

Hi,

I test and found that, when I set

static const struct camera_common_frmfmt ox05b1s_frmfmt[] = {
	{{2592, 1944}, ox05b1s_30fps, 1, 0, OX05B1S_MODE_2592X1944_CROP_30FPS},
};

Format Video Capture:
	Width/Height      : 2592/1944

the VI report timeout error, but when I set to 2592x1948, the VI can receive image, but the bottom of the image is green line,

static const struct camera_common_frmfmt ox05b1s_frmfmt[] = {
	{{2592, 1948}, ox05b1s_30fps, 1, 0, OX05B1S_MODE_2592X1944_CROP_30FPS},
};

Format Video Capture:
	Width/Height      : 2592/1948