In version R35.4.1, how do I get the EBD data of the sensor in the driver when using a jetson orin nano device?

FANYX · July 1, 2024, 5:40am

I have referred to the article linked below before asking the question, but, due to the device and software version, I did not go well. Therefore, for the R35.4.1 version, I want to know how to modify which specific aspects of the kernel and drivers, so that I can get EBD data?
[TX2 L4T28.1] How to get embedded data? - Jetson & Embedded Systems / Jetson TX2 - NVIDIA Developer Forums
I mentioned "add below piece of code in tegra_channel_capture_frame() in the above article “vi4_fops.c”, but when I found the tegra_channel_capture_frame() function in vi4_fops.c, I found that the variables used in the added content did not correspond to the parameters of the tegra_channel_capture_frame() function. As follows:
Added content:

void* frm_buffer = vb2_plane_vaddr(&(vb->vb2_buf), 0);
	if(frm_buffer != NULL) {
		memcpy(frm_buffer,chan->vi->emb_buf_addr,chan->vi->emb_buf_size);
	}

The original code for tegra_channel_capture_frame() :

static int tegra_channel_capture_frame(struct tegra_channel *chan,
				struct tegra_channel_buffer *buf)
{
	int ret = 0;

	if (chan->low_latency)
		ret = tegra_channel_capture_frame_multi_thread(chan, buf);
	else
		ret = tegra_channel_capture_frame_single_thread(chan, buf);

	return ret;
}

Therefore, I think this can not achieve my purpose, I hope to get your advice!

ShaneCCC · July 1, 2024, 5:56am

Suppose you can get the embedded data from argus API if your sensor is bayer sensor. If YUV sensor you need to check vi5_fops.c for Orin NX.

Thanks

FANYX · July 1, 2024, 6:04am

Yes, I am using the raw camera for validation, how do I call the argus API to get the EBD data?

ShaneCCC · July 1, 2024, 6:12am

Check argus::Ext::ISensorPrivateMetadata::getMetadata() for it.

FANYX · July 1, 2024, 6:35am

Ok, I’ll try. Also, what parts of the vi5_fops.c file should I check for my YUV camera?

ShaneCCC · July 1, 2024, 10:21am

Get the emb_buf in the vi5_capture_dequeue()

FANYX · July 1, 2024, 10:26am

Is there a history that has been implemented that you can refer to?

ShaneCCC · July 1, 2024, 12:14pm

Please just check the vi6_fops.c to implement it by yourself.

Thanks

FANYX · July 2, 2024, 8:35am

Hello ShaneCCC:
I am currently using the isx031 camera for the ebd data acquisition test, isx031 output uyvy format. I made the following changes to the vi5_capture_dequeue () function in the vi5_fosp.c file:

static void vi5_capture_dequeue(struct tegra_channel *chan,
	struct tegra_channel_buffer *buf)
{
	int err = 0;
	int vi_port = 0;
	int gang_prev_frame_id = 0;
	unsigned long flags;
	struct tegra_mc_vi *vi = chan->vi;
	struct vb2_v4l2_buffer *vb = &buf->buf;

#if KERNEL_VERSION(5, 4, 0) > LINUX_VERSION_CODE
	struct timespec ts;
#else
	struct timespec64 ts;
#endif

	struct capture_descriptor *descr = NULL;
	void* frm_buffer = vb2_plane_vaddr(&(vb->vb2_buf), 0);

	pr_info("%s: a. embedded_buffer_size: %d\n", __func__, chan->emb_buf_size);

	for (vi_port = 0; vi_port < chan->valid_ports; vi_port++) {
		descr = &chan->request[vi_port][buf->capture_descr_index[vi_port]];

		if (buf->vb2_state != VB2_BUF_STATE_ACTIVE)
			goto rel_buf;

		pr_info("%s: b. embedded_buffer_size: %d, vi_port: %d\n", __func__, chan->emb_buf_size, vi_port);
		
		if(frm_buffer != NULL) {
			pr_info("%s: copying the embedded data onto the frame buffer\n", __func__);
			memcpy(frm_buffer,chan->emb_buf_addr,chan->emb_buf_size);
		}

		/* Dequeue a frame and check its capture status */
		err = vi_capture_status(chan->tegra_vi_channel[vi_port], CAPTURE_TIMEOUT_MS);
		if (err) {
			if (err == -ETIMEDOUT) {
				dev_err(vi->dev,
					"uncorr_err: request timed out after %d ms\n",
					CAPTURE_TIMEOUT_MS);
			} else {
				dev_err(vi->dev, "uncorr_err: request err %d\n", err);
			}
			goto uncorr_err;
		} else if (descr->status.status != CAPTURE_STATUS_SUCCESS) {
			if ((descr->status.flags
					& CAPTURE_STATUS_FLAG_CHANNEL_IN_ERROR) != 0) {
				chan->queue_error = true;
				dev_err(vi->dev, "uncorr_err: flags %d, err_data %d\n",
					descr->status.flags, descr->status.err_data);
			} else {
				dev_warn(vi->dev,
					"corr_err: discarding frame %d, flags: %d, "
					"err_data %d\n",
					descr->status.frame_id, descr->status.flags,
					descr->status.err_data);
#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 4, 0)
					buf->vb2_state = VB2_BUF_STATE_REQUEUEING;
#else
					buf->vb2_state = VB2_BUF_STATE_ERROR;
#endif
			goto done;
			}
		} else if (!vi_port) {
			gang_prev_frame_id = descr->status.frame_id;
		} else if (descr->status.frame_id != gang_prev_frame_id) {
			dev_err(vi->dev, "frame_id out of sync: ch2 %d vs ch1 %d\n",
					gang_prev_frame_id, descr->status.frame_id);
			goto uncorr_err;
		}

		pr_info("%s: c. embedded_buffer_size: %d, vi_port: %d\n", __func__, chan->emb_buf_size, vi_port);

		spin_lock_irqsave(&chan->capture_state_lock, flags);
		if (chan->capture_state != CAPTURE_ERROR) {
			chan->capture_reqs_enqueued -= 1;
			chan->capture_state = CAPTURE_GOOD;
		}
		spin_unlock_irqrestore(&chan->capture_state_lock, flags);

		pr_info("%s: d. embedded_buffer_size: %d, vi_port: %d\n", __func__, chan->emb_buf_size, vi_port);
	}

	pr_info("%s: e. embedded_buffer_size: %d, vi_port: %d\n", __func__, chan->emb_buf_size, vi_port);

	wake_up_interruptible(&chan->start_wait);
	/* Read SOF from capture descriptor */
#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 4, 0)
	ts = ns_to_timespec((s64)descr->status.sof_timestamp);
#else
	ts = ns_to_timespec64((s64)descr->status.sof_timestamp);
#endif
#if KERNEL_VERSION(5, 4, 0) > LINUX_VERSION_CODE
	trace_tegra_channel_capture_frame("sof", ts);
#else
	trace_tegra_channel_capture_frame("sof", &ts);
#endif
	vb->vb2_buf.timestamp = descr->status.sof_timestamp;

	buf->vb2_state = VB2_BUF_STATE_DONE;
	/* Read EOF from capture descriptor */
#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 4, 0)
	ts = ns_to_timespec((s64)descr->status.eof_timestamp);
#else
	ts = ns_to_timespec64((s64)descr->status.eof_timestamp);
#endif
#if KERNEL_VERSION(5, 4, 0) > LINUX_VERSION_CODE
	trace_tegra_channel_capture_frame("eof", ts);
#else
	trace_tegra_channel_capture_frame("eof", &ts);
#endif

done:
	goto rel_buf;

uncorr_err:
	spin_lock_irqsave(&chan->capture_state_lock, flags);
	chan->capture_state = CAPTURE_ERROR;
	spin_unlock_irqrestore(&chan->capture_state_lock, flags);

	buf->vb2_state = VB2_BUF_STATE_ERROR;

rel_buf:
	vi5_release_buffer(chan, buf);
}

My current configuration in the device tree is as follows：

mode0 {/*mode SENSOR_MODE_1920X1080_CROP_30FPS*/
						mclk_khz = "24000";
						num_lanes = "4";
						tegra_sinterface = "serial_c";
						phy_mode = "DPHY";
						discontinuous_clk = "no";
						dpcm_enable = "false";
						cil_settletime = "0";
						lane_polarity = "0";

						active_w = "1920";
						active_h = "1536";
						mode_type = "yuv";
						pixel_phase = "uyvy";
						csi_pixel_bit_depth = "16";
						dynamic_pixel_bit_depth = "16";
						readout_orientation = "0";
						line_length = "2200";
						inherent_gain = "1";
						mclk_multiplier = "6.1875";
						pix_clk_hz = "216000000";
						//serdes_pix_clk_hz = "300000000";
											
						gain_factor = "10";
						min_gain_val = "10"; /* dB */
						max_gain_val = "300"; /* dB */
						step_gain_val = "3"; /* 0.3 */
						default_gain = "10";
						min_hdr_ratio = "1";
						max_hdr_ratio = "1";
						framerate_factor = "1000000";
						min_framerate = "30000000";
						max_framerate = "30000000";
						step_framerate = "1";
						default_framerate = "30000000";
						exposure_factor = "1000000";
						min_exp_time = "59"; /*us, 2 lines*/
						max_exp_time = "33333";
						step_exp_time = "1";
						default_exp_time = "33333";/* us */
						embedded_metadata_height = "1";

				};

dmesg information is as follows:

[  109.695763] isx031 10-001a: sensor5 set mode in 0!!!
[  109.725683] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 115, flags: 0, err_data 262144
[  109.736086] vi5_capture_dequeue: a. embedded_buffer_size: 4096
[  109.736088] vi5_capture_dequeue: b. embedded_buffer_size: 4096, vi_port: 0
[  109.736090] vi5_capture_dequeue: copying the embedded data onto the frame buffer
[  112.372070] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  112.381213] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  112.391499] vi5_capture_dequeue: a. embedded_buffer_size: 4096
[  112.391504] vi5_capture_dequeue: a. embedded_buffer_size: 4096
[  112.391513] (NULL device *): vi_capture_control_message: NULL VI channel received
[  112.399263] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=2, csi_port=2
[  112.409968] (NULL device *): vi_capture_control_message: NULL VI channel received
[  112.417705] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 2 vc- 0
[  112.428537] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  112.438107] vi5_capture_dequeue: a. embedded_buffer_size: 4096
[  112.438108] vi5_capture_dequeue: b. embedded_buffer_size: 4096, vi_port: 0
[  112.438110] vi5_capture_dequeue: copying the embedded data onto the frame buffer
[  112.458895] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 197, flags: 0, err_data 262144
[  112.469317] vi5_capture_dequeue: a. embedded_buffer_size: 4096
[  112.469318] vi5_capture_dequeue: b. embedded_buffer_size: 4096, vi_port: 0
[  112.469320] vi5_capture_dequeue: copying the embedded data onto the frame buffer
[  115.188005] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  115.197203] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  115.207228] vi5_capture_dequeue: a. embedded_buffer_size: 4096
[  115.207233] vi5_capture_dequeue: a. embedded_buffer_size: 4096
[  115.207241] (NULL device *): vi_capture_control_message: NULL VI channel received
[  115.214979] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=2, csi_port=2
[  115.225650] (NULL device *): vi_capture_control_message: NULL VI channel received
[  115.233379] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 2 vc- 0
[  115.244202] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  115.253703] vi5_capture_dequeue: a. embedded_buffer_size: 4096
[  115.253705] vi5_capture_dequeue: b. embedded_buffer_size: 4096, vi_port: 0
[  115.253706] vi5_capture_dequeue: copying the embedded data onto the frame buffer
[  115.292304] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 282, flags: 0, err_data 262144
[  115.302681] vi5_capture_dequeue: a. embedded_buffer_size: 4096
[  115.302682] vi5_capture_dequeue: b. embedded_buffer_size: 4096, vi_port: 0
[  115.302684] vi5_capture_dequeue: copying the embedded data onto the frame buffer
[  118.003981] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  118.013122] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  118.023192] vi5_capture_dequeue: a. embedded_buffer_size: 4096
[  118.023196] vi5_capture_dequeue: a. embedded_buffer_size: 4096
[  118.023203] (NULL device *): vi_capture_control_message: NULL VI channel received
[  118.030922] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=2, csi_port=2
[  118.041594] (NULL device *): vi_capture_control_message: NULL VI channel received
[  118.049316] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 2 vc- 0
[  118.060118] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  118.069620] vi5_capture_dequeue: a. embedded_buffer_size: 4096
[  118.069621] vi5_capture_dequeue: b. embedded_buffer_size: 4096, vi_port: 0
[  118.069623] vi5_capture_dequeue: copying the embedded data onto the frame buffer
[  118.092061] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 366, flags: 0, err_data 262144
[  118.102467] vi5_capture_dequeue: a. embedded_buffer_size: 4096
[  118.102469] vi5_capture_dequeue: b. embedded_buffer_size: 4096, vi_port: 0
[  118.102470] vi5_capture_dequeue: copying the embedded data onto the frame buffer
[  120.628027] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  120.637160] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  120.647235] vi5_capture_dequeue: a. embedded_buffer_size: 4096
[  120.647240] vi5_capture_dequeue: a. embedded_buffer_size: 4096
[  120.647246] (NULL device *): vi_capture_control_message: NULL VI channel received
[  120.654972] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=2, csi_port=2
[  120.665649] (NULL device *): vi_capture_control_message: NULL VI channel received
[  120.673369] t194-nvcsi 13e40000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 2 vc- 0
[  120.684158] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel

The trace log is as follows:
trace.txt (255.2 KB)

Currently isx031 only opens the top row of ebd data, and the data format is UYVY. Based on the printed information, I don’t seem to see ebd data being added to frame_buf. According to the information I provide, I hope to get some of your suggestions!

ShaneCCC · July 2, 2024, 8:46am

You can print the EBD data in vi5_fops.c to confirm then try to report to user space later.

Thanks

FANYX · July 3, 2024, 3:19am

Hi ShaneCCC：
When I enable senosr’s ebd data, I print out the data stored in frame_buf.

[   60.769286] vi5_capture_dequeue: frm_buffer_add: 0x8ed0f6e0,embedded data byte[0]:0x0010024c
[   60.769288] vi5_capture_dequeue: frm_buffer_add: 0x151283c8,embedded data byte[1]:0xa6001002
[   60.769290] vi5_capture_dequeue: frm_buffer_add: 0x9f226f48,embedded data byte[2]:0xfea60010
[   60.769291] vi5_capture_dequeue: frm_buffer_add: 0xe43d5cd7,embedded data byte[3]:0xc0fea600
[   60.769293] vi5_capture_dequeue: frm_buffer_add: 0x7daa36a4,embedded data byte[4]:0xb9c0fea6
[   60.769295] vi5_capture_dequeue: frm_buffer_add: 0x5601bf23,embedded data byte[5]:0x23b9c0fe
[   60.769297] vi5_capture_dequeue: frm_buffer_add: 0xf823f02b,embedded data byte[6]:0x0623b9c0
[   60.769299] vi5_capture_dequeue: frm_buffer_add: 0xa9dbc3ad,embedded data byte[7]:0x010623b9
[   60.769301] vi5_capture_dequeue: frm_buffer_add: 0xe5d1f548,embedded data byte[8]:0x11010623

So next, how do I pass the frame_buf data into user space and get that part of the data for parsing?

ShaneCCC · July 3, 2024, 3:56am

You need to figure it out by yourself.

Thanks

system · July 31, 2024, 2:12am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
How do I get the embedded data from sensor out in AGX Xavier? Jetson AGX Xavier camera	14	742	March 25, 2024
Capture image error Jetson Orin Nano camera	20	173	December 3, 2024
When the camera driver was upgraded from r35.4.1 to r35.5.0, a kernel crash occurred when vi could not receive data Jetson AGX Orin camera	9	257	September 20, 2024
Jetson AGX Orin bring up camera bug Jetson AGX Orin camera	6	1554	July 6, 2022
Jetson AGX Orin camera capture error Jetson AGX Orin camera , gstreamer	15	2147	August 10, 2022
Mipi dphy rate 1.5Gbps issue Jetson Orin Nano camera	18	440	May 24, 2024
Ov428 drives tegra-capture-vi timeout (jetpack 5.1.3-b29) Jetson Orin Nano camera	41	351	July 17, 2024
Orin Nano and tc358743 capture issue Jetson Orin Nano camera	101	5183	March 18, 2024
Porting a Custom Camera from AGX Xavier to Orin NX Jetson Orin Nano camera	56	250	April 23, 2025
Xavier NX camera RAW16 Jetson Xavier NX camera	18	1888	October 18, 2021

In version R35.4.1, how do I get the EBD data of the sensor in the driver when using a jetson orin nano device?

Related topics