V4L2 Time Stamp does not work for L4T 28.1 (Jetpack 3.1)

Our V4L2 based applications do not work after updating to L4T 28.1 and I tracked down it’s V4L time stamp issue.

The V4L2 buffer sequence(frame number), timestamp.tv_sec(second value), timestamp.tv_us (micro-second value) are all zero now.

It seems changes made in L4T 28.1 prevented time stamp of V4L2 buffer being populated.

@yahoo
Does it about encode/decode or camera timestamp?
Could you post you code here to check?

@ShaneCCC,

We do not use ISP for encode/decode, we use V4L2 functions for raw acquisition and timestamps. A link to the example code:

https://stackoverflow.com/questions/10266451/where-does-v4l2-buffer-timestamp-value-starts-counting

i.e.,

struct v4l2_buffer buf;

    //make the v4l call to  xioctl(fd, VIDIOC_DQBUF, &buf)

    //then: 
    long temp_ms = 1000 * buf.timestamp.tv_sec + (long) round(  buf.timestamp.tv_usec / 1000.0);
    long epochTimeStamp_ms = temp_ms + toEpochOffset_ms ;

    printf( "the frame's timestamp in epoch ms is: %ld", epochTimeStamp_ms);

@yahoo
Do query the data from UVC sensor or onboard CSI camera sensor? Could you share you binary if you get the problem with CSI camera sensor? If not it could be the UVC driver issue on kernel-4.4

I was testing onboard OV5693, I tried USB camera today.

There are timestamps for both OV5693 and USB camera under L4T R24.2.1 but no timestamp for either OV5693 or USB camera under L4T R28.1 when the same V4L2 code (listed in previous post) was used.

@yahoo
Just check the Tegra VI driver did report the timestamp by the driver …/kernel/kernel-4.4/drivers/media/platform/tegra/camera/vi/channel.c
Could you print some message here to check.

@ShaneCCC,

Thanks for investigating the issue.

Timestamp in “kernel-4.4/drivers/media/platform/tegra/camera/vi/channel.c” is not filled with values.

In L4T 24.2.1 and earlier, “soc_channel_capture_done(chan, buf, &ts)” is called in “drivers/media/platform/tegra/camera/vi/channel.c” and timestamp is filled in “soc_channel_capture_done” in “/drivers/media/platform/soc_camera/tegra_camera/vi2.c”

static int vi2_channel_capture_done(struct vi2_channel *chan,
				    struct tegra_camera_buffer *buf)
{
	........

	/* Captured one frame */
	do_gettimeofday(&vb->v4l2_buf.timestamp); // Fill V4L2 buffer with kernel time
	vb->v4l2_buf.sequence = chan->sequence++; // Fill V4L2 buffer with frame number
	vb->v4l2_buf.field = V4L2_FIELD_NONE;
	vb2_buffer_done(vb, err < 0 ? VB2_BUF_STATE_ERROR : VB2_BUF_STATE_DONE);

	return err;
}

There is no vi2.c in L4T 28.1.

For the r28.1 should be filled in this function.

void tegra_channel_ring_buffer(struct tegra_channel *chan,
					struct vb2_v4l2_buffer *vb,
					struct timespec *ts, int state)

{
	if (!chan->bfirst_fstart)
		chan->bfirst_fstart = true;
	else
		update_state_to_buffer(chan, state);

	/* Capture state is not GOOD, release all buffers and re-init state */
	if (chan->capture_state != CAPTURE_GOOD) {
		free_ring_buffers(chan, chan->num_buffers);
		tegra_channel_init_ring_buffer(chan);
		return;
	} else {
		/* update time stamp of the buffer */
		vb->timestamp.tv_sec = ts->tv_sec;
		vb->timestamp.tv_usec = ts->tv_nsec / NSEC_PER_USEC;
		pr_info("timestamp sec %ld\n", ts->tv_sec);  //debug
	}

	/* release buffer N at N+2 frame start event */
	if (chan->num_buffers >= (QUEUED_BUFFERS - 1))
		free_ring_buffers(chan, 1);
}

It appears “tegra_channel_ring_buffer” in R28.1 uses different “struct vb2_v4l2_buffer” than “struct v4l2_buffer” used in earlier versions of L4T:

i.e., in R28.1, time stamp is at end of video buffer:

struct vb2_v4l2_buffer {
	struct vb2_buffer	vb2_buf;

	__u32			flags;
	__u32			field;
	struct timeval		timestamp;
	struct v4l2_timecode	timecode;
	__u32			sequence;
};

in R24.2.1, time stamp is at beginning of video buffer:

struct v4l2_buffer {
	__u32			index;
	__u32			type;
	__u32			bytesused;
	__u32			flags;
	__u32			field;
	struct timeval		timestamp;
	struct v4l2_timecode	timecode;
	__u32			sequence;

	/* memory location */
	__u32			memory;
......
}

@yahoo
Thanks for the finding, since the kernel-4.4 have new define /kernel/include/media/videobuf2-v4l2.h I think the user space need to change too. Could you use the vb2_v4l2_buffer instead of v4l2_buffer.

Yes, I’m trying to find the right include path to built V4L2 app on TX1 using “videobuf2-v4l2.h”.

Default include path “/usr/include” of R28.1 does not have “videobuf2-v4l2.h”, I found it in “/usr/src/linux-headers-4.4.38-tegra/include” but am still having issues using “linux-headers-4.4.38-tegra/include” (missing files and other errors).

Hi yahoo2016,

Have you clarified the cause and resolved this file include issue?
Any further result can be shared?

Thanks

We are currently very busy to implement time tagging in FPGAs but will return to investigate L4T R28.1 time tagging issue in few weeks.
Thanks

Hi,
I’m using L4T_R28.2.1/TX2 and facing this issue (timestamp.tv_sec, timestamp.tv_us are all zero) with CSI camera using a simple v4l2 application. Any solutions?

@dasarpmar
Please have a try this patch for 28.2

commit e8f332fb4bca5490ab2c7d7d710da496594d3423
Author: Ahung Cheng <ahcheng@nvidia.com>
Date:   Wed Feb 7 15:37:34 2018 +0800

    media: tegra: fix missing timestamp for vi4
    
    bug 200376318
    
    Change-Id: Ibdd4d17ff5a54d6ed518af41316c05019db3f55b
    Signed-off-by: Ahung Cheng <ahcheng@nvidia.com>
    Reviewed-on: https://git-master.nvidia.com/r/1653384
    Reviewed-by: mobile promotions <svcmobile_promotions@nvidia.com>
    Tested-by: mobile promotions <svcmobile_promotions@nvidia.com>

diff --git a/drivers/media/platform/tegra/camera/vi/channel.c b/drivers/media/platform/tegra/camera/vi/channel.c
index 702309f..43864ea 100644
--- a/drivers/media/platform/tegra/camera/vi/channel.c
+++ b/drivers/media/platform/tegra/camera/vi/channel.c
@@ -46,6 +46,16 @@
 
 static s64 queue_init_ts;
 
+/*
+ * Update the timestamp of the buffer
+ */
+void set_timestamp(struct tegra_channel_buffer *buf,
+				const struct timespec *ts)
+{
+	buf->buf.timestamp.tv_sec = ts->tv_sec;
+	buf->buf.timestamp.tv_usec = ts->tv_nsec / NSEC_PER_USEC;
+}
+
 static void gang_buffer_offsets(struct tegra_channel *chan)
 {
 	int i;
diff --git a/drivers/media/platform/tegra/camera/vi/vi2_fops.c b/drivers/media/platform/tegra/camera/vi/vi2_fops.c
index d8cb190..a1fb024 100644
--- a/drivers/media/platform/tegra/camera/vi/vi2_fops.c
+++ b/drivers/media/platform/tegra/camera/vi/vi2_fops.c
@@ -37,14 +37,8 @@ extern void enqueue_inflight(struct tegra_channel *chan,
 extern struct tegra_channel_buffer *dequeue_inflight(struct tegra_channel *chan);
 extern int tegra_channel_set_power(struct tegra_channel *chan, bool on);
 extern void release_buffer(struct tegra_channel *chan, struct tegra_channel_buffer* buf);
-
-/*
- * Update the timestamp of the buffer
- */
-static void set_timestamp(struct tegra_channel_buffer *buf, const struct timespec* ts) {
-	buf->buf.timestamp.tv_sec = ts->tv_sec;
-	buf->buf.timestamp.tv_usec = ts->tv_nsec / NSEC_PER_USEC;
-}
+extern void set_timestamp(struct tegra_channel_buffer *buf,
+				const struct timespec *ts);
 
 static void vi_write(struct tegra_mc_vi *vi, unsigned int addr, u32 val)
 {
diff --git a/drivers/media/platform/tegra/camera/vi/vi4_fops.c b/drivers/media/platform/tegra/camera/vi/vi4_fops.c
index d1c4936..c7efb6a 100644
--- a/drivers/media/platform/tegra/camera/vi/vi4_fops.c
+++ b/drivers/media/platform/tegra/camera/vi/vi4_fops.c
@@ -52,6 +52,8 @@ static void tegra_channel_error_worker(struct work_struct *status_work);
 static void tegra_channel_notify_error_callback(void *);
 extern void release_buffer(struct tegra_channel *chan,
 				struct tegra_channel_buffer *buf);
+extern void set_timestamp(struct tegra_channel_buffer *buf,
+				const struct timespec *ts);
 
 u32 csimux_config_stream[] = {
 	CSIMUX_CONFIG_STREAM_0,
@@ -547,6 +549,8 @@ static int tegra_channel_capture_frame(struct tegra_channel *chan,
 		chan->capture_state = CAPTURE_GOOD;
 	spin_unlock_irqrestore(&chan->capture_state_lock, flags);
 
+	set_timestamp(buf, &ts);
+
 	if (chan->capture_state == CAPTURE_GOOD) {
 		/*
 		 * Set the buffer version to match
@@ -692,6 +696,7 @@ static void tegra_channel_capture_done(struct tegra_channel *chan)
 
 	/* Mark capture state to IDLE as capture is finished */
 	chan->capture_state = CAPTURE_IDLE;
+	set_timestamp(buf, &ts);
 	release_buffer(chan, buf);
 }

Thanks, ShaneCCC. I can get the timestamp numbers now. Is there any way to confirm that these timestamp values are correct? Comparing these timestamps with the ones of dmesg (running parallelly), I see the dmesg values lag by around 10 secs!!

dmesg later that is should be normally, cause the console print is much low priority. The timestamp is the time vi get the start of frame from sensor I believe it should be correct.

Okay, thanks :)