We have a high performance app running on a Jetson Nano based on stereo cameras, that uses the timestamp on the v4l2 image buffer to match frames from the two cameras. The app also uses the time difference between the buffer timestamp and the time at which the application first receives the buffer as an approximate measure of the “transfer time”, which is the the time taken to transfer the image over the mipi bus.
We recently updated from the older version of L4T to a more recent one, and found that the frame matching was unreliable, and the transfer time had increased from roughly 1.5ms to 12ms, and now seemed to be dependent on the frame rate.
Investigation showed some changes were made to the vi2 driver between V32.6.1 and V32.7.1 of L4T. The significant one for our purposes was this change in the file vi2_fops.c, function tegra_channel_capture_frame_multi_thread, lines 625-644 in the original file, lines 612 -613 in the new version:
chan->capture_state = CAPTURE_GOOD;
- for (index = 0; index < valid_ports; index++) {
- err = nvhost_syncpt_wait_timeout_ext(chan->vi->ndev,
- chan->syncpt[index][0], thresh[index],
- chan->timeout, NULL, &ts);
- if (err) {
- dev_err(&chan->video->dev,
- "frame start syncpt timeout!%d\n", index);
- buf->state = VB2_BUF_STATE_REQUEUEING;
- /* perform error recovery for timeout */
- tegra_channel_ec_recover(chan);
- chan->capture_state = CAPTURE_TIMEOUT;
- break;
- }
-
- dev_dbg(&chan->video->dev,
- "%s: vi2 got SOF syncpt buf[%p]\n", __func__, buf);
- }
-
getrawmonotonic(&ts);
The old code used to wait for the SOF syncpt before applying the time stamp and queuing the buffer. The new code applies the time stamp and queues the buffer without waiting. The time stamp is now usually applied a couple of frame periods before the buffer is used.
The effect of the change can be seen by adding the following debug code into vi2_fops.c, function tegra_channel_release_frame, line 726 in the original file, line 683 in the new version:
{
struct timespec b_ts;
u64 ts_ns;
u64 delta_ns;
getrawmonotonic(&b_ts);
ts_ns = timespec_to_ns(&b_ts);
delta_ns = ts_ns - buf->buf.vb2_buf.timestamp;
dev_info(&chan->video->dev,
"%s: buf[%p] frame[%d] buf.ts = %lld ts = %lld, delta = %lld\n",
__func__, buf, chan->sequence, buf->buf.vb2_buf.timestamp, ts_ns, delta_ns);
}
which gives a message like this for the old code:
[ 185.248466] video4linux video0: tegra_channel_release_frame: buf[00000000f1d130c6] frame[3] buf.ts = 185094321179 ts = 185095553054 delta = 1231875
and a message like this for the new code:
[94428.938866] video4linux video0: tegra_channel_release_frame: buf[0000000066a03f23] frame[3] buf.ts = 94428786049707 ts = 94428796107259, delta = 10057552
Times are in nanoseconds, so the first message has a delta of 1.2ms and the second has a delta of 10.0ms.
As a work around, the code has been changed to reapply the timestamp once the End Of Frame has been received in the tegra_channel_release_frame function with the following patch:
--- a/nvidia/drivers/media/platform/tegra/camera/vi/vi2_fops.c
+++ b/nvidia/drivers/media/platform/tegra/camera/vi/vi2_fops.c
@@ -646,6 +646,8 @@ static void tegra_channel_release_frame(struct tegra_channel *chan,
int index;
int err = 0;
int restart_version = 0;
+ u64 ts_ns;
+ u64 delta_ns;
/*
* If the frame capture was started on a different reset version
@@ -678,6 +680,25 @@ static void tegra_channel_release_frame(struct tegra_channel *chan,
__func__, buf);
}
+ getrawmonotonic(&ts);
+ ts_ns = timespec_to_ns(&ts);
+
+ /* delta_ns is the difference between the buffer time stamp and the time
+ * now, which is at EndOfFrame. If the buffer time stamp
+ * is applied at StartOfFrame, then this is a rough measure of the transfer
+ * time of the frame from the camera to the driver. However, now it seems
+ * that the buffer time stamp is applied at some point before StartOfFrame,
+ * so this value includes the transfer time, plus the time the buffer is
+ * waiting to be used, which makes it more variable and less useful. */
+ delta_ns = ts_ns - buf->buf.vb2_buf.timestamp;
+ dev_dbg(&chan->video->dev,
+ "%s: releasing buf[%p] frame[%d] buf.ts = %lld, delta = %lld\n",
+ __func__, buf, chan->sequence, buf->buf.vb2_buf.timestamp, delta_ns);
+ /* Set the buffer timestamp here (at EndOfFrame) as the original timestamp
+ * may be applied some time before the start of frame is received.
+ */
+ buf->buf.vb2_buf.timestamp = ts_ns;
+
atomic_dec(&chan->syncpt_depth);
buf->state = VB2_BUF_STATE_DONE;
This allows the frames to be matched, but does not give a realistic measure the transfer time, which is always very small.
Why was this change made, and is there a way of applying the timestamp when the start of frame is received?
Thanks