V4L2_BUF_FLAG_TSTAMP_SRC_SOE + CLOCK_REALTIME timestamping for V4L2 frames?

By default, I noticed with an IMX378 that:

v4l2-ctl -d /dev/video0 --set-fmt-video=width=4056,height=3040,pixelformat=RG10 \
 --set-ctrl frame_length=3200 \
 --set-ctrl coarse_time=3000 \
 --set-ctrl gain=100 \
 --set-ctrl bypass_mode=0 \
 --stream-mmap --stream-skip=1 --stream-count=1 --stream-to=test.raw --verbose

Gives us Monotonic + End of Frame timestamps:

...
VIDIOC_STREAMON: ok
        Index    : 0
        Type     : Video Capture
        Flags    : mapped, done
        Field    : None
        Sequence : 0
        Length   : 24903680
        Bytesused: 24903680
        Timestamp: 4367.209302s (Monotonic, End-of-Frame)

        Index    : 1
        Type     : Video Capture
        Flags    : mapped, done
        Field    : None
        Sequence : 1
        Length   : 24903680
        Bytesused: 24903680
        Timestamp: 4367.242628s (Monotonic, End-of-Frame)

VIDIOC_STREAMOFF: ok

I’m interested in knowing how we can switch this to both realtime clock and start of exposure (V4L2_BUF_FLAG_TSTAMP_SRC_SOE)?

The only obvious reference to V4L2_BUF_FLAG_TSTAMP_SRC_EOF I see is the following in:

/usr/src/kernel/kernel-4.4/drivers/media/platform/tegra/camera/vi/channel.c

...
chan->queue.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
chan->queue.io_modes = VB2_MMAP | VB2_DMABUF | VB2_READ | VB2_USERPTR;
chan->queue.lock = &chan->video_lock;
chan->queue.drv_priv = chan;
chan->queue.buf_struct_size = sizeof(struct tegra_channel_buffer);
chan->queue.ops = &tegra_channel_queue_qops;
#if defined(CONFIG_VIDEOBUF2_DMA_CONTIG)
chan->queue.mem_ops = &vb2_dma_contig_memops;
#endif
chan->queue.timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC
                           | V4L2_BUF_FLAG_TSTAMP_SRC_EOF;
...

if we change this to:

chan->queue.timestamp_flags = V4L2_BUF_FLAG_TSTAMP_SRC_SOE;

we see:

nvidia@jetson-randy2:~$ date
Thu Jan 17 15:52:24 UTC 2019
nvidia@jetson-randy2:~$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=4056,height=3040,pixelformat=RG10 \
>  --set-ctrl frame_length=3200 \
>  --set-ctrl coarse_time=3000 \
>  --set-ctrl gain=100 \
>  --set-ctrl bypass_mode=0 \
>  --stream-mmap --stream-skip=1 --stream-count=1 --stream-to=test.raw --verbos
VIDIOC_QUERYCAP: ok
VIDIOC_S_EXT_CTRLS: ok
VIDIOC_G_FMT: ok
VIDIOC_S_FMT: ok
Format Video Capture:
        Width/Height      : 4056/3040
        Pixel Format      : 'RG10'
        Field             : None
        Bytes per Line    : 8192
        Size Image        : 24903680
        Colorspace        : sRGB
        Transfer Function : Default
        YCbCr Encoding    : Default
        Quantization      : Default
        Flags             :
VIDIOC_REQBUFS: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_STREAMON: ok
        Index    : 0
        Type     : Video Capture
        Flags    : mapped, done
        Field    : None
        Sequence : 0
        Length   : 24903680
        Bytesused: 24903680
        Timestamp: 970.640632s (Unknown, Start-of-Exposure)

        Index    : 1
        Type     : Video Capture
        Flags    : mapped, done
        Field    : None
        Sequence : 1
        Length   : 24903680
        Bytesused: 24903680
        Timestamp: 970.673958s (Unknown, Start-of-Exposure)

VIDIOC_STREAMOFF: ok

From: https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/buffer.html#buffer-flags

V4L2_BUF_FLAG_TIMESTAMP_UNKNOWN	0x00000000	

Unknown timestamp type. This type is used by drivers before Linux 3.9 and may be either monotonic (see below) or realtime (wall clock). Monotonic clock has been favoured in embedded systems whereas most of the drivers use the realtime clock. Either kinds of timestamps are available in user space via clock_gettime() using clock IDs CLOCK_MONOTONIC and CLOCK_REALTIME, respectively.

Which seems correct. However, if we move the date back 2 minutes, we’d expect the timestamp to be less than 970.673958s if the timestamping is using the realtime clock and not a monotonic clock:

nvidia@jetson-randy2:~$ sudo date -s "17 Jan 2019 15:50:00"
Thu Jan 17 15:50:00 UTC 2019
nvidia@jetson-randy2:~$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=4056,height=3040,pixelformat=RG10  --set-ctrl frame_length=3200  --set-ctrl coarse_time=3000  --set-ctrl gain=100  --set-ctrl bypass_mode=0  --stream-mmap --stream-skip=1 --stream-count=1 --stream-to=test.raw --verbos
VIDIOC_QUERYCAP: ok
VIDIOC_S_EXT_CTRLS: ok
VIDIOC_G_FMT: ok
VIDIOC_S_FMT: ok
Format Video Capture:
        Width/Height      : 4056/3040
        Pixel Format      : 'RG10'
        Field             : None
        Bytes per Line    : 8192
        Size Image        : 24903680
        Colorspace        : sRGB
        Transfer Function : Default
        YCbCr Encoding    : Default
        Quantization      : Default
        Flags             :
VIDIOC_REQBUFS: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_STREAMON: ok
        Index    : 0
        Type     : Video Capture
        Flags    : mapped, done
        Field    : None
        Sequence : 0
        Length   : 24903680
        Bytesused: 24903680
        Timestamp: 1014.709862s (Unknown, Start-of-Exposure)

        Index    : 1
        Type     : Video Capture
        Flags    : mapped, done
        Field    : None
        Sequence : 1
        Length   : 24903680
        Bytesused: 24903680
        Timestamp: 1014.743188s (Unknown, Start-of-Exposure)

VIDIOC_STREAMOFF: ok

nvidia@jetson-randy2:~$ date
Thu Jan 17 15:50:04 UTC 2019

which doesn’t seem to be the case. Perhaps I am overlooking something here. How do we get the V4L2 to timestamp the start of frames using the realtime clock?

Did you apply the patch for the timestamp, you should be able know the detail from the vi driver and the change for timestamp.

https://elinux.org/Jetson_TX2/28.2.1_patches

I had not, thanks for mentioning it, I’ll give that a try.

I was having an issue with timestamps being zero, that patch fixes this, however, removing the V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC flag to get the timestamps in real-time does not seem to have effect, I still see the timestamps in monotonic time.

I see that the timestamps are filled in the calls to vi_notify_get_capture_status in vi4_fops.c, Do you know if the vi driver supports real-time timestamps?

Thanks

I think you can modify the driver to support it.

Looking into this, I haven’t found the piece of software that gets the system time and sets it to the status struct that contains the sof_ts and eof_ts which are the ones reported by v4l2, what I found was that the memory that holds the status is dma_coherent, is it possible that these timestamps are being set by hardware?

I read in the Parker TRM that “VI4 contains a time-stamping unit tracking system time” but didn’t found any more information about this, the only reference to timestamps that I found was the NOTIFY register that holds the timestamp of the events, and looking at capture traces, these timestamps are very different to the reported sof_ts, the trace timestamp reporting the CHANSEL_PXL_SOF is actually in the past by almost 5s compared with the sof_ts, I think that this was reported previously here: https://devtalk.nvidia.com/default/topic/1038131/jetson-tx2/vi-capture-system-time-offset/1 is there any update regarding this issue?

Additionally, do you know if the sof timestamp is captured when the first pixel packet arrives to VI4? or is it captured previous to that, more close to the start of exposure?

Thanks

Any suggestions ShaneCCC?

Hi ShaneCCC,

Does the VI use a binary firmware? Is this firmware the one in charge of controlling the timestamping? If so, is there any access to it? If the timestamp registers are memory mapped (assuming they are given the dma_coherent declaration), is there any documentation about the registers mapping and offset?

Best Regards,

It’s not memory mapped, it’s message communication. Please have a check the vi_fops.c and vi_notify.h

struct vi_capture_status {
u8 st; /* source stream from CSI block /
u8 vc; /
CSI virtual channel /
u16 frame; /
frame id /
u32 status; /
capture status /
u64 sof_ts; /
timestamp of START_OF_FRAME /
u64 eof_ts; /
timestamp of END_OF_FRAME /
u32 data; /
error data from vi notify module in VI /
u32 capture_id; /
identifier of associated capture request */
};

Hi ShaneCCC,

Ok, do you know which is the sender of the message? RTCPU, NOTIFY?

Also, can you check my previous question on the frame timestamp being out of sync with system time, any update on this?

Thank you

Yes, from RTCPU notify.
Could you have more explain for the timestamp being out of sync?

Looking at capture traces, the timestamps are very different to the reported sof_ts. The trace timestamp (kernel time I suppose) reporting the CHANSEL_PXL_SOF event is actually in the past by almost 5s compared with the actual sof_ts value, I think that this was reported previously here: https://devtalk.nvidia.com/default/topic/1038131/jetson-tx2/vi-capture-system-time-offset/1 is there any update regarding this issue?

Additionally, do you know if the sof timestamp is captured when the first pixel packet arrives to VI4? or is it captured previous to that, more close to the start of exposure?

Thank you

We still working on it, Please stick with that topic for the solution.

Understood, thanks!

We wish to modify the clock being used for the v4l2 frame timestamp (to use real time clock instead of monotonic) in its very source if possible, instead of trying to ovewrite it somewhere later.

For this reason, I’ve been looking into the internals of the frame timestamping mechanism for a while, and hit a wall in the search for the piece of software that actually gets the current time and sets the value.

Here are my findings so far:

The timestamp assigned to the v4l2 frames is extracted in tegra_ivc_vi_notify_get_capture_status function[1].

The memory from where its extracted, is initialized in the function tegra_ivc_channel_vi_notify_probe[1] as dma_coherent memory as follows:

ivn->status_mem = dma_alloc_coherent(dev,ivn->status_mem_size,&ivn->status_dmaptr, GFP_KERNEL | __GFP_ZERO);

Now, somehow the pointer of this memory has to get to the RTCPU so that it can write the timestamp.
Tracing the ways this could be happening, I found that the status_dmaptr (which in the documentation is referred to as the value given to the device as the DMA address base of the region [2]) is asigned to one of the fields of the struct vi_notify_req in the function tegra_ivc_vi_notify_enable_reports[1] and then the function tegra_ivc_vi_notify_send [1] is called with this struct as argument.

Now this tegra_ivc_vi_notify_send [1] function calls tegra_ivc_write [3] with that vi_notify_req struct as argument casted to const void *buf. The function tegra_ivc_write[3] calls ivc_write_frame[3] passing the vi_notify_req struct as well.

Finally in ivc_write_frame[3] some interesting things happen, there is a call to ivc_check_write[3] which calls ivc_invalidate_counter[3], which calls dma_sync_single_for_cpu. This dma_sync_single_for_cpu according to the documentation [4] gives the ownership of the DMA buffer to the processor. Now returning to the ivc_write_frame function[3] it obtains an ivc_frame_pointer[3], and performs a memcpy of the vi_notify_req struct to that ivc_frame_pointer[3] location, and then calls a ivc_flush_frame[3], that calls dma_sync_single_for_device which according to the documentation [4] allows the device to access the DMA buffer again.

With these findings, I come to believe that the timestamp is written by a device to that dma memory, accessible to the cpu for reading whenever it knows that a PXL_SOF event has occurred in the vi_notify_wait[5].

My question is, Is the timestamp being written by some firmware or hardware that we don’t have access to modify?

Thank you

[1] kernel/t18x/drivers/platform/tegra/rtcpu/vi-notify.c
[2] https://www.kernel.org/doc/Documentation/DMA-API.txt
[3] kernel/kernel-4.4/drivers/platform/tegra/tegra-ivc.c
[4] https://lwn.net/Articles/75780/
[5] kernel/kernel-4.4/drivers/media/platform/tegra/camera/vi/vi4_fops.c

Yes, it’s write from RTCPU firmware.

Hi ShaneCCC,

Is it possible to modify or get access to the RTCPU firmware? Or to get a new firmware that solves this problem?

Best Regards,

I am sorry we don’t have plan to release RTCPU firmware. We are working on the solution for the next release.

Hi Shane,

Great, do you have an approximate timeline for the release?

Hi All
Does any one verify the timestamp problem r32 release?