getSensorTimestamp, getSensorTimestampTsc and v4l2 timestamp

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin NX
1

Hello nvidia gurus,

I work with jetpack-6.1 on a orin NX board.

I noticed that the timestamps given by nvarguscamerasrc are not in the same time domain as the timestamps given by the v4l2 driver.

Moreover v4l2src discards the timestamps given by the v4l2 driver, because they are not in the MONOTONIC time domain nor in the REALTIME time domain. The following simple test

~$ GST_DEBUG=3 gst-launch-1.0 v4l2src ! fakesink

shows it explicitly

Timestamp does not correlate with any clock, ignoring driver timestamps

Secondly, like What is the relationship between RCE timestamp and TSC timestamp? - #4 by JerryChang, I have patched the yuvJpeg argus sample to use getSensorTimestampTsc in addition of getSensorTimestamp, and computed the differences between the two timestamps for the same image. I have noticed that the difference is not constant. Where does that come from ? Shouldn’t the values differ only by a constant ‘offset_ns’ ? Which one is recorded closer to the arrival of the first pixel on the mipi csi connection ?

Like other people on the forum, we need to have precise and consistent walltime (aka realtime) timestamps for our captured images regardless of the way they are captured (pure v4l2, ISP+argus, gstreamer) in the most straightforward way, not by tweaking each userland application.

https://forums.developer.nvidia.com/t/how-can-i-get-exact-systemtime-timestamp-with-gstreamer/146324/2

https://forums.developer.nvidia.com/t/nvarguscamerasrc-timestamping/199563/2

https://forums.developer.nvidia.com/t/argus-timestamps-in-kernel-space/314121/2

https://forums.developer.nvidia.com/t/camera-rce-timestamp/313528/21

https://forums.developer.nvidia.com/t/what-is-the-relationship-between-rce-timestamp-and-tsc-timestamp/313792/9

2

*** Please note that this reply is generated by LLM automatically ***

The question revolves around the issue of timestamp inconsistencies between nvarguscamerasrc and v4l2 driver on a Jetson Orin NX board, as well as the relationship between RCE (Real-Time Clock) timestamp and TSC (Time-Stamp Counter) timestamp. To address this, let’s break down the components and concepts involved:

  1. Timestamp Domains: There are two primary timestamp domains mentioned - MONOTONIC and REALTIME. The MONOTONIC clock starts at an unspecified point in the past and increments monotonically, representing the time elapsed since some unspecified starting point. The REALTIME clock represents the number of seconds since the Unix epoch (January 1, 1970).

  2. nvarguscamerasrc and v4l2 Driver: nvarguscamerasrc is a part of the NVIDIA Argus API, which provides a high-level interface for camera control and image capture. The v4l2 driver, on the other hand, is a part of the Video4Linux2 (V4L2) framework, which provides a standard interface for video capture devices. The issue here is that v4l2src discards timestamps from the v4l2 driver because they are not in the MONOTONIC or REALTIME time domain.

  3. RCE Timestamp and TSC Timestamp:

    • RCE (Real-Time Clock) Timestamp: This refers to timestamps generated by the system’s real-time clock, which is synchronized with the system’s wall clock time.
    • TSC (Time-Stamp Counter) Timestamp: This is a counter that increments at each clock cycle, providing a high-resolution timer. TSC timestamps are not directly related to wall clock time but can be used to measure time intervals.

  4. Relationship Between RCE and TSC Timestamps: The difference between RCE and TSC timestamps should ideally be a constant offset, assuming that the TSC frequency is stable and known. However, variations in the difference, as observed by JerryChang, could be due to several factors, including:

    • Frequency Variations: If the TSC frequency is not perfectly stable, this could introduce variations in the timestamp differences.
    • Synchronization Issues: If there are issues with synchronizing the RCE and TSC clocks, this could also lead to non-constant differences.
    • Software or Hardware Delays: Delays in the software or hardware pipeline could affect when timestamps are recorded, leading to inconsistencies.

  5. Recording Timestamps Closer to the Arrival of the First Pixel: To achieve precise and consistent wall-time (real-time) timestamps for captured images, it’s essential to record timestamps as close as possible to the arrival of the first pixel on the MIPI CSI connection. This might involve using hardware timestamps, such as those provided by the camera sensor or the MIPI CSI interface, and then converting these to real-time timestamps.

  6. Solutions and Workarounds: Given the complexity of the issue, potential solutions might include:

    • Using Hardware Timestamps: If possible, use timestamps generated by the camera sensor or the MIPI CSI interface, as these are likely to be closer to the actual time of image capture.
    • Synchronizing Clocks: Ensure that all clocks involved (RCE, TSC, and any others) are properly synchronized to minimize discrepancies.
    • Software Adjustments: Implement software adjustments to compensate for known delays or discrepancies in the timestamping process.
    • Kernel Space Solutions: Some issues might be addressed by modifying kernel space code, such as the v4l2 driver or the Argus API, to handle timestamps more accurately.

In summary, achieving precise and consistent real-time timestamps for images captured on a Jetson Orin NX board involves understanding the different timestamp domains, the relationship between RCE and TSC timestamps, and addressing potential issues such as clock synchronization, hardware or software delays, and the use of appropriate timestamping mechanisms. This might require a combination of hardware and software adjustments, as well as careful consideration of the specific requirements of the application.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

3

What is ‘LLM’ ?

Are the RCE and TSC clocks really distinct or are there different counters started at different times but running at the same frequency from a single hardware clock signal ?

Does nvidia plan to at least fix the vi5 driver to provide timestamps accepted by v4l2src, and documentation for people like me and others mentioned in the links to be able to provide realtime timestamps by tweaking (patch or kernel parameter) the kernel in the isp-to-argus communication ?

5

For the timestamp of the v4l2 need check by the v4l2-ctl for example.

v4l2-ctl -c bypass_mode=0 --stream-mmap --verbose
VIDIOC_QUERYCAP: ok
VIDIOC_S_EXT_CTRLS: ok
		VIDIOC_REQBUFS returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_STREAMON returned 0 (Success)
cap dqbuf: 0 seq:      0 bytesused: 16163840 ts: 587551.736236 (ts-monotonic, ts-src-eof)
cap dqbuf: 1 seq:      1 bytesused: 16163840 ts: 587551.783437 delta: 47.201 ms (ts-monotonic, ts-src-eof)
cap dqbuf: 2 seq:      2 bytesused: 16163840 ts: 587551.830638 delta: 47.201 ms (ts-monotonic, ts-src-eof)
cap dqbuf: 3 seq:      3 bytesused: 16163840 ts: 587551.877839 delta: 47.201 ms (ts-monotonic, ts-src-eof)
cap dqbuf: 0 seq:      4 bytesused: 16163840 ts: 587551.925040 delta: 47.201 ms fps: 21.19 (ts-monotonic, ts-src-eof)
cap dqbuf: 1 seq:      5 bytesused: 16163840 ts: 587551.972241 delta: 47.201 ms fps: 21.19 (ts-monotonic, ts-src-eof)
cap dqbuf: 2 seq:      6 bytesused: 16163840 ts: 587552.019442 delta: 47.201 ms fps: 21.19 (ts-monotonic, ts-src-eof)
6

Hello ShaneCCC,

thank you for the example.

Here is a demo of the inconsistency between timestamps using v4l2 and timestamps using argus.

v4l2-ctl -c bypass_mode=0 --stream-mmap --stream-count=10 --verbose; ./argus_yuvjpeg -t 1 | grep Timestamp:
VIDIOC_QUERYCAP: ok
VIDIOC_S_EXT_CTRLS: ok
                VIDIOC_REQBUFS returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_STREAMON returned 0 (Success)
cap dqbuf: 0 seq:      0 bytesused: 10303488 ts: 736.999092 (ts-monotonic, ts-src-eof)
cap dqbuf: 1 seq:      1 bytesused: 10303488 ts: 737.032423 delta: 33.331 ms (ts-monotonic, ts-src-eof)
cap dqbuf: 2 seq:      2 bytesused: 10303488 ts: 737.065754 delta: 33.331 ms (ts-monotonic, ts-src-eof)
cap dqbuf: 3 seq:      3 bytesused: 10303488 ts: 737.099084 delta: 33.330 ms (ts-monotonic, ts-src-eof)
cap dqbuf: 0 seq:      4 bytesused: 10303488 ts: 737.132415 delta: 33.331 ms fps: 30.00 (ts-monotonic, ts-src-eof)
cap dqbuf: 1 seq:      5 bytesused: 10303488 ts: 737.165746 delta: 33.331 ms fps: 30.00 (ts-monotonic, ts-src-eof)
cap dqbuf: 2 seq:      6 bytesused: 10303488 ts: 737.199076 delta: 33.330 ms fps: 30.00 (ts-monotonic, ts-src-eof)
cap dqbuf: 3 seq:      7 bytesused: 10303488 ts: 737.232407 delta: 33.331 ms fps: 30.00 (ts-monotonic, ts-src-eof)
cap dqbuf: 0 seq:      8 bytesused: 10303488 ts: 737.265738 delta: 33.331 ms fps: 30.00 (ts-monotonic, ts-src-eof)
cap dqbuf: 1 seq:      9 bytesused: 10303488 ts: 737.299069 delta: 33.331 ms fps: 30.00 (ts-monotonic, ts-src-eof)

CONSUMER:       Sensor Timestamp: 713517057000, LUX: 3038.924072
CONSUMER:       Sensor Timestamp: 713550402000, LUX: 3039.257324
CONSUMER:       Sensor Timestamp: 713583727000, LUX: 3039.812256
CONSUMER:       Sensor Timestamp: 713617057000, LUX: 3040.017578
CONSUMER:       Sensor Timestamp: 713650389000, LUX: 3040.390137
CONSUMER:       Sensor Timestamp: 713683724000, LUX: 3040.666748
CONSUMER:       Sensor Timestamp: 713717054000, LUX: 5867.164062
CONSUMER:       Sensor Timestamp: 713750388000, LUX: 6126.350586
CONSUMER:       Sensor Timestamp: 713783719000, LUX: 6328.284180
CONSUMER:       Sensor Timestamp: 713817046000, LUX: 6476.250977
CONSUMER:       Sensor Timestamp: 713850378000, LUX: 6554.108398
CONSUMER:       Sensor Timestamp: 713883707000, LUX: 6613.342285
CONSUMER:       Sensor Timestamp: 713917040000, LUX: 6668.977539
CONSUMER:       Sensor Timestamp: 713950373000, LUX: 6719.968262
CONSUMER:       Sensor Timestamp: 713983701000, LUX: 6768.780273
CONSUMER:       Sensor Timestamp: 714017034000, LUX: 6819.678223
CONSUMER:       Sensor Timestamp: 714050375000, LUX: 6874.002930
CONSUMER:       Sensor Timestamp: 714083697000, LUX: 6931.125488
CONSUMER:       Sensor Timestamp: 714117038000, LUX: 6992.253418
CONSUMER:       Sensor Timestamp: 714150360000, LUX: 7059.019531

As one can see, the timestamps come from different time domains. The timestamps from v4l2 are about 24 seconds above the ones given by argus. And as I wrote above, and already a year ago https://forums.developer.nvidia.com/t/how-to-obtain-accurate-utc0-frame-timestamps/286100/8 the timestamps provided by vi5_fops are not compliant with v4l2 ! And they are discarded by the v4l2src plugin of gstreamer.

Please make vi5_fops and argus provide natively realtime timestamps, or give us enough information on how to fix that ourselves in the kernel.

7

Does the timestamp from vi5_fops.c the same with v4l2-ctl?

8

Yes. They come directy from the VIDIOC_DQBUF ioctl. See for yourself.

$ strace -o /tmp/xxx -e ioctl v4l2-ctl -c bypass_mode=0 --stream-mmap --stream-count=10 --verbose
VIDIOC_QUERYCAP: ok
VIDIOC_S_EXT_CTRLS: ok
                VIDIOC_REQBUFS returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QUERYBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_QBUF returned 0 (Success)
                VIDIOC_STREAMON returned 0 (Success)
cap dqbuf: 0 seq:      0 bytesused: 10303488 ts: 2967.562259 (ts-monotonic, ts-src-eof)
cap dqbuf: 1 seq:      1 bytesused: 10303488 ts: 2967.595589 delta: 33.330 ms (ts-monotonic, ts-src-eof)
cap dqbuf: 2 seq:      2 bytesused: 10303488 ts: 2967.628920 delta: 33.331 ms (ts-monotonic, ts-src-eof)
cap dqbuf: 3 seq:      3 bytesused: 10303488 ts: 2967.662251 delta: 33.331 ms (ts-monotonic, ts-src-eof)
cap dqbuf: 0 seq:      4 bytesused: 10303488 ts: 2967.695581 delta: 33.330 ms fps: 30.00 (ts-monotonic, ts-src-eof)
cap dqbuf: 1 seq:      5 bytesused: 10303488 ts: 2967.728912 delta: 33.331 ms fps: 30.00 (ts-monotonic, ts-src-eof)
cap dqbuf: 2 seq:      6 bytesused: 10303488 ts: 2967.762243 delta: 33.331 ms fps: 30.00 (ts-monotonic, ts-src-eof)
cap dqbuf: 3 seq:      7 bytesused: 10303488 ts: 2967.795573 delta: 33.330 ms fps: 30.00 (ts-monotonic, ts-src-eof)
cap dqbuf: 0 seq:      8 bytesused: 10303488 ts: 2967.828904 delta: 33.331 ms fps: 30.00 (ts-monotonic, ts-src-eof)
cap dqbuf: 1 seq:      9 bytesused: 10303488 ts: 2967.862234 delta: 33.330 ms fps: 30.00 (ts-monotonic, ts-src-eof)

$ grep DQBUF /tmp/xxx
ioctl(3, VIDIOC_DQBUF, {type=V4L2_BUF_TYPE_VIDEO_CAPTURE, index=0, memory=V4L2_MEMORY_MMAP, m.offset=0, length=10303488, bytesused=10303488, flags=V4L2_BUF_FLAG_MAPPED|V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC|V4L2_BUF_FLAG_TSTAMP_SRC_EOF, timestamp={tv_sec=2967, tv_usec=562259}, ...}) = 0
ioctl(3, VIDIOC_DQBUF, {type=V4L2_BUF_TYPE_VIDEO_CAPTURE, index=1, memory=V4L2_MEMORY_MMAP, m.offset=0x9d4000, length=10303488, bytesused=10303488, flags=V4L2_BUF_FLAG_MAPPED|V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC|V4L2_BUF_FLAG_TSTAMP_SRC_EOF, timestamp={tv_sec=2967, tv_usec=595589}, ...}) = 0
ioctl(3, VIDIOC_DQBUF, {type=V4L2_BUF_TYPE_VIDEO_CAPTURE, index=2, memory=V4L2_MEMORY_MMAP, m.offset=0x13a8000, length=10303488, bytesused=10303488, flags=V4L2_BUF_FLAG_MAPPED|V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC|V4L2_BUF_FLAG_TSTAMP_SRC_EOF, timestamp={tv_sec=2967, tv_usec=628920}, ...}) = 0
ioctl(3, VIDIOC_DQBUF, {type=V4L2_BUF_TYPE_VIDEO_CAPTURE, index=3, memory=V4L2_MEMORY_MMAP, m.offset=0x1d7c000, length=10303488, bytesused=10303488, flags=V4L2_BUF_FLAG_MAPPED|V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC|V4L2_BUF_FLAG_TSTAMP_SRC_EOF, timestamp={tv_sec=2967, tv_usec=662251}, ...}) = 0
ioctl(3, VIDIOC_DQBUF, {type=V4L2_BUF_TYPE_VIDEO_CAPTURE, index=0, memory=V4L2_MEMORY_MMAP, m.offset=0, length=10303488, bytesused=10303488, flags=V4L2_BUF_FLAG_MAPPED|V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC|V4L2_BUF_FLAG_TSTAMP_SRC_EOF, timestamp={tv_sec=2967, tv_usec=695581}, ...}) = 0
ioctl(3, VIDIOC_DQBUF, {type=V4L2_BUF_TYPE_VIDEO_CAPTURE, index=1, memory=V4L2_MEMORY_MMAP, m.offset=0x9d4000, length=10303488, bytesused=10303488, flags=V4L2_BUF_FLAG_MAPPED|V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC|V4L2_BUF_FLAG_TSTAMP_SRC_EOF, timestamp={tv_sec=2967, tv_usec=728912}, ...}) = 0
ioctl(3, VIDIOC_DQBUF, {type=V4L2_BUF_TYPE_VIDEO_CAPTURE, index=2, memory=V4L2_MEMORY_MMAP, m.offset=0x13a8000, length=10303488, bytesused=10303488, flags=V4L2_BUF_FLAG_MAPPED|V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC|V4L2_BUF_FLAG_TSTAMP_SRC_EOF, timestamp={tv_sec=2967, tv_usec=762243}, ...}) = 0
ioctl(3, VIDIOC_DQBUF, {type=V4L2_BUF_TYPE_VIDEO_CAPTURE, index=3, memory=V4L2_MEMORY_MMAP, m.offset=0x1d7c000, length=10303488, bytesused=10303488, flags=V4L2_BUF_FLAG_MAPPED|V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC|V4L2_BUF_FLAG_TSTAMP_SRC_EOF, timestamp={tv_sec=2967, tv_usec=795573}, ...}) = 0
ioctl(3, VIDIOC_DQBUF, {type=V4L2_BUF_TYPE_VIDEO_CAPTURE, index=0, memory=V4L2_MEMORY_MMAP, m.offset=0, length=10303488, bytesused=10303488, flags=V4L2_BUF_FLAG_MAPPED|V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC|V4L2_BUF_FLAG_TSTAMP_SRC_EOF, timestamp={tv_sec=2967, tv_usec=828904}, ...}) = 0
ioctl(3, VIDIOC_DQBUF, {type=V4L2_BUF_TYPE_VIDEO_CAPTURE, index=1, memory=V4L2_MEMORY_MMAP, m.offset=0x9d4000, length=10303488, bytesused=10303488, flags=V4L2_BUF_FLAG_MAPPED|V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC|V4L2_BUF_FLAG_TSTAMP_SRC_EOF, timestamp={tv_sec=2967, tv_usec=862234}, ...}) = 0
$
9

I mean print the timestamp like SOF/EOF in the vi5_fops.c

10

Which way would you do that to demonstrate the inconsistency ?

11

What I want to tell is the timestamp in the vi5_fops.c is the same with the getSensorTimestampTsc.

12

Hello ShaneCCC,

thank you.

We have two sensors on our boards, and we can make them work synchronously, putting one of them in ‘slave’ mode relying on XVS and XHS signals coming from the first one.

With v4l2, I had already confirmed that this works perfectly. Images coming from both sensors have the same timestamps, sometimes differing by 1 ns.

Now using yuvJpeg modified to use getSensorTimestampTsc for the first sensor, while using v4l2 for the second one, I can also confirm that both sensors provide the same timestamps.

What I would like to know is :

  • why does nvarguscamerasrc provide getSensorTimestamp-based timestamps instead of getSensorTimestampTsc-based timestamps, in order to ensure consistency with v4l2 timestamps ?
  • more importantly, how can I in kernel space modify the answer of getSensorTimestampTsc like i can modify the v4l2 timestamps in vi5_fops ? Remember, the timestamps provided currently by v4l2 are not compliant with v4l2 specifications and are discarded by the v4l2src plugin of gstreamer.