GTE Timestamp Reporting Latency and Incorrect/Stale Timestamps reported under CPU load

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
1

On Orin AGX running Jetpack 5.1.4, I’m seeing a couple of (potentially unrelated) issues. The attached patch simply adds timestamp (in the TSC’s clock domain) prints to the GTE demo app; this patch can be used to reproduce both issues listed below.

Test Setup:

  • GTE app patched using the attached patch.
  • A 30 Hz input signal on pin 320
  • stress tool installed to help reproduce issue #2.
  1. Latency between a timestamp recorded by the GTE, and the time at which this is presented to the application, varies between runs of the application.

Run 1: Application reads timestamp ~66us after the event occurred

$ sudo ./tegra_gte_mon -d gtechip0 -g 320 -r

Monitoring line 320 on gtechip0
HW timestamp GPIO EVENT 14782432945792
now_tsc_ns() - HW timestamp GPIO EVENT 66080
HW timestamp GPIO EVENT 14782466277920
now_tsc_ns() - HW timestamp GPIO EVENT 65536
HW timestamp GPIO EVENT 14782499610496
now_tsc_ns() - HW timestamp GPIO EVENT 60512
HW timestamp GPIO EVENT 14782532945536
now_tsc_ns() - HW timestamp GPIO EVENT 64672

Run 2: Application reads timestamp ~33.33 ms after the event occurred

$ sudo ./tegra_gte_mon -d gtechip0 -g 320 -r
Monitoring line 320 on gtechip0
HW timestamp GPIO EVENT 14776811928384
now_tsc_ns() - HW timestamp GPIO EVENT 21081472
HW timestamp GPIO EVENT 14776832943552
now_tsc_ns() - HW timestamp GPIO EVENT 33401472
HW timestamp GPIO EVENT 14776866275104
now_tsc_ns() - HW timestamp GPIO EVENT 33400384
HW timestamp GPIO EVENT 14776899610048
now_tsc_ns() - HW timestamp GPIO EVENT 33423776
HW timestamp GPIO EVENT 14776932942720
now_tsc_ns() - HW timestamp GPIO EVENT 33399424
HW timestamp GPIO EVENT 14776966275776
now_tsc_ns() - HW timestamp GPIO EVENT 33402432
HW timestamp GPIO EVENT 14776999610048
  1. When the Orin experiences brief periods of high CPU load, the GTE gets into a state where it reports extremely old timestamps. e.g. an application recording a 30 Hz input, will start reporting timestamps >500 ms in the past, instead of the expected ~33.33 ms-old timestamps. This happens rarely, but once the GTE enters this state, the application needs to be restarted to get the GTE to report the latest timestamps

Normal Case:

  now_tsc_ns() - HW timestamp GPIO EVENT 33414208
  HW timestamp GPIO EVENT 15461065139680
  now_tsc_ns() - HW timestamp GPIO EVENT 33402240
  HW timestamp GPIO EVENT 15461098477088
  now_tsc_ns() - HW timestamp GPIO EVENT 33395296
  HW timestamp GPIO EVENT 15461131808032
  now_tsc_ns() - HW timestamp GPIO EVENT 33392480

Error Case: Simulate a brief period of high CPU load by running the below command for a second. Adjust number of cores depending on 8 or 12-core AGX. For 8-core AGX use 7 or 8.

 $ sudo chrt -f 50 stress -c 7
stress: info: [649641] dispatching hogs: 7 cpu, 0 io, 0 vm, 0 hdd
^C

Latency between time of timestamp reporting and timestamp of the actual event jumps to 1033 ms (and stays there)

now_tsc_ns() - HW timestamp GPIO EVENT 1033399392
HW timestamp GPIO EVENT 15464798461088
now_tsc_ns() - HW timestamp GPIO EVENT 1033400192
HW timestamp GPIO EVENT 15464831795808
now_tsc_ns() - HW timestamp GPIO EVENT 1033398880
HW timestamp GPIO EVENT 15464865129024
now_tsc_ns() - HW timestamp GPIO EVENT 1033482720
HW timestamp GPIO EVENT 15464898466784
now_tsc_ns() - HW timestamp GPIO EVENT 1033404160

Restart the application. Observe the latency drops to the expected value

$ sudo ./tegra_gte_mon -d gtechip0 -g 320 -r
Monitoring line 320 on gtechip0
HW timestamp GPIO EVENT 15733870769088
now_tsc_ns() - HW timestamp GPIO EVENT 26890560
HW timestamp GPIO EVENT 15733897593440
now_tsc_ns() - HW timestamp GPIO EVENT 33400800
HW timestamp GPIO EVENT 15733930928224
now_tsc_ns() - HW timestamp GPIO EVENT 33390912

Patch File:

tegra_gte_mon.patch.txt (944 Bytes)

4

Hi,
Please check the document and see if you enable GTE correctly:
Generic Timestamp Engine — NVIDIA Jetson Linux Developer Guide 1 documentation
SPE: GTE provided TSC have unexpected values - #18 by jachen

And FYI:
Generic Timestamp Engine — NVIDIA Jetson Linux Developer Guide

GTE is replaced with HTE on Jetpack 6.

5

Hi @DaneLLL ,

I have checked the document to ensure GTE is setup correctly. The timestamping works most of the time; but like I mentioned under rare periods of heavy CPU load, it gets into an error state.

The method used to reproduce the issue, shared earlier in the thread, simply runs the example demo app provided by NVIDIA. It would be great if you can try to reproduce the issue.

6

Hi,
Are yo able to try AGX Orin developer with Jetpack 6.2.2 r36.5 and see if the issue still appears? Certain issues are found and fixed in later release. Would suggest try the latest Jetpack 6 version.