JP6.2系统时间跳变问题

您好,我们在AGX Orin上部署JP6.2版本固件,发现了一个很奇怪的现象:我们使用Linuxptp的phc2sys将系统时间同步给MGBE网卡(比如MGBE1,MGBE2),然后会出现时间偏差突然增大上百微秒的情况,如下图:


但是当我们把MGBE0网卡down掉后,此现象就没有了,使用phc2sys同步系统时间到网卡时间的offset会长期稳定在百ns以下
我这里确认系统本身会触发修改系统时间的一些服务,比如ntp, gpsd服务和systemd-timesync服务都已经关闭. 所以请问,MGBE0的设备树配置是否会影响到系统时间的更新?
linuxptp版本:3.1
MGBE0设备树配置:
mgbe0.txt (2.7 KB)
Jetpack:6.2

Please try to reproduce this issue on Orin AGX devkit first.

好的,我们用开发板看一下
另外补充个信息,就是我们相同的自己的硬件,使用JP512是没有问题的,但是更新到JP6.2后就出现了这个问题;看上去像是JP6.2的一些适配问题,所以也想请教下,网卡的设备树的配置是否会因为冲突影响到系统时间

Just to clarify. No one reported this error before. So this is unknown question to us too.

您好,我这边在开发板上,烧录Nvidia的官方版本JP6.2+Sample rootfs同样出现了此问题(表现与上面相同),通过如下指令将系统时间同步给网卡(eno1),同步过程中概率系统时间跳变导致,系统时间与网卡硬件时间offset跳变到100us+
sudo ./phc2sys -s CLOCK_REALTIME -c eno1 -O 0 --step_threhold 0.1

同时发现,JP6.2有多个不同的系统时钟源(arch_sys_counter/tsc/osc/usec),默认使用的arch_sys_counter存在此问题(JP512默认使用的也是arch_sys_counter,表现正常),尝试切换为tsc后依然存在此问题;最后使用usec(软件模拟时钟源),系统时间不再跳变

请问下,Nvidia这边是否可以在本地复现下此问题,目前看似乎是JP6.2的一个BUG?

please provide the exact step by step to so that we can try to reproduce your issue.

step 1
登录到Orin内,git clone linuxptp,linuxptp链接:GitHub - richardcochran/linuxptp: User space PTP stack for the GNU/Linux operating system.

step 2
安装linuxptp(我们这边测试了3.1版本和4.4版本,可以通过checkout切换到不同版本分支),我这边的安装方式如下:

  1. 进入到linuxptp文件夹,执行make
  2. make完成后,执行sudo make install --prefix=./
  3. linuxptp工具安装到 linuxptp/sbin路径下

step 3
使用linuxptp开源工具的phc2sys工具,将系统时间同步给网卡(官方JP6.2版本在开发板上的使能网卡名称为eno1)

  1. 进入到linuxptp/sbin路径下
  2. 执行指令如下:
    sudo ./phc2sys -s CLOCK_REALTIME -c eno1 -O 0 -m --step_threshold 0.1

运行过程中,会发现系统时间会概率跳变,导致phc2sys计算的Orin系统时间与网卡硬件时间戳的offset值会突然增到到100us+(下图中单位为ns,即100,000+ns),如下图

您好,请问您这边复现有什么进展吗?

We are able to reproduce this and still investigate what is going on.

It might be an issue of tuning some configurations?

sudo apt install linuxptp

provides config files in /etc/linuxptp
some other /usr/share/doc/linuxptp/configs
and services if wanted /usr/lib/systemd/system/
phc2sys@.service ptp4l@.service timemaster.service

I had already built linuxptp version 4.4 and since I’m just experimenting I copied hwstamp_ctl phc2sys phc_ctl pmc ptp4l timemaster ts2phc tz2alt, version 4.4 executables over the apt installed /usr/sbin/ version 3.1 files.

Make sure the PHC really exists

ls -l /dev/ptp0

The following should return “PTP Hardware Clock: 0” #which is good

sudo ethtool -T eno1 | grep -E ‘PTP Hardware Clock|Capabilities’


Stop anything else that steers clock_realtime.

sudo timedatectl set-ntp false
sudo systemctl stop systemd-timesyncd chronyd ntpd || true


Edit /etc/linuxptp/ptp4l.conf and change following:
tx_timestamp_timeout 20
slaveOnly to clientOnly
masterOnly to serverOnly


Then following return better times e.g.

phc2sys[26160.608]: CLOCK_REALTIME phc offset -2 s2 freq +20362 delay 2592

sudo phc2sys -s eno1 -c CLOCK_REALTIME -O 0 -m --step_threshold 0.1 -f /etc/linuxptp/phc2sys.conf  

sudo phc2sys -s CLOCK_REALTIME -c eno1 -O 0 -m --step_threshold 0.1 -f /etc/linuxptp/phc2sys.conf
sudo phc2sys -s CLOCK_REALTIME -c eno1 -O 0 -m -S 0.001 -f /etc/linuxptp/phc2sys.conf  #1ms threshold
 


If needed have eno1 locked to the grand-master or acting as gm

sudo ptp4l -f /etc/linuxptp/ptp4l.conf -i eno1 -m -2 -H #-H is hardware time-stamps only
sudo ptp4l -f /etc/linuxptp/ptp4l.conf -i eno1 -m
2 Likes

您好,请问下,我这边按照操作通过apt下载了linuxptp后,/etc/linuxptp/下没有phc2sys.conf这个配置文件,您这边是否方便提供下呢?我这里本地验证一下

man phc2sys

There isn’t one. I just added 2 things from the man page. I had previously unminimized my agx orin to get manuals.

/etc/linuxptp/phc2sys.conf

[global]
step_threshold 0.001
first_step_threshold 0.001

按照您这边提供的方法,我在JP6.1上尝试了下,还是会出现系统时间跳变的问题
这个感觉是系统时钟源的问题,和phc2sys或者ptp4l的配置没有关系

Here’s some docs that may address the system time jumps problem.

synchronizing time with linuxptp docs

/etc/linuxptp/timemaster.conf and man timemaster.8

These configs are in /usr/share/doc/linuxptp/configs

configuring_ptp_using_ptp4l

chrony

您好,请问目前有什么进展吗?
感觉应该和linuxptp的配置没有太大关系,是Orin系统本身的时间存在概率跳变。

Hi,

What is the usecase for using phc2sys here? Also, did you run ptp4l before running phc2sys to make sure PHC is synced with grandmaster?

您好,我这里是使用Orin作为ptp server,将系统时间同步给网卡,然后通过二层将网卡的硬件时间同步给外部网络设备
目前的问题就是Orin的系统时间会概率跳变100+us,然后导致外部的ptp client设备计算的时间跟Orin系统时间差也概率超过100us+,从而报错
另外一个比较严重的问题是系统时间有概率向过去跳,导致外部的设备时间比系统时间更未来

Hello, what are your command lines to the ptp executables that you are using?

您好,ptp client指令如下:
ptp4l -2 -i MGBE -m -l 6 -f /etc/linuxptp/ptp4l.conf
phc2sys -s CLOCK_REALTIME -c MGBE -O 0 -m --step_threshold 0.1

ptp4l.conf的配置文件如下:
ptp4l.txt (1.6 KB)

I have tried multiple variations and can not get ptp to keep offset from periodically spiking.

Today I tried this and got significantly higher offsets; but I’m going through a router and switch between the 2 computers.

Configuring PTP orin as source.

On source Orin
sudo ptp4l -i eno1 -m -4 -f /etc/linuxptp/ptp4l.conf

On sink
sudo ptp4l -i enp6s0f1 -m -s -4 -f /etc/linuxptp/ptp4l.conf

Synchronize System Times of Source and Sink

On source Orin
sudo phc2sys -s CLOCK_REALTIME -c enpo1 -m –w

On sink
sudo phc2sys -s enp6s0f1 -c CLOCK_REALTIME -m -w

Here are things I tried yesterday. Better results but still offset spikes


sudo phc2sys -s /dev/ptp0 -c eno1 -m -R 1000
sudo phc2sys -s CLOCK_REALTIME -c eno1 -O 0 -m -S 0.001 -f /etc/linuxptp/phc2sys.conf -R 1000

sudo phc2sys -s /dev/ptp0 -c CLOCK_REALTIME -O 0 -m -S 0.001 -f /etc/linuxptp/phc2sys.conf -R 1
sudo phc2sys -s /dev/ptp0 -c CLOCK_REALTIME --step_threshold 0 --first_step_threshold 0 -m -w -R 1

37 s (ahead) expressed with a minus sign because we want the PHC time

subtracted before it is compared with CLOCK_REALTIME

sudo phc2sys -s /dev/ptp0 -c CLOCK_REALTIME
-O -37 \ # compensate TAI->UTC
–step_threshold 0
–first_step_threshold 1
-m -w -R 1

sudo systemctl disable --now chronyd ntpd systemd-timesyncd gpsd

sudo ptp4l -2 -i eno1 -m -f /etc/linuxptp/ptp4l.conf

in /etc/linuxptp/ptp4l.conf

[global]
priority1 10
clockClass 6

utc_offset 37
network_transport L2
time_stamping hardware

tsproc_mode filter
delay_filter running_median
delay_filter_length 5

sudo ptp4l -2 -i MGBE -m -l 6 -f /etc/linuxptp/ptp4l.conf

2. Start phc2sys AFTER ptp4l reports “MASTER”

sudo phc2sys -s /dev/ptp0 -c CLOCK_REALTIME --first_step_threshold 1 --step_threshold 0.0002 -m -w -R 1
sudo phc2sys -s /dev/ptp0 -c CLOCK_REALTIME --first_step_threshold 1 --step_threshold 0.0002 -m -w -R 1

sudo phc2sys -s /dev/ptp0 -c CLOCK_REALTIME --first_step_threshold 1 --step_threshold 0.0001 -m -w -R 1

=====================================

sudo ptp4l -2 -i eno1 -m -f /etc/linuxptp/ptp4l.conf

sudo phc2sys -s /dev/ptp0 -c CLOCK_REALTIME --first_step_threshold 1 --step_threshold 0.0002 -m -w -R 1

Synchronizing the System Clock

The System clock synchronization step is mandatory only for those systems where applications rely on system clock to schedule traffic or present data (such as AVTP plugins from ALSA and GStreamer frameworks).

PHC time is set in TAI coordinate [1] while System clock time is set in UTC coordinate [2]. To ensure System clocks (CLOCK_REALTIME and CLOCK_TAI) are properly set, configure the UTC-TAI offset in the system. This is done by a run-time option from ptp4l that is set via pmc utility tool as shown below.

sudo pmc -u -b 0 -t 1 “SET GRANDMASTER_SETTINGS_NP clockClass 248
clockAccuracy 0xfe offsetScaledLogVariance 0xffff
currentUtcOffset 37 leap61 0 leap59 0 currentUtcOffsetValid 1
ptpTimescale 1 timeTraceable 1 frequencyTraceable 0
timeSource 0xa0”

sudo ptp4l -2 -i eno1 -m -f /usr/share/doc/linuxptp/configs/gPTP.cfg

sudo phc2sys -s eno1 -c CLOCK_REALTIME --step_threshold=1 --transportSpecific=1 -w -m

sudo phc2sys -s /dev/ptp0 -c CLOCK_REALTIME -O 0 -q -m -S 1000 --first_step_threshold 1 --step_threshold 0.0001 -R 1 -E linreg

Download, compile and run https://tsn.readthedocs.io/_downloads/f329e8dec804247b1dbb5835bd949e6f/check_clocks.c

gcc -o check_clocks check_clocks.c

sudo ./check_clocks -d eno1 -v