SN2100 NTP Debugging

We have three SN2100 switches all running Onyx 3.10.4404. All are able to pick up NTP from the mgmt0 VRF interface using ntpdate, so we know there is connectivity to a server…

ar-sn2100-03 [standalone: master] (config) # ntpdate time.google.com
 4 Mar 15:33:06 ntpdate[3098184]: adjust time server 216.239.35.12 offset 0.000134 sec

But we cant seem to see the switch query the NTP server on its own. And we always see “Clock is unsynchronized”. For example…

ar-sn2100-03 [standalone: master] (config) # show ntp

NTP is administratively            : enabled
VRF name                           : default
NTP Authentication administratively: disabled
NTP server role                    : enabled

Clock is unsynchronized.

Active servers and peers:
  2001:4860:4806:4:::
    Configured as      : time.google.com
    Conf Type          : serv
    Status             : pending    
    Stratum            : 16 
    Offset(msec)       : 0.000
    Ref clock          : .INIT.         
    Poll Interval (sec): 128 
    Last Response (sec): N/A 
    Auth state         : none

Feels like I’m missing something basic - would appreciate a pointer to my cockpit error without too much shaming.

please share the output of these 2 commands

show ntp vrf all source-interface
show ntp configured

1 Like

Both of the switches I tried have the same source-interface setting as shown here…

ar-sn2100-02 [standalone: master] (config) # show ntp vrf all source-interface 

VRF name: default

Source IP for ntp client:
  Configured: none
  Current   : none
  IPv4-addr : none
  IPv6-addr : none

And slightly different results of the two switches ntp configured…

ar-sn2100-02 [standalone: master] (config) # show ntp configured 
NTP enabled: yes
NTP Authentication enabled: no
NTP Server role disabled: yes

No NTP peers configured.
NTP server 132.163.96.5
  Enabled: yes
  NTP version: 4
  Key ID: none
  Trusted: no

NTP server ntp-b.nist.gov
  Resolved as: 132.163.96.5
  Enabled: yes
  NTP version: 4
  Key ID: none
  Trusted: no

NTP server time.google.com
  Resolved as: 2001:4860:4806:4::
  Enabled: yes
  NTP version: 3
  Key ID: none
  Trusted: no

and

ar-sn2100-03 [standalone: master] (config) # show ntp configured 
NTP enabled: yes
NTP Authentication enabled: no
NTP Server role disabled: no

No NTP peers configured.
NTP server time.google.com
  Resolved as: 2001:4860:4806:4::
  Enabled: yes
  NTP version: 4
  Key ID: none
  Trusted: yes

It looks like there is a clue here, but I’m not sure where. Next steps @derxu ? And thank you so much in advance! -Shep

Please configure a source IP for NTP on vrf default, then see if it can be synced automatically.
If no, please try NTP syncing with some other public or your private NTP servers better as the close ones (with Stratum in low values). Also please try this with the source IP set.
The command ‘ntpdate’ is ‘a one -time operation and does not cause the clock to be kept in sync’ from Onyx user manual.

Hi @derxu . I’m not exactly sure how to do that. Let me explain what we are trying to do, maybe there is a better way. We have a set of sn2100/2700 switches for our 100G infrastructure. All the management interfaces on are on a class-c control plane LAN with gateway access to the internet. When the switches are power-cycled, they lose their clock. We would like to at least get them close with a configuration that uses NTP to get approximate time. That the one-time ‘ntpdate’ works, suggests that there may be a solution. Can you explain to me exactly how to configure a source IP for NTP on vrf default? It was not clear to me from the docs.

I did create a loopback0 interface and gave it a legit unused address on the control plane, but then nothing worked, not even ‘ntpdate’…

ar-sn2100-02 [standalone: master] (config) # show ntp vrf all source-interface 

VRF name: default

Source IP for ntp client:
  Configured: loopback0
  Current   : loopback0
  IPv4-addr : 10.0.209.42
  IPv6-addr : none
ar-sn2100-02 [standalone: master] (config) # ntpdate ntp-b.nist.gov
%  7 Mar 15:32:46 ntpdate[412659]: no server suitable for synchronization found

If it is is needed, we can place a hardware NTP server on the class-c control plane LAN to which all the management ports connect. That’s not ideal, but we can do that.

Also, we do have multiple PTP grandmasters connected to several switches (for other purposes besides this). These are connected directly to data plane switch ports of the switch. We could enable the NTP server(s) in any of the PTP grandmasters to allows the switch NTP client to try to get NTP time from some port other than the management port(s). Feels like this gymnastic exercise should not be needed, just saying that it is there too.

Obviously this isn’t my day job - so appreciate the kind feedback. Guessing that this dance would be about the same if we were running Cumulus instead of Onyx?

Hi Shepard,

Please run these commands first to check the present switch configuration,

show vrf – which vrfs are ready out there

show interface mgmt0 - this displays on which vrf mgmt0 is located on

Once located the mgmt 0 interface, then config the IP add used to communicate to the NTP server.

ntp vrf source-interface

Then save the config by

write memory

show ntp – validation

All the above commands can be executed in config mode.

BTW, Onyx has been announced with extended Tech support till 2025 on releases for bug fix and security patches. No more new features will be developed.

Please have this in mind and prepare for transition to cumulus linux.

Thanks

show VRF makes sense…

ar-sn2100-02 [standalone: master] (config) # show vrf

VRF Info:
   Name: default
   RD: NA
   Description: NA
   IP routing state: Enabled
   IPv6 routing state: Disabled
   IP multicast routing state: Disabled
   Protocols: IPv4
   Interfaces: Loopback0

so does show interface…

ar-sn2100-02 [standalone: master] (config) # show interfaces mgmt0 

Interface mgmt0 status:
  Comment         : 
  VRF             : vrf-default
  Admin up        : yes
  Link up         : yes
  DHCP running    : yes
  IP address      : 10.0.209.183
  Netmask         : 255.255.255.0
  IPv6 enabled    : no
  Speed           : 1000Mb/s (auto)
  Duplex          : full (auto)
  Interface type  : ethernet
  Interface source: bridge
  Bonding master  : vrf_vrf-default
  MTU             : 1500
  HW address      : B8:59:9F:EA:FC:40

  Rx:
    1326895890  bytes
       6184036  packets
             0  mcast packets
             0  discards
             0  errors
             0  overruns
             0  frame

  Tx:
    1818115531  bytes
        351826  packets
             0  discards
             0  errors
             0  overruns
             0  carrier
             0  collisions
          1000  queue len

Now for the VRF, I assumed hooking it to loopback0 was the correct thing to do?

ar-sn2100-02 [standalone: master] (config) # ntp vrf default source-interface loopback0 

After write memory and some other fooling around, still no luck…

ar-sn2100-02 [standalone: master] (config) # show ntp

NTP is administratively            : enabled
VRF name                           : default
NTP Authentication administratively: disabled
NTP server role                    : disabled

Clock is unsynchronized.

Active servers and peers:
  132.163.96.5:
    Configured as      : ntp-b.nist.gov
    Conf Type          : serv
    Status             : pending    
    Stratum            : 16 
    Offset(msec)       : 0.000
    Ref clock          : .INIT.         
    Poll Interval (sec): 1024
    Last Response (sec): N/A 
    Auth state         : none    

  2001:4860:4806:4:::
    Configured as      : time.google.com
    Conf Type          : serv
    Status             : pending    
    Stratum            : 16 
    Offset(msec)       : 0.000
    Ref clock          : .INIT.         
    Poll Interval (sec): 1024
    Last Response (sec): N/A 
    Auth state         : none    

So not sure where Im going wrong. Also, ever since I created the loopback0 VRF, this switch doesn’t even allow ntpdate…

ar-sn2100-02 [standalone: master] (config) # ntpdate ntp-b.nist.gov
%  7 Mar 16:58:36 ntpdate[418156]: no server suitable for synchronization found

Here are some sections from “show configuration” in case these is a clue to what I am doing wrong…

##
## L3 configuration
##
   ip routing vrf default
   interface loopback 0
   interface loopback 0 ip address 10.0.209.42/32 primary
   interface loopback 0 ip address 10.0.209.183/32
   ntp vrf default source-interface loopback0
   
##
## Other IP configuration
##
   hostname ar-sn2100-02
   ip route vrf default 0.0.0.0/0 10.0.209.1
   
##
## Other IPv6 configuration
##
no ipv6 enable

##
## Network management configuration
##
# web proxy auth basic password ********
no ntp server 132.163.96.5 disable
   ntp server 132.163.96.5 keyID 0
no ntp server 132.163.96.5 trusted-enable
   ntp server 132.163.96.5 version 4
no ntp server ntp-b.nist.gov disable
   ntp server ntp-b.nist.gov keyID 0
no ntp server ntp-b.nist.gov trusted-enable
   ntp server ntp-b.nist.gov version 4
no ntp server time.google.com disable
   ntp server time.google.com keyID 0
no ntp server time.google.com trusted-enable
   ntp server time.google.com version 3
   ntp server-role disable
   ntp vrf default enable
   telnet-server enable
   terminal sysrq enable
no web auto-logout
   web http enable
no web https ssl secure-cookie enable
   web vrf default enable
   
##
## IPv4 packet filtering configuration
##
no ip filter chain forward rule all
no ip filter chain input rule all
no ip filter chain logging rule all
no ip filter chain output rule all
no ip filter enable

When you config the Lo0 as the source interface, can the lo0 interface IP ping to the NTP server?

If you do want to use a separate vrf, mgmt can be defined as the name. And interface mgmt0 or lo0 needs to be placed into the vrf with the route reachable from the interface to the NTP server.

Currently, only one ‘default’ vrf is there. To make it simple, you can leave the Lo0 with vrf default but needs to make sure lo0 IP is able to ping to the NTP server whichever it is, such as 8.8.8.8 or 8.8.4.4.

No luck with ping through the Lo0 interface.

ar-sn2100-02 [standalone: master] (config) # ping -I loopback0 8.8.8.8

PING 8.8.8.8 (8.8.8.8) from 10.0.209.183 loopback0: 56(84) bytes of data.

^C

--- 8.8.8.8 ping statistics ---

8 packets transmitted, 0 received, 100% packet loss, time 7207ms

Hi @derxu If you have the time, I would be happy to jump on a screenshare call with you. I’m free this afternoon except for 2PM-330PM ET. If you can, shoot me you email address privately and I’ll send you an invite at the day/time of your preference. Shep

shepard.siegel@atomicrules.com

Hi Shepard,

If need more detailed tech support, please kindly raise Tech support ticket.

We will have Tech support team assissting you for best effort.

Thanks

1 Like

So from your Lo0 IP is not reachable to the NTP server.
This is routing issue, you need to get this fixed before syncing NTP with Lo0.

Thanks derxu. I’ve created a case 00674360 in the Enterprise Support Portal referencing this forum thread. You’ve been super-helpful, thanks. I’ll update this thread when we understated why this works:

ar-sn2100-03 [standalone: master] (config) # ping google.com
PING google.com (142.251.40.142) 56(84) bytes of data.
64 bytes from lga25s80-in-f14.1e100.net (142.251.40.142): icmp_seq=1 ttl=56 time=27.9 ms
64 bytes from lga25s80-in-f14.1e100.net (142.251.40.142): icmp_seq=2 ttl=56 time=24.2 ms
^C
--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 24.197/26.056/27.916/1.866 ms

and this fails:

ar-sn2100-03 [standalone: master] (config) # ping -I loopback0 google.com
PING google.com (142.250.80.46) from 10.0.209.203 loopback0: 56(84) bytes of data.
^C
--- google.com ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time 6137ms