On our AGX Orin devkit, we see an occasional bug when booting up.
There seems to be a link flap 4-5 secs after first connecting via the Ethernet port, and this link flap apparently comes exactly at a moment when NetworkManager registers the connection as active. I’m not sure what’s the reason of the link flap, but nevertheless, the NetworkManager should be able to catch all the networking events in order so that it doesn’t end up in a wrong state after boot.
Simplified, the event log looks like:
- nvethernet link up
- networkmanager link up event, start connecting
- networkmanager connected
- nvethernet link down (immediately after the previous one)
- nvethernet link up (4-5 s later)
- networkmanger disconnected (probably because of the previous carrier loss)
- this is it, nvethernet says link up, but ip link and networkmanager say no-carrier
To workaround this, a simple ip link set end0 down/up helps (only ip link set end0 up does not). Sometimes this needs to be done several times after the boot until it succeeds (we have a cron job running every minute, depending on particular boot, this needs to be done 0 to 3 times).
I don’t know the raw culprit, but it seems to me that some netlink events from the driver are not sent correctly or they’re sent at wrong time/order.
Our setup: Jetson AGX Orin devkit’s Ethernet port connected via Cat5e cable to a 1 Gbps Ethernet switch ( GigaBlox - Small GigaBlox Switch 5 Port GigaBit Switch ). This switch has known errata that when you connect 2 of those via same ports and start them at the same time, the link might not be established. However, the switch is powered on long before the Jetson does its boot, so I’m pretty sure this is not the case. The switch uses chip Realtek RTL8367N with integrated PHYs if it helps. We’re on Jetpack 7.2, we manage network exclusively via NetworkManager.
This is the complete relevant log. It seems the kernel messages from nvethernet are somehow a little bit delayed in the log because they usually come after the relevant systemd-networkd events:
Jul 28 17:04:48 helhest-jr-jetson modprobe: dwmac_tegra: not loading in-tree driver, out-of-tree nvethernet is present
Jul 28 17:04:48 helhest-jr-jetson kernel: nvethernet 6800000.ethernet: Adding to iommu group 38
Jul 28 17:04:48 helhest-jr-jetson kernel: nvethernet 6800000.ethernet: Virtualization is not enabled
Jul 28 17:04:48 helhest-jr-jetson kernel: nvethernet 6800000.ethernet: failed to read skip mac reset flag, default 0
Jul 28 17:04:48 helhest-jr-jetson kernel: nvethernet 6800000.ethernet: Failed to read nvida,disable-rx-checksum, so setting to default - rx checksum offload enabled
Jul 28 17:04:48 helhest-jr-jetson kernel: nvethernet 6800000.ethernet: setting to default DMA bit mask
Jul 28 17:04:48 helhest-jr-jetson kernel: nvethernet 6800000.ethernet: failed to read UPHY GBE mode- default to 10G
Jul 28 17:04:48 helhest-jr-jetson kernel: nvethernet 6800000.ethernet: failed to read or invalid MDC CR - default to 5
Jul 28 17:04:48 helhest-jr-jetson kernel: nvethernet 6800000.ethernet: failed to read or invalid force-restart-lane-bringup - default to 0
Sep 13 15:57:39 helhest-jr-jetson kernel: nvethernet 6800000.ethernet: macsec parameter is missing or disabled
Sep 13 15:57:39 helhest-jr-jetson kernel: nvethernet 6800000.ethernet: Macsec not supported/Not enabled
Sep 13 15:57:39 helhest-jr-jetson kernel: nvethernet 6800000.ethernet: eth0 created with 10 DMA channels
Sep 13 15:57:39 helhest-jr-jetson kernel: nvethernet 6800000.ethernet eth_switch: renamed from eth0
Sep 13 15:57:39 helhest-jr-jetson systemd-networkd[434]: eth0: Interface name change detected, renamed to eth_switch.
Sep 13 15:57:39 helhest-jr-jetson kernel: nvethernet 6800000.ethernet eth_switch: ether_get_wol: phydev is null check iface up status
Sep 13 15:57:44 helhest-jr-jetson NetworkManager[2173]: <info> [1789307864.9063] manager: (eth_switch): new Ethernet device (/org/freedesktop/NetworkManager/Devices/4)
Sep 13 15:57:44 helhest-jr-jetson NetworkManager[2173]: <info> [1789307864.9067] device (eth_switch): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Sep 13 15:57:45 helhest-jr-jetson kernel: nvethernet 6800000.ethernet: IVC channel not reserved - virtualization disabled
Sep 13 15:57:46 helhest-jr-jetson systemd-networkd[434]: eth_switch: Link UP
Sep 13 15:57:50 helhest-jr-jetson systemd-networkd[434]: eth_switch: Gained carrier
Sep 13 15:57:50 helhest-jr-jetson NetworkManager[2173]: <info> [1789307870.2542] device (eth_switch): carrier: link connected
Sep 13 15:57:50 helhest-jr-jetson kernel: nvethernet 6800000.ethernet: [xpcs_lane_bring_up][798][type:0x4][loga-0x0] PCS block lock SUCCESS
Sep 13 15:57:50 helhest-jr-jetson NetworkManager[2173]: <info> [1789307870.3639] device (eth_switch): state change: unavailable -> disconnected (reason 'carrier-changed', sys-iface-state: 'managed')
Sep 13 15:57:50 helhest-jr-jetson NetworkManager[2173]: <info> [1789307870.3654] device (eth_switch): Activation: starting connection 'eth' (667abb96-cbb8-32a9-bc2e-3db4e9027411)
Sep 13 15:57:50 helhest-jr-jetson NetworkManager[2173]: <info> [1789307870.3655] device (eth_switch): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Sep 13 15:57:50 helhest-jr-jetson kernel: nvethernet 6800000.ethernet eth_switch: Link is Up - 1Gbps/Full - flow control rx/tx
Sep 13 15:57:50 helhest-jr-jetson NetworkManager[2173]: <info> [1789307870.3667] device (eth_switch): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Sep 13 15:57:50 helhest-jr-jetson NetworkManager[2173]: <info> [1789307870.3674] device (eth_switch): state change: config -> ip-config (reason 'none', sys-iface-state: 'managed')
Sep 13 15:57:50 helhest-jr-jetson NetworkManager[2173]: <info> [1789307870.3685] policy: set 'eth' (eth_switch) as default for IPv4 routing and DNS
Sep 13 15:57:50 helhest-jr-jetson NetworkManager[2173]: <info> [1789307870.5087] device (eth_switch): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'managed')
Sep 13 15:57:50 helhest-jr-jetson kernel: nvethernet 6800000.ethernet eth_switch: EEE off. Set Tx & Rx LPI off
Sep 13 15:57:50 helhest-jr-jetson NetworkManager[2173]: <info> [1789307870.5236] device (eth_switch): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'managed')
Sep 13 15:57:50 helhest-jr-jetson NetworkManager[2173]: <info> [1789307870.5240] device (eth_switch): state change: secondaries -> activated (reason 'none', sys-iface-state: 'managed')
Sep 13 15:57:50 helhest-jr-jetson NetworkManager[2173]: <info> [1789307870.5248] device (eth_switch): Activation: successful, device activated.
Sep 13 15:57:50 helhest-jr-jetson kernel: nvethernet 6800000.ethernet eth_switch: Link is Down
Sep 13 15:57:50 helhest-jr-jetson kernel: nvethernet 6800000.ethernet: [xpcs_lane_bring_up][798][type:0x4][loga-0x0] PCS block lock SUCCESS
Sep 13 15:57:51 helhest-jr-jetson systemd-networkd[434]: eth_switch: Lost carrier
Sep 13 15:57:54 helhest-jr-jetson systemd-networkd[434]: eth_switch: Gained carrier
Sep 13 15:57:54 helhest-jr-jetson NetworkManager[2173]: <info> [1789307874.6308] device (eth_switch): carrier: link connected
Sep 13 15:57:54 helhest-jr-jetson kernel: nvethernet 6800000.ethernet: [xpcs_lane_bring_up][798][type:0x4][loga-0x0] PCS block lock SUCCESS
Sep 13 15:57:54 helhest-jr-jetson systemd-networkd[434]: eth_switch: Lost carrier
Sep 13 15:57:54 helhest-jr-jetson kernel: nvethernet 6800000.ethernet eth_switch: Link is Up - 1Gbps/Full - flow control rx/tx
Sep 13 15:58:00 helhest-jr-jetson NetworkManager[2173]: <info> [1789307880.7807] device (eth_switch): state change: activated -> unavailable (reason 'carrier-changed', sys-iface-state: 'managed')