I have a Jetson Mate Advanced Carrier board with 4 NX cards mounted.
The NX card are all installed with NVidia SDK manager and flashed with Jetson OS + SDK Components + Deepstream.
I’ve experienced this issue since the beginning, that the internet connection randomly goes down. To fix it I have to run the commands:
sudo ifconfig eth0 down
sudo ifconfig eth0 up
The problem is that we’re controlling these over SSH so I cant really do those commands since the SSH connection goes down the moment eth0 goes down. All NX cards disappear from the network, but all comes back if I recycle with down/up command.
It seems that the Network Manager tries to reconnect and establish a link but hangs.
I’ve captured the syslog and the newtork manager service status:
Extracted from syslog file located /var/log/syslog
Nov 30 16:17:01 nx1-desktop CRON[21154]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Nov 30 16:17:50 nx1-desktop kernel: [ 1233.414610] uvcvideo: Failed to query (GET_CUR) UVC control 1 on unit 3: -32 (exp. 1024).
Nov 30 16:21:42 nx1-desktop systemd[1]: Starting Daily apt upgrade and clean activities...
Nov 30 16:21:44 nx1-desktop systemd[1]: Started Daily apt upgrade and clean activities.
Nov 30 16:22:50 nx1-desktop NetworkManager[4960]: <info> [1638285770.4447] manager: NetworkManager state is now CONNECTED_SITE
Nov 30 16:22:50 nx1-desktop whoopsie[5373]: [16:22:50] offline
Nov 30 16:22:50 nx1-desktop dbus-daemon[4791]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.9' (uid=0 pid=4960 comm="/usr/sbin/NetworkManager --no-daemon ")
Nov 30 16:22:50 nx1-desktop systemd[1]: Starting Network Manager Script Dispatcher Service...
Nov 30 16:22:50 nx1-desktop dbus-daemon[4791]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Nov 30 16:22:50 nx1-desktop systemd[1]: Started Network Manager Script Dispatcher Service.
Nov 30 16:22:50 nx1-desktop nm-dispatcher: req:1 'connectivity-change': new request (1 scripts)
Nov 30 16:22:50 nx1-desktop nm-dispatcher: req:1 'connectivity-change': start running ordered scripts...
Nov 30 16:23:01 nx1-desktop whoopsie[5373]: [16:23:01] Cannot reach: https://daisy.ubuntu.com
Extracted from command: ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.143 netmask 255.255.255.0 broadcast 192.168.1.255
inet6 fe80::4ab0:2dff:fe51:31a7 prefixlen 64 scopeid 0x20<link>
ether 48:b0:2d:51:31:a7 txqueuelen 1000 (Ethernet)
RX packets 740015 bytes 79722952 (79.7 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 691702 bytes 660803499 (660.8 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 37
Extracted from command: systemctl status NetworkManager.service
● NetworkManager.service - Network Manager
Loaded: loaded (/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2021-11-30 15:57:23 CET; 29min ago
Docs: man:NetworkManager(8)
Main PID: 4960 (NetworkManager)
Tasks: 4 (limit: 4915)
CGroup: /system.slice/NetworkManager.service
├─4960 /usr/sbin/NetworkManager --no-daemon
└─7174 /sbin/dhclient -d -q -sf /usr/lib/NetworkManager/nm-dhcp-helper -pf /run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-4f982c5
Nov 30 15:59:48 nx1-desktop dhclient[9009]: XMT: Info-Request on eth0, interval 1930ms.
Nov 30 15:59:50 nx1-desktop dhclient[9009]: XMT: Info-Request on eth0, interval 3870ms.
Nov 30 15:59:54 nx1-desktop dhclient[9009]: XMT: Info-Request on eth0, interval 7830ms.
Nov 30 16:00:01 nx1-desktop dhclient[9009]: XMT: Info-Request on eth0, interval 15480ms.
Nov 30 16:00:17 nx1-desktop dhclient[9009]: XMT: Info-Request on eth0, interval 30940ms.
Nov 30 16:00:32 nx1-desktop NetworkManager[4960]: <warn> [1638284432.4111] dhcp6 (eth0): request timed out
Nov 30 16:00:32 nx1-desktop NetworkManager[4960]: <info> [1638284432.4112] dhcp6 (eth0): state changed unknown -> timeout
Nov 30 16:00:32 nx1-desktop NetworkManager[4960]: <info> [1638284432.4126] dhcp6 (eth0): canceled DHCP transaction, DHCP client pid 9009
Nov 30 16:00:32 nx1-desktop NetworkManager[4960]: <info> [1638284432.4134] dhcp6 (eth0): state changed timeout -> done
Nov 30 16:22:50 nx1-desktop NetworkManager[4960]: <info> [1638285770.4447] manager: NetworkManager state is now CONNECTED_SITE
lines 1-20/20 (END)
Here is antoher syslog from when the eth0 went down:
Nov 30 09:08:37 nx1-desktop rsyslogd: [origin software="rsyslogd" swVersion="8.32.0" x-pid="4936" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Nov 30 09:08:38 nx1-desktop anacron[7889]: Job `cron.daily' terminated
Nov 30 09:08:38 nx1-desktop anacron[7889]: Normal exit (1 job run)
Nov 30 09:12:11 nx1-desktop dhclient[7173]: DHCPREQUEST of 192.168.1.143 on eth0 to 192.168.1.1 port 67 (xid=0xa6e36d7)
Nov 30 09:12:22 nx1-desktop dhclient[7173]: message repeated 2 times: [ DHCPREQUEST of 192.168.1.143 on eth0 to 192.168.1.1 port 67 (xid=0xa6e36d7)]
Nov 30 09:12:22 nx1-desktop dhclient[7173]: DHCPACK of 192.168.1.143 from 192.168.1.1
Nov 30 09:12:22 nx1-desktop NetworkManager[4927]: <info> [1638259942.3500] dhcp4 (eth0): address 192.168.1.143
Nov 30 09:12:22 nx1-desktop NetworkManager[4927]: <info> [1638259942.3502] dhcp4 (eth0): plen 24 (255.255.255.0)
Nov 30 09:12:22 nx1-desktop NetworkManager[4927]: <info> [1638259942.3502] dhcp4 (eth0): gateway 192.168.1.1
Nov 30 09:12:22 nx1-desktop NetworkManager[4927]: <info> [1638259942.3503] dhcp4 (eth0): lease time 86400
Nov 30 09:12:22 nx1-desktop NetworkManager[4927]: <info> [1638259942.3503] dhcp4 (eth0): hostname 'nx1-desktop'
Nov 30 09:12:22 nx1-desktop NetworkManager[4927]: <info> [1638259942.3504] dhcp4 (eth0): nameserver '192.168.1.1'
Nov 30 09:12:22 nx1-desktop NetworkManager[4927]: <info> [1638259942.3505] dhcp4 (eth0): state changed bound -> bound
Nov 30 09:12:22 nx1-desktop dbus-daemon[4686]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.9' (uid=0 pid=4927 comm="/usr/sbin/NetworkManager --no-daemon ")
Nov 30 09:12:22 nx1-desktop systemd[1]: Starting Network Manager Script Dispatcher Service...
Nov 30 09:12:22 nx1-desktop dhclient[7173]: bound to 192.168.1.143 -- renewal in 32126 seconds.
Nov 30 09:12:22 nx1-desktop dbus-daemon[4686]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Nov 30 09:12:22 nx1-desktop systemd[1]: Started Network Manager Script Dispatcher Service.
Nov 30 09:12:22 nx1-desktop nm-dispatcher: req:1 'dhcp4-change' [eth0]: new request (1 scripts)
Nov 30 09:12:22 nx1-desktop nm-dispatcher: req:1 'dhcp4-change' [eth0]: start running ordered scripts...
Nov 30 09:13:15 nx1-desktop kernel: [ 613.543266] uvcvideo: Failed to query (GET_CUR) UVC control 1 on unit 3: -32 (exp. 1024).
Nov 30 09:17:01 nx1-desktop CRON[1458]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Nov 30 09:18:08 nx1-desktop systemd-resolved[3358]: Grace period over, resuming full feature set (UDP+EDNS0) for DNS server 192.168.1.1.
Nov 30 09:18:08 nx1-desktop systemd[1]: Starting Cleanup of Temporary Directories...
Nov 30 09:18:08 nx1-desktop systemd[1]: Started Cleanup of Temporary Directories.
Nov 30 09:22:16 nx1-desktop kernel: [ 1155.018051] uvcvideo: Failed to query (GET_CUR) UVC control 1 on unit 3: -32 (exp. 1024).
Nov 30 09:28:35 nx1-desktop NetworkManager[4927]: <info> [1638260915.1407] manager: NetworkManager state is now CONNECTED_SITE
Nov 30 09:28:35 nx1-desktop dbus-daemon[4686]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.9' (uid=0 pid=4927 comm="/usr/sbin/NetworkManager --no-daemon ")
Nov 30 09:28:35 nx1-desktop whoopsie[5360]: [09:28:35] offline
Nov 30 09:28:35 nx1-desktop systemd[1]: Starting Network Manager Script Dispatcher Service...
Nov 30 09:28:35 nx1-desktop dbus-daemon[4686]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Nov 30 09:28:35 nx1-desktop systemd[1]: Started Network Manager Script Dispatcher Service.
Nov 30 09:28:35 nx1-desktop nm-dispatcher: req:1 'connectivity-change': new request (1 scripts)
Nov 30 09:28:35 nx1-desktop nm-dispatcher: req:1 'connectivity-change': start running ordered scripts...
Nov 30 09:28:46 nx1-desktop whoopsie[5360]: [09:28:46] Cannot reach: https://daisy.ubuntu.com
Nov 30 09:28:54 nx1-desktop systemd-timesyncd[3390]: Timed out waiting for reply from 62.169.217.225:123 (1.pool.ntp.org).
Nov 30 09:29:04 nx1-desktop systemd-timesyncd[3390]: Timed out waiting for reply from 208.67.75.242:123 (1.pool.ntp.org).
Nov 30 09:29:14 nx1-desktop systemd-timesyncd[3390]: Timed out waiting for reply from 162.254.66.243:123 (1.pool.ntp.org).
Nov 30 09:29:25 nx1-desktop systemd-timesyncd[3390]: Timed out waiting for reply from 185.17.70.106:123 (1.pool.ntp.org).
Nov 30 09:33:08 nx1-desktop systemd[1]: Starting Daily apt upgrade and clean activities...
Nov 30 09:33:09 nx1-desktop systemd[1]: Started Daily apt upgrade and clean activities.
And here is yet antoher one:
Dec 1 11:15:27 nx1-desktop kernel: [ 5421.015907] eqos 2490000.ether_qos eth0: Link is Down
Dec 1 11:15:30 nx1-desktop NetworkManager[4876]: <info> [1638353730.4998] device (eth0): carrier: link connected
Dec 1 11:15:30 nx1-desktop kernel: [ 5423.984033] eqos 2490000.ether_qos eth0: Link is Up - 1Gbps/Full - flow control rx/tx
Dec 1 11:15:30 nx1-desktop NetworkManager[4876]: <info> [1638353730.5001] device (eth0): DHCPv4 lease renewal requested
Dec 1 11:15:30 nx1-desktop NetworkManager[4876]: <info> [1638353730.5657] dhcp4 (eth0): canceled DHCP transaction, DHCP client pid 7193
Dec 1 11:15:30 nx1-desktop NetworkManager[4876]: <info> [1638353730.5658] dhcp4 (eth0): state changed bound -> done
Dec 1 11:15:30 nx1-desktop NetworkManager[4876]: <info> [1638353730.5692] dhcp4 (eth0): activation: beginning transaction (timeout in 45 seconds)
Dec 1 11:15:30 nx1-desktop NetworkManager[4876]: <info> [1638353730.5747] dhcp4 (eth0): dhclient started with pid 24343
Dec 1 11:15:30 nx1-desktop dhclient[24343]: DHCPREQUEST of 192.168.1.143 on eth0 to 255.255.255.255 port 67 (xid=0xe86b5e7)
Dec 1 11:15:30 nx1-desktop dhclient[24343]: DHCPACK of 192.168.1.143 from 192.168.1.1
Dec 1 11:15:30 nx1-desktop NetworkManager[4876]: <info> [1638353730.7147] dhcp4 (eth0): address 192.168.1.143
Dec 1 11:15:30 nx1-desktop NetworkManager[4876]: <info> [1638353730.7153] dhcp4 (eth0): plen 24 (255.255.255.0)
Dec 1 11:15:30 nx1-desktop NetworkManager[4876]: <info> [1638353730.7154] dhcp4 (eth0): gateway 192.168.1.1
Dec 1 11:15:30 nx1-desktop NetworkManager[4876]: <info> [1638353730.7155] dhcp4 (eth0): lease time 86400
Dec 1 11:15:30 nx1-desktop NetworkManager[4876]: <info> [1638353730.7156] dhcp4 (eth0): hostname 'nx1-desktop'
Dec 1 11:15:30 nx1-desktop NetworkManager[4876]: <info> [1638353730.7156] dhcp4 (eth0): nameserver '192.168.1.1'
Dec 1 11:15:30 nx1-desktop NetworkManager[4876]: <info> [1638353730.7157] dhcp4 (eth0): state changed unknown -> bound
Dec 1 11:15:30 nx1-desktop dbus-daemon[4650]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.9' (uid=0 pid=4876 comm="/usr/sbin/NetworkManager --no-daemon ")
Dec 1 11:15:30 nx1-desktop systemd[1]: Starting Network Manager Script Dispatcher Service...
Dec 1 11:15:30 nx1-desktop dhclient[24343]: bound to 192.168.1.143 -- renewal in 41994 seconds.
Dec 1 11:15:30 nx1-desktop dbus-daemon[4650]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Dec 1 11:15:30 nx1-desktop systemd[1]: Started Network Manager Script Dispatcher Service.
Dec 1 11:15:30 nx1-desktop nm-dispatcher: req:1 'dhcp4-change' [eth0]: new request (1 scripts)
Dec 1 11:15:30 nx1-desktop nm-dispatcher: req:1 'dhcp4-change' [eth0]: start running ordered scripts...
Dec 1 11:15:42 nx1-desktop NetworkManager[4876]: <info> [1638353742.8832] connectivity: (eth0) timed out
Dec 1 11:15:42 nx1-desktop NetworkManager[4876]: <info> [1638353742.8845] manager: NetworkManager state is now CONNECTED_SITE
Dec 1 11:15:42 nx1-desktop whoopsie[5476]: [11:15:42] offline
Dec 1 11:15:42 nx1-desktop dbus-daemon[4650]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.9' (uid=0 pid=4876 comm="/usr/sbin/NetworkManager --no-daemon ")
Dec 1 11:15:42 nx1-desktop systemd[1]: Starting Network Manager Script Dispatcher Service...
Dec 1 11:15:42 nx1-desktop dbus-daemon[4650]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Dec 1 11:15:42 nx1-desktop nm-dispatcher: req:1 'connectivity-change': new request (1 scripts)
Dec 1 11:15:42 nx1-desktop systemd[1]: Started Network Manager Script Dispatcher Service.
Dec 1 11:15:42 nx1-desktop nm-dispatcher: req:1 'connectivity-change': start running ordered scripts...
Dec 1 11:17:01 nx1-desktop CRON[29755]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)