Jetson Nano Disconnects from WiFi and Never Reconnects

aast226 · March 21, 2020, 12:56am

I am currently using Jetson Nanos in order to stream images over the internet and am experiencing problems with maintaining a wifi connection. Specifically, the Nanos disconnect from the internet and are never able to reconnect. Why might this be the case?

Additional context:

I’ve experienced this problem when using each of the following WiFi dongles:
I am currently in the early stages of evaluating the following WiFi dongles:
- Panda 300Mbps Wireless N USB Adapter (currently testing)
- Panda Wireless PAU06 300Mbps Wireless N USB Adapter - w/High Gain Antenna (currently testing)
I’ve also tested with the Intel Dual Band Wireless-Ac 8265 and it did not solve the problem.
I’ve also experienced this problem on two different OS’s: BalenaOS and Ubuntu 18.04 from Jetpack 32.3.1.
- For a while, I worked with Balena on this issue since I assumed that the problem had to do with their OS. After working with them for some time, they released a software package called keep-wifi-up which, for the most part, failed to keep devices online.
- Recently, I’ve been testing WiFi connectivity on devices running Ubuntu 18.04 to see if they eventually disconnect from the internet and never reconnect. Unfortunately, these devices also went offline and never recovered.
Miscellaneous notes:
- When an Ubuntu goes offline, usually running “sudo service network-manager restart” will quickly bring the device back online.
- WiFi dongles always appear to have power saving mode enabled by default.
- Rebooting a device usually brings it back online (I do not have direct access to many of the devices I test, so manually rebooting is not always an option).
Questions:
- Why might my devices be going offline and not reconnecting?
- What are best practices/things to know for connecting the Jetson Nano to WiFi?
- What are some good sources on how to build a “health monitor” for ensuring stable network connectivity?

alanz · March 23, 2020, 10:34am

Hi, aast226

Questions:

Why might my devices be going offline and not reconnecting?
There are a few rootcase may lead to this disconnection.

The signal lost. Too week to keep the connection.
The power manager disconnect the connection to save the power. Which should be reconnection when the system recover back to normal state.
There are some critical bugs dead the driver.

What are best practices/things to know for connecting the Jetson Nano to WiFi?
Ubuntu on jetson should behavior the same with other patforms as x86.
What are some good sources on how to build a “health monitor” for ensuring stable network connectivity?
Simply you can try to ping the board to monitor the stability of the network. When there is a disconnection, save and share the log of the dmesg to see what happend there.

aast226 · March 23, 2020, 6:19pm

Hi alanz,

Thanks for the reply. I’ve responded to each of your suggestions below:

I have three Nanos connected to the same network at my house. Over the weekend, only one of them disconnected and never came back online. As far as I can tell, the other two remained online the entire time.

What causes the system to change back to a normal state?

Do you have any more information on this? This seems like the most likely cause of the problem, but I’m not sure what the best way to debug this is.

As I mentioned above, one of my devices went offline over the weekend. This is the output of running dmesg on that device:

Let me know if there’s any other information that I can provide you with.

mdegans · March 24, 2020, 2:25am

I’ve been testing WiFi connectivity on devices running Ubuntu 18.04 to see if they eventually disconnect from the internet and never reconnect. Unfortunately, these devices also went offline and never recovered.

Have you tried with another hotspot?
How are you checking the connectivity?
You say 2 out of the 3 Nanos are fine. Have you tried swapping locations, sd cards, and/or power supplies?

I see a lot of this in your log:

[135194.233276] wlan0: authenticate with 50:d4:f7:0f:ca:02
[135194.254370] wlan0: send auth to 50:d4:f7:0f:ca:02 (try 1/3)
[135194.418143] wlan0: send auth to 50:d4:f7:0f:ca:02 (try 2/3)
[135194.523142] wlan0: send auth to 50:d4:f7:0f:ca:02 (try 3/3)
[135194.666764] wlan0: authentication with 50:d4:f7:0f:ca:02 timed out
[135196.963609] wlan0: authenticate with 50:d4:f7:0f:ca:02
[135196.998385] wlan0: send auth to 50:d4:f7:0f:ca:02 (try 1/3)
[135197.012715] wlan0: authenticated

It looks like a good portion of the time it tries to authenticate with your base station, it’s timing out, while occasionally it works. I do not believe it to be driver related since you are using various usb chipsets as well as an Intel card which is known to be very stable (Intel has excellent Linux support).

I suspect more it’s an issue with the hotspot not liking how Ubuntu tries to authenticate or possibly signal strength related. It may be a known issue for your base station and Linux if you Google in which case there could be a firmware update. In any case, if you try with anoter hotspot (eg. a phone), you’ll be able to know for sure.

aast226 · March 25, 2020, 4:01am

Hey mdegans,

Thanks for the input. With respect to your questions:

With these devices, I have not. I have a WiFi range extender that I could set up in close proximity (< 1 meter away) to these devices, so I will try that and see if behavior changes.

I wrote a systemd service that periodically runs the following command:
wget -q --spider http://google.com
If nothing is returned, I know that the device is online

The three Nanos are all next to each other, but I can try swapping SD cards and power supplies.

Today, I have noticed one of my devices has been frequently disconnecting and reconnecting from the internet. Trying to debug this, I ran the commands journalctl -xe and sudo service NetworkManager status and saw the following output:
journalctl -xe:

Mar 24 16:22:11 nvidia containerd[4640]: time="2020-03-24T16:22:11.727787125-07:00" level=info msg="shim reaped" id=b50508c00e6bbff52259f2097bbbd2eaf1a8d575d7ee540866a26726a4d1eb77
Mar 24 16:22:11 nvidia dockerd[4645]: time="2020-03-24T16:22:11.745831068-07:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Mar 24 16:22:14 nvidia bash[8692]: Cannot get the NetworkManager service state
Mar 24 16:22:14 nvidia bash[8692]: [network_manager::dbus_api:ERROR] org.freedesktop.NetworkManager.Settings::ListConnections method call failed on /org/freedesktop/NetworkManager/Settings
Mar 24 16:22:14 nvidia bash[8692]: Error: Deleting access point connection profile failed
Mar 24 16:22:14 nvidia bash[8692]:   caused by: D-Bus failure: org.freedesktop.NetworkManager.Settings::ListConnections method call failed on /org/freedesktop/NetworkManager/Settings
Mar 24 16:22:14 nvidia bash[8692]:   caused by: "The name org.freedesktop.NetworkManager was not provided by any .service files"
Mar 24 16:22:15 nvidia containerd[4640]: time="2020-03-24T16:22:15.255901910-07:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/b50508c00e6bbff52259f2097bbbd2eaf1a8d575d7ee540866a26726a4d1eb77/shim.sock" debug=false pid=22772
Mar 24 16:22:15 nvidia kernel: usbcore: deregistering interface driver uvcvideo
Mar 24 16:22:15 nvidia kernel: usbcore: registered new interface driver uvcvideo
Mar 24 16:22:15 nvidia kernel: USB Video Class driver (1.1.1)
Mar 24 16:22:19 nvidia sudo[22877]:     root : TTY=unknown ; PWD=/home/nvidia ; USER=root ; COMMAND=/usr/sbin/service network-manager restart
Mar 24 16:22:19 nvidia sudo[22877]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 24 16:22:32 nvidia NetworkManager[22657]: <info>  [1585092152.9001] hostname: hostname: hostnamed not used as proxy creation failed with: Error calling StartServiceByName for org.freedesktop.hostname1: Timeout was reached
Mar 24 16:22:32 nvidia dbus-daemon[3735]: [system] Failed to activate service 'org.freedesktop.hostname1': timed out (service_start_timeout=25000ms)
Mar 24 16:22:32 nvidia dbus-daemon[3735]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Mar 24 16:22:32 nvidia NetworkManager[22657]: <info>  [1585092152.9056] hostname: hostname changed from (none) to "nvidia"
Mar 24 16:22:32 nvidia NetworkManager[22657]: <info>  [1585092152.9067] dns-mgr[0x5590773160]: init: dns=systemd-resolved, rc-manager=symlink, plugin=systemd-resolved
Mar 24 16:22:32 nvidia NetworkManager[22657]: <info>  [1585092152.9094] rfkill1: found WiFi radio killswitch (at /sys/devices/70090000.xusb/usb1/1-2/1-2.1/1-2.1:1.0/ieee80211/phy1/rfkill1) (driver rt2800usb)
Mar 24 16:22:32 nvidia NetworkManager[22657]: <info>  [1585092152.9099] manager[0x5590784040]: rfkill: WiFi hardware radio set enabled
Mar 24 16:22:32 nvidia NetworkManager[22657]: <info>  [1585092152.9100] manager[0x5590784040]: rfkill: WWAN hardware radio set enabled
Mar 24 16:22:32 nvidia dbus-daemon[3735]: [system] Activating systemd to hand-off: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.1698' (uid=0 pid=22657 comm="/usr/sbin/NetworkManager --no-daemon ")
Mar 24 16:22:51 nvidia /usr/lib/gdm3/gdm-x-session[7090]: (--) NVIDIA(GPU-0): LG Electronics LG HDR WQHD (DFP-0): connected
Mar 24 16:22:51 nvidia /usr/lib/gdm3/gdm-x-session[7090]: (--) NVIDIA(GPU-0): LG Electronics LG HDR WQHD (DFP-0): External TMDS
Mar 24 16:22:57 nvidia dbus-daemon[3735]: [system] Failed to activate service 'org.freedesktop.nm_dispatcher': timed out (service_start_timeout=25000ms)
Mar 24 16:22:57 nvidia NetworkManager[22657]: <error> [1585092177.9269] dispatcher: could not get dispatcher proxy! Error calling StartServiceByName for org.freedesktop.nm_dispatcher: Timeout was reached
Mar 24 16:22:57 nvidia dbus-daemon[3735]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9473] init!
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9496]       interface-parser: parsing file /etc/network/interfaces
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9515]       interface-parser: source line includes interfaces file(s) /etc/network/interfaces.d
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9532]       interface-parser: finished parsing file /etc/network/interfaces
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9538] guessed connection type (wwan0) = 802-3-ethernet
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9539] update_connection_setting_from_if_block: name:wwan0, type:802-3-ethernet, id:Ifupdown (wwan0), uuid: 00a39b5d-18d2-ea89-fbec-a181dcc3ba54
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9546] adding wwan0 to connections
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9547] adding iface wwan0 to eni_ifaces
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9548] autoconnect
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9549] management mode: unmanaged
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9617] devices added (path: /sys/devices/1003000.pcie/pci0000:00/0000:00:02.0/0000:01:00.0/net/eth0, iface: eth0)
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9618] device added (path: /sys/devices/1003000.pcie/pci0000:00/0000:00:02.0/0000:01:00.0/net/eth0, iface: eth0): no ifupdown configuration found.
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9620] devices added (path: /sys/devices/70090000.xusb/usb1/1-2/1-2.1/1-2.1:1.0/net/wlan0, iface: wlan0)
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9621] device added (path: /sys/devices/70090000.xusb/usb1/1-2/1-2.1/1-2.1:1.0/net/wlan0, iface: wlan0): no ifupdown configuration found.
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9623] devices added (path: /sys/devices/700d0000.xudc/gadget/net/rndis0, iface: rndis0)
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9623] device added (path: /sys/devices/700d0000.xudc/gadget/net/rndis0, iface: rndis0): no ifupdown configuration found.
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9625] devices added (path: /sys/devices/700d0000.xudc/gadget/net/usb0, iface: usb0)
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9626] device added (path: /sys/devices/700d0000.xudc/gadget/net/usb0, iface: usb0): no ifupdown configuration found.
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9627] devices added (path: /sys/devices/virtual/net/docker0, iface: docker0)
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9628] device added (path: /sys/devices/virtual/net/docker0, iface: docker0): no ifupdown configuration found.
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9629] devices added (path: /sys/devices/virtual/net/dummy0, iface: dummy0)
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9630] device added (path: /sys/devices/virtual/net/dummy0, iface: dummy0): no ifupdown configuration found.
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9632] devices added (path: /sys/devices/virtual/net/l4tbr0, iface: l4tbr0)
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9632] device added (path: /sys/devices/virtual/net/l4tbr0, iface: l4tbr0): no ifupdown configuration found.
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9634] devices added (path: /sys/devices/virtual/net/lo, iface: lo)
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9634] device added (path: /sys/devices/virtual/net/lo, iface: lo): no ifupdown configuration found.
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9635] end _init.
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9636] settings: loaded plugin ifupdown: (C) 2008 Canonical Ltd.  To report bugs please use the NetworkManager mailing list. (/usr/lib/aarch64-linux-gnu/NetworkManager/libnm-settings-plugin-ifupdown.so)
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9638] settings: loaded plugin keyfile: (c) 2007 - 2016 Red Hat, Inc.  To report bugs please use the NetworkManager mailing list.
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9639] (-1870928512) ... get_connections.
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9639] (-1870928512) ... get_connections (managed=false): return empty list.
Mar 24 16:22:57 nvidia NetworkManager[22657]: <info>  [1585092177.9991] keyfile: new connection /etc/NetworkManager/system-connections/NETGEAR92 (7985aeb1-bfc2-4f01-a0b5-b223c2116c4a,"NETGEAR92")
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.0317] keyfile: new connection /etc/NetworkManager/system-connections/soracom (2a21a2b1-ecb0-4ebb-a8d9-4febb38985e1,"soracom")
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.0663] keyfile: new connection /etc/NetworkManager/system-connections/curbFlow (27609f8c-6232-4df0-a3dd-38dd8845f096,"curbFlow")
Mar 24 16:22:58 nvidia NetworkManager[22657]: <warn>  [1585092178.0666] keyfile: error loading connection from file /etc/NetworkManager/system-connections/CurbSensor: invalid connection: connection.type: property is missing
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.0745] get unmanaged devices count: 0
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.0746] manager: rfkill: WiFi enabled by radio killswitch; enabled by state file
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.0747] manager: rfkill: WWAN enabled by radio killswitch; enabled by state file
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.0748] manager: Networking is enabled by state file
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.0750] dhcp-init: Using DHCP client 'dhclient'
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.0751] Loaded device plugin: NMBondDeviceFactory (internal)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.0752] Loaded device plugin: NMBridgeDeviceFactory (internal)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.0752] Loaded device plugin: NMDummyDeviceFactory (internal)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.0753] Loaded device plugin: NMEthernetDeviceFactory (internal)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.0753] Loaded device plugin: NMInfinibandDeviceFactory (internal)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.0754] Loaded device plugin: NMIPTunnelDeviceFactory (internal)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.0755] Loaded device plugin: NMMacsecDeviceFactory (internal)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.0755] Loaded device plugin: NMMacvlanDeviceFactory (internal)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.0756] Loaded device plugin: NMPppDeviceFactory (internal)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.0756] Loaded device plugin: NMTunDeviceFactory (internal)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.0757] Loaded device plugin: NMVethDeviceFactory (internal)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.0757] Loaded device plugin: NMVlanDeviceFactory (internal)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.0758] Loaded device plugin: NMVxlanDeviceFactory (internal)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.0818] Loaded device plugin: NMTeamFactory (/usr/lib/aarch64-linux-gnu/NetworkManager/libnm-device-plugin-team.so)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.0855] Loaded device plugin: NMWifiFactory (/usr/lib/aarch64-linux-gnu/NetworkManager/libnm-device-plugin-wifi.so)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.0930] Loaded device plugin: NMWwanFactory (/usr/lib/aarch64-linux-gnu/NetworkManager/libnm-device-plugin-wwan.so)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.0972] Loaded device plugin: NMAtmManager (/usr/lib/aarch64-linux-gnu/NetworkManager/libnm-device-plugin-adsl.so)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.1036] Loaded device plugin: NMBluezManager (/usr/lib/aarch64-linux-gnu/NetworkManager/libnm-device-plugin-bluetooth.so)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.1079] device (lo): carrier: link connected
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.1101] manager: (lo): new Generic device (/org/freedesktop/NetworkManager/Devices/1)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.1133] manager: (docker0): new Bridge device (/org/freedesktop/NetworkManager/Devices/2)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.1214] keyfile: add connection in-memory (31e0eb5a-8da4-43c3-84dc-8f4a059a9ca1,"docker0")
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.1240] device (docker0): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external')
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.1250] device (docker0): state change: unavailable -> disconnected (reason 'connection-assumed', sys-iface-state: 'external')
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.1278] device (docker0): Activation: starting connection 'docker0' (31e0eb5a-8da4-43c3-84dc-8f4a059a9ca1)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.1304] manager: (dummy0): new Dummy device (/org/freedesktop/NetworkManager/Devices/3)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.1342] manager: (eth0): new Ethernet device (/org/freedesktop/NetworkManager/Devices/4)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.1367] device (eth0): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Mar 24 16:22:58 nvidia kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.1459] manager: (l4tbr0): new Bridge device (/org/freedesktop/NetworkManager/Devices/5)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.1492] manager: (rndis0): new Ethernet device (/org/freedesktop/NetworkManager/Devices/6)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.1525] manager: (usb0): new Ethernet device (/org/freedesktop/NetworkManager/Devices/7)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.1536] wifi-nl80211: (wlan0): using nl80211 for WiFi device control
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.1542] device (wlan0): driver supports Access Point (AP) mode
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.1572] manager: (wlan0): new 802.11 WiFi device (/org/freedesktop/NetworkManager/Devices/8)
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.1597] device (wlan0): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Mar 24 16:22:58 nvidia kernel: IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
Mar 24 16:22:58 nvidia kernel: IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
Mar 24 16:22:58 nvidia NetworkManager[22657]: <warn>  [1585092178.5548] Error: failed to open /run/network/ifstate
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.5556] caught SIGTERM, shutting down normally.
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.5769] device (rndis0): released from master device l4tbr0
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.5782] device (usb0): released from master device l4tbr0
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.5798] device (wlan0): state change: unavailable -> unmanaged (reason 'unmanaged', sys-iface-state: 'managed')
Mar 24 16:22:58 nvidia NetworkManager[22657]: <info>  [1585092178.6802] exiting (success)
Mar 24 16:22:58 nvidia systemd[1]: Stopped Network Manager.
-- Subject: Unit NetworkManager.service has finished shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit NetworkManager.service has finished shutting down.
Mar 24 16:22:58 nvidia systemd[1]: Starting Network Manager...
-- Subject: Unit NetworkManager.service has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit NetworkManager.service has begun starting up.
Mar 24 16:22:58 nvidia NetworkManager[22941]: <info>  [1585092178.7916] NetworkManager (version 1.10.6) is starting... (after a restart)
Mar 24 16:22:58 nvidia NetworkManager[22941]: <info>  [1585092178.7918] Read config: /etc/NetworkManager/NetworkManager.conf (lib: 10-dns-resolved.conf, 20-connectivity-ubuntu.conf, no-mac-addr-change.conf) (etc: 10-globally-managed-devices.conf, 10-ubuntu-fan.conf, default-wifi-powersave-on.conf)
Mar 24 16:22:58 nvidia NetworkManager[22941]: <info>  [1585092178.8134] manager[0x558cba1040]: monitoring kernel firmware directory '/lib/firmware'.
Mar 24 16:22:58 nvidia NetworkManager[22941]: <info>  [1585092178.8136] monitoring ifupdown state file '/run/network/ifstate'.
Mar 24 16:22:58 nvidia dbus-daemon[3735]: [system] Activating systemd to hand-off: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.1701' (uid=0 pid=22941 comm="/usr/sbin/NetworkManager --no-daemon ")
Mar 24 16:23:09 nvidia deviceplane-agent[29381]: 2020/03/24 16:23:09 error get bundle                error=Get https://cloud.deviceplane.com:443/api/projects/prj_1XJl4OuWQqg9GYUkFlUOT6g1Awl/devices/dev_1YufJBNoPi25YvuDaXcNKGFJtYW/bundle: context deadline exceeded
Mar 24 16:23:10 nvidia /usr/lib/gdm3/gdm-x-session[7090]: (II) event2  - Logitech K400 Plus: Enabling spurious button debouncing, see https://wayland.freedesktop.org/libinput/doc/1.10.4/button_debouncing.html for details
Mar 24 16:23:17 nvidia deviceplane-agent[29381]: 2020/03/24 16:23:17 error report service state      error=Post https://cloud.deviceplane.com:443/api/projects/prj_1XJl4OuWQqg9GYUkFlUOT6g1Awl/devices/dev_1YufJBNoPi25YvuDaXcNKGFJtYW/applications/app_1XJlNYPbNtoeCWy7kn56XQWqmBQ/services/myapp/deviceservicestates: context deadline exceeded
Mar 24 16:23:23 nvidia dbus-daemon[3735]: [system] Failed to activate service 'org.freedesktop.hostname1': timed out (service_start_timeout=25000ms)
Mar 24 16:23:23 nvidia NetworkManager[22941]: <info>  [1585092203.8381] hostname: hostname: hostnamed not used as proxy creation failed with: Error calling StartServiceByName for org.freedesktop.hostname1: Timeout was reached
Mar 24 16:23:23 nvidia dbus-daemon[3735]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Mar 24 16:23:23 nvidia NetworkManager[22941]: <info>  [1585092203.8392] hostname: hostname changed from (none) to "nvidia"
Mar 24 16:23:23 nvidia NetworkManager[22941]: <info>  [1585092203.8439] dns-mgr[0x558cb90160]: init: dns=systemd-resolved, rc-manager=symlink, plugin=systemd-resolved
Mar 24 16:23:23 nvidia NetworkManager[22941]: <info>  [1585092203.8511] rfkill1: found WiFi radio killswitch (at /sys/devices/70090000.xusb/usb1/1-2/1-2.1/1-2.1:1.0/ieee80211/phy1/rfkill1) (driver rt2800usb)
Mar 24 16:23:23 nvidia NetworkManager[22941]: <info>  [1585092203.8519] manager[0x558cba1040]: rfkill: WiFi hardware radio set enabled
Mar 24 16:23:23 nvidia NetworkManager[22941]: <info>  [1585092203.8523] manager[0x558cba1040]: rfkill: WWAN hardware radio set enabled
Mar 24 16:23:23 nvidia dbus-daemon[3735]: [system] Activating systemd to hand-off: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.1701' (uid=0 pid=22941 comm="/usr/sbin/NetworkManager --no-daemon ")

This appears to show NetworkManager shutting down and starting up repeatedly and changing the state of the wlan0 device from unmanaged to unavailable.
sudo service NetworkManager status:

● NetworkManager.service - Network Manager
   Loaded: loaded (/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled)
   Active: activating (start) since Tue 2020-03-24 16:25:20 PDT; 38s ago
     Docs: man:NetworkManager(8)
 Main PID: 23677 (NetworkManager)
    Tasks: 4 (limit: 4174)
   CGroup: /system.slice/NetworkManager.service
           └─23677 /usr/sbin/NetworkManager --no-daemon

Mar 24 16:25:21 nvidia NetworkManager[23677]: <info>  [1585092321.0136] NetworkManager (version 1.10.6) is starting... (after a restart)
Mar 24 16:25:21 nvidia NetworkManager[23677]: <info>  [1585092321.0137] Read config: /etc/NetworkManager/NetworkManager.conf (lib: 10-dns-resolved.conf, 20-connectivity-ubuntu.conf, no-mac-addr-change.conf) (etc: 10-globally-managed-devices.conf, 10-ubuntu-fan.conf, default-wifi-powersave-on.conf)
Mar 24 16:25:21 nvidia NetworkManager[23677]: <info>  [1585092321.0328] manager[0x5599a03040]: monitoring kernel firmware directory '/lib/firmware'.
Mar 24 16:25:21 nvidia NetworkManager[23677]: <info>  [1585092321.0329] monitoring ifupdown state file '/run/network/ifstate'.
Mar 24 16:25:46 nvidia NetworkManager[23677]: <info>  [1585092346.0424] hostname: hostname: hostnamed not used as proxy creation failed with: Error calling StartServiceByName for org.freedesktop.hostname1: Timeout was reached
Mar 24 16:25:46 nvidia NetworkManager[23677]: <info>  [1585092346.0428] hostname: hostname changed from (none) to "nvidia"
Mar 24 16:25:46 nvidia NetworkManager[23677]: <info>  [1585092346.0462] dns-mgr[0x55999f1160]: init: dns=systemd-resolved, rc-manager=symlink, plugin=systemd-resolved
Mar 24 16:25:46 nvidia NetworkManager[23677]: <info>  [1585092346.0513] rfkill1: found WiFi radio killswitch (at /sys/devices/70090000.xusb/usb1/1-2/1-2.1/1-2.1:1.0/ieee80211/phy1/rfkill1) (driver rt2800usb)
Mar 24 16:25:46 nvidia NetworkManager[23677]: <info>  [1585092346.0521] manager[0x5599a03040]: rfkill: WiFi hardware radio set enabled
Mar 24 16:25:46 nvidia NetworkManager[23677]: <info>  [1585092346.0522] manager[0x5599a03040]: rfkill: WWAN hardware radio set enabled

Do these logs provide any additional insight into the connectivity problems that I’m experiencing?

mdegans · March 25, 2020, 6:06am

Re: range extender . If it’s a wifi to WiFi range extender, those things can have unavoidable issues based on how they work. You might try with a cell phone as a base station.

Re: logs only thing that jumps out at me at this point is that the host name is “nvidia”. Is the host name “nvidia” on all Nano’s? If so, depending on the base station, that might be what’s kicking the devices off. By chance did you clone the SD cards?

aast226 · March 26, 2020, 2:01am

I set the hostname as nvidia on all of my devices when I configure them for the first time.

mdegans · March 26, 2020, 4:29am

Try different hostnames and see what happens. Your router/base station might not like that. Mine won’t allow it, so it could be what’s causing the issue.

Generally hostnames are unique. Many people use naming schemes like characters on a particular TV show. (Tyrion might be a laptop. Drogon might be a powerful GPU box, and so on).

WayneWWW · April 8, 2020, 8:27am

Hi,

Want to check this issue. Did you mean the wired ethernet connection is off due to the wifi hotspot?

I have three Nanos connected to the same network at my house. Over the weekend, only one of them disconnected and never came back online. As far as I can tell, the other two remained online the entire time.

This behavior sounds not stable. Could you share latest status?

aast226 · April 8, 2020, 11:52pm

None of these devices are connected to the internet via a wired ethernet connection. They are all connected using WiFi.

I’m still having this issue where devices disconnect from the network and never come back online. What kind of additional info can I share to help?

WayneWWW · April 9, 2020, 3:09am

ok, so it is the wifi itself is down.

Please attach your syslog and dmesg as attachment after this issue happens.
Also, are you using pure devkit and BSP from sdkmanager?

alanz · April 9, 2020, 3:20am

@aast226

So there are three same types of wifi connected, but only one of them is disconnected.
How about the antenna, Could you switch them between the stable one and the disconnected one?
I believe the driver should be OK, let’s find something different. Antenna is important to keep a strong signal and stable connection.

aast226 · April 14, 2020, 6:56pm

@WayneWWW How can I tell if I am using pure devkit and BSP from sdkmanager?

The devices in my home have stayed online for the past week without going offline, however, I have some devices in other locations that have gone offline. I will gather the logs from these devices when possible.

aast226 · April 14, 2020, 9:11pm

@WayneWWW one of my devices in a different location went offline. Here is the output of running dmesg on that device: [595531.960820] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready[595572.7107 - Pastebin.com

Also, I have a systemd service that runs on my devices that launches wifi-connect, an open source application maintained by Balena, and when the device went offline, wifi-connect was outputting the error message: Error: Cannot find a wifi device. Here is a link to the wifi-connect repo: GitHub - balena-os/wifi-connect: Easy WiFi setup for Linux devices from your mobile phone or laptop

Looking through their code, they have the following function:

pub fn find_device(manager: &NetworkManager, interface: &Option<String>) -> Result<Device> {
    if let Some(ref interface) = *interface {
        let device = manager
            .get_device_by_interface(interface)
            .chain_err(|| ErrorKind::DeviceByInterface(interface.clone()))?;

        info!("Targeted WiFi device: {}", interface);

        if *device.device_type() != DeviceType::WiFi {
            bail!(ErrorKind::NotAWiFiDevice(interface.clone()))
        }

        if device.get_state()? == DeviceState::Unmanaged {
            bail!(ErrorKind::UnmanagedDevice(interface.clone()))
        }

        Ok(device)
    } else {
        let devices = manager.get_devices()?;

        if let Some(device) = find_wifi_managed_device(devices)? {
            info!("WiFi device: {}", device.interface());
            Ok(device)
        } else {
            bail!(ErrorKind::NoWiFiDevice)
        }
    }
}

Which appears to be the part of their code that triggered the error message.

WayneWWW · April 15, 2020, 2:22am

Hi,

@WayneWWW How can I tell if I am using pure devkit and BSP from sdkmanager?

If you are using nvidia devkit and it is installed with sdkmanager. Then yes, you are using pure devkit + BSP from sdkmanager.

However, I am not sure if pure one supports those wifi device you mentioned here. So if you install any extra driver, you need to let us know.

Panda 300Mbps Wireless N USB Adapter (currently testing)

Panda Wireless PAU06 300Mbps Wireless N USB Adapter - w/High Gain Antenna (currently testing)

Also, could you tell us what is the exact wifi module suffering this disconnection problem now? Are you able to connect the wifi by using native ubuntu tools like netowork-manager instead of launching it by systemd?

It is better to narrow down and simplify the case. We cannot help you if the cause of this issue turns out some bug in the script from balena-io.

aast226 · April 15, 2020, 6:02pm

Yes, this is what I’m doing.

I did not install any extra drivers to use these devices. I am able to initially connect to a wifi network using the native ubuntu tools, but eventually the device disconnects from wifi and never reconnects.

mdegans · April 15, 2020, 10:57pm

Have you tried separate hostnames for the devices? I know my setup will not allow for this, and disconnects such devices, but some may allow it. Have you tried without the WiFi extenders?

WayneWWW · April 16, 2020, 3:03am

Hi,

You said “only one of them disconnected and never came back online”, so which one is disconnected here? You list so many modules in this post.

I’ve experienced this problem when using each of the following WiFi dongles:

LOTEKOO 150Mbps USB WiFi Adapter

Alfa AWUS036NHA

Atheros AR9271

I am currently in the early stages of evaluating the following WiFi dongles:

Panda 300Mbps Wireless N USB Adapter (currently testing)
. * Panda Wireless PAU06 300Mbps Wireless N USB Adapter - w/High Gain Antenna (currently testing)

I’ve also tested with the Intel Dual Band Wireless-Ac 8265 and it did not solve the problem.

What I notice is even AC8265 has this problem. Is it correct?

Also, could you paste the full kernel message and syslog as attachment here when error happens?
How long would it take to disconnect?
BTW, somehow pastebin is being blocked by our office network. Could you use other space to upload your files? Thanks.

aast226 · April 20, 2020, 7:17pm

I have tried this, but still have issues where devices don’t come back online after they disconnect from the network.

Yes, this is correct.

Sorry, I’ll try to make this more clear:
Over the past few months, I’ve tested with each of the modules that I mentioned in my original post. Every single one of them encountered the problem where the device disconnected from a wireless network and never came back online.
The device at my home which failed most recently (and resulted in me posting this) was the LOTEKOO 150Mbps USB WiFi Adapter.

What command do I have to run for this?

This is highly variable, some devices I’ve previously tested with would disconnect after a few days, while others would take weeks.

I’ve attached the output from running dmesg on the most recent device that disconnected here:
dmesg_nano_wifi_fail.log (104.2 KB)

WayneWWW · April 21, 2020, 3:51am

Yes, this is correct.

Actually we have ac8265 too and we cannot see such behavior. Is it possible for you to use ac8265 to share the log? It would be easy for us to compare the log when using same module.

What command do I have to run for this?

The kernel message (dmesg) would show the status of kernel driver and the syslog (/var/log/syslog) could show the status from userspace tool like Networkmanager.

I saw there is authentication failure in your log and it keeps showing up.

[28480.314016] wlan0: authentication with 50:d4:f7:0f:ca:02 timed out
[28482.550868] wlan0: authenticate with 50:d4:f7:0f:ca:02
[28482.594316] wlan0: send auth to 50:d4:f7:0f:ca:02 (try 1/3)
[28482.698418] wlan0: send auth to 50:d4:f7:0f:ca:02 (try 2/3)
[28482.825781] wlan0: send auth to 50:d4:f7:0f:ca:02 (try 3/3)
[28482.930295] wlan0: authentication with 50:d4:f7:0f:ca:02 timed out
[28485.711501] wlan0: authenticate with 50:d4:f7:0f:ca:02
[28485.747215] wlan0: send auth to 50:d4:f7:0f:ca:02 (try 1/3)
[28485.850365] wlan0: send auth to 50:d4:f7:0f:ca:02 (try 2/3)
[28485.960887] wlan0: send auth to 50:d4:f7:0f:ca:02 (try 3/3)
[28486.061891] wlan0: authentication with 50:d4:f7:0f:ca:02 timed out
[28486.372302] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[28513.342578] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[28518.122363] wlan0: authenticate with 50:d4:f7:0f:ca:02
[28518.144920] wlan0: send auth to 50:d4:f7:0f:ca:02 (try 1/3)
[28518.153217] wlan0: authenticated
[28518.156597] wlan0: associate with 50:d4:f7:0f:ca:02 (try 1/3)
[28518.165626] wlan0: RX AssocResp from 50:d4:f7:0f:ca:02 (capab=0x431 status=0 aid=11)
[28518.171454] wlan0: associated
[28518.215183] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[28522.235705] wlan0: disassociated from 50:d4:f7:0f:ca:02 (Reason: 2)
[28523.863508] wlan0: authenticate with 50:d4:f7:0f:ca:02

The reason 2 means the previous authencation is no longer valid. Do you always use the same AP for all these modules to test?