Jetson AGX Xavier Wifi disabled after waking up from sleep (latest Jetpack 5.1.4)

[infox22oo]

A post on the Xavier NX forum seems to indicate the same issue.

Everything including Wifi and Bluetooth works well with the recommended Intel 8265NGW Wifi card until suspend.

Doesn’t matter if the Xavier enters suspend using the GUI menu or CLI command, once the Xavier goes to sleep and wakes back up it no longer reconnects to Wifi. In fact the Wifi doesn’t even look like it’s installed any more. A clean shutdown (power off) and restart with button on side of the dev kit works as expected and Wifi and Bluetooth are functional again.

It’s a new AGX Xavier dev kit with the recommended Intel 8265NGW. I believe I have the latest Jetpack the Xavier (5.1.4) and L4T says Version: 35.6.0-20240828020325

All Ubuntu updates are applied. The output of dmesg is attached.
dmesg.txt (260.7 KB)

I wonder why nobody else is reporting the issue. This is a standard dev kit, new out of box and flashed with latest available Jetpack. The only perhaps unusual circumstance is that I also installed a 2 TB Samsung NVME, per the recommendations of the Isaac ROS documentation.

Can I help with any other info ? Is there a fix ?

Thanks

[DaneLLL]

Hi,
We don’t see similar issue before. Will set up Jetpack 5.1.4/Xavier developer kit + RTL8822CE and try to reproduce the issue.

[infox22oo]

Thank you so much. Just a quick note, the Wifi is the Intel 8265NGW chip as recommended in the Isaac SDK setup documentation, not RTL. Also, there is a 2TB Samsung NVME installed as recommended in the Isaac ROS documentation, not sure if having both slots occupied makes a difference. It’s not so easy to remove the SSD because as recommended in the Isaac ROS documentation, the AGX Xavier was flashed with OS on the SSD, so if we remove the SSD the OS can’t boot.

[DaneLLL]

Hi,
We can reproduce the issue on AGX Xavier developer kit. Will check and update.

[infox22oo]

Wonderful, thanks for confirming.

By the way other than this issue we found Jetpack 5.1.4 to work very well. You may want to update the sticky post at the top of this forum to emphasize that 5.1.4 is now the latest official version.

Maybe management could consider Jetpack 6 for AGX Xavier as well, then all our devices could run the same Jetpack version.

[DaneLLL]

Hi,
We are checking it. As a quick solution, please re-load the driver after resuming from suspend like:

$ sudo rmmod rtl8822ce
$ sudo modprobe rtl8822ce

[infox22oo]

Thank you so much for checking it and for working on a temporary solution.

Keeping mind that this is for the Intel 8265NGW card that is recommended in your Isaac SDK setup documentation for AGX Xavier, we tried your suggestion with “iwlwifi” instead of rtl8822ce and it didn’t work. Maybe iwlwifi needs another step ?

After waking up from suspend, wifi always disappears. Assuming that the module we need to remove for the 8265NGW is iwlwifi, we tried this command

rmmod iwlwifi

Can’t remove iwlwifi because it says “ERROR: Module iwlwifi is in use by: iwlmvn”. But can’t remove that one either because it apparently isn’t loaded.

And then the modprobe iwlwifi doesn’t seem to actually do anything, wifi is still nonexistent.

[infox22oo]

By the way I tried using Intel AX210NGW instead of the 8265NGW. It worked using the iwlwifi backports, but connected only at 300 mb while the 8265NGW connected at 866 mb to the very same router. I switched Wifi cards several times, same result.

Also the AX210NGW with the backports driver was even worse in its suspend behavior, it did not allow suspend at all.

So hopefully the wake-from-suspend behavior for the recommended Intel 8265NGW could be fixed

Thanks so much

[WayneWWW]

Hi,

Just to clarify. What you told here.

Jetson AGX Xavier Wifi disabled after waking up from sleep (latest Jetpack 5.1.4)

Thank you so much for checking it and for working on a temporary solution. Keeping mind that this is for the Intel 8265NGW card that is recommended in your Isaac SDK setup documentation for AGX Xavier, we tried your suggestion with “iwlwifi” instead of rtl8822ce and it didn’t work. Maybe iwlwifi needs another step ? After waking up from suspend, wifi always disappears. Assuming that the module we need to remove for the 8265NGW is iwlwifi, we tried this command rmmod iwlwifi Can’t remove iw…

You had a typo there.

Your command line told you there is a driver “iwlmvm” in use but you tried to remove “iwlmvn”

So actually you didn’t try what we asked there.

[infox22oo]

Excellent catch on the typo, and I can confirm that your temporary solution works.

To help with debugging the original issue, I’m attaching:

• screenshot showing connection quality after clean powerup
• output of dmesg for a clean powerup
• output of dmesg after Xavier wakes up from suspend
  (before applying the temporary fix)

Please let me know if there is any additional data I can supply from this Xavier. We have left this setup undisturbed to be ready if you have something for us to try.

dmesg_powerup.txt (74.8 KB)
dmesg_wakefromsuspend.txt (137.6 KB)

[infox22oo]

By the way, I found that after waking up from suspend, the following is sufficient to restart the wifi connection:

carter@ubuntu:~$ sudo rmmod iwlmvm
carter@ubuntu:~$ sudo rmmod iwlwifi
carter@ubuntu:~$ sudo modprobe iwlmvm

notice that ‘modprobe iwlwifi’ is not actually needed. After a suspend and wakeup, the Wifi icon at the top right is missing. The icon then reappears as soon as both modules are removed and the ‘modprobe iwlmvm’ command is typed into a terminal, and then Wifi works normal with excellent connection quality.

I tried it twice, it’s consistent. Don’t know if this helps narrow down the issue but I thought I’d mention it. Also noteworthy is that the commands each take a while to execute, like 20 seconds or so.

[WayneWWW]

We have a fix in pcie driver. Please give it a try.

diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
index 57d0a93..0250069 100644
--- a/drivers/pci/controller/dwc/pcie-designware-host.c
+++ b/drivers/pci/controller/dwc/pcie-designware-host.c
@@ -300,6 +300,7 @@
 	struct pci_host_bridge *bridge;
 	struct resource *cfg_res;
 	int ret;
+	u32 ctrl, num_ctrls;
 
 	raw_spin_lock_init(&pci->pp.lock);
 
@@ -383,6 +384,10 @@
 		}
 
 		if (!pp->ops->msi_host_init) {
+			num_ctrls = pp->num_vectors / MAX_MSI_IRQS_PER_CTRL;
+			for (ctrl = 0; ctrl < num_ctrls; ctrl++)
+				pp->irq_mask[ctrl] = ~0;
+
 			pp->msi_irq_chip = &dw_pci_msi_bottom_irq_chip;
 
 			ret = dw_pcie_allocate_domains(pp);
@@ -580,7 +585,6 @@
 
 		/* Initialize IRQ Status array */
 		for (ctrl = 0; ctrl < num_ctrls; ctrl++) {
-			pp->irq_mask[ctrl] = ~0;
 			dw_pcie_writel_dbi(pci, PCIE_MSI_INTR0_MASK +
 					    (ctrl * MSI_REG_CTRL_BLOCK_SIZE),
 					    pp->irq_mask[ctrl]);

[infox22oo]

Thanks so much. I used your excellent guide for downloading the Kernel for Jetpack 5.1.4 / L4T35.6.0 and cross compilation toolchain:
https://docs.nvidia.com/jetson/archives/r35.6.0/DeveloperGuide/SD/Kernel/KernelCustomization.html#building-the-kernel

Then I first compiled the kernel without any mods, then patched that one file and confirmed that the kernel can be compiled with the patch in place. After compiling the kernel I followed steps 5, 6, and 7 of the above guide to place the new files in the specified folders. The new Image was way larger than the previous one, perhaps because it wasn’t automatically gzipped yet.

I didn’t want to risk bricking a Xavier that was fully configured for Isaac ROS 2.1. So to verify the bug fix I set up an identical AGX Xavier dev kit brand new out of the box, with same Intel 8265NGW and Samsung NVMe. Like all other systems I initially used SDKmanager in GUI mode for complete setup of Jetpack 5.4.1 with all options, and in SDKmanager I specified the option to flash to NVMe

Then I was looking for a way to only update the kernel. From the Jetpack/L4T documentation I thought the following should do it:

sudo ./flash.sh -k kernel jetson-agx-xavier-devkit nvme0n1p1

However, it did not actually seem to update the kernel. The bug was still there, and the beginning of dmesg showed that the kernel was built by buildbrain, not my computer.

Then I used the following flash command on this experimental setup, knowing it would wipe out everything:

sudo ./nvsdkmanager_flash.sh --storage nvme0n1p1

Indeed it wiped out everything from before, but it did use the updated kernel and it worked!

Now the first part of dmesg shows my computer name as the one that built the Kernel, and the suspend/wakeup didn’t make the Wifi disappear. So, great job on fixing the bug!

I just need a little help with the command line options for the flash script so that only the kernel is updated on fully configured AGX Xavier dev kits, where Jetpack 5.1.4 was originally flashed to NVMe using the option in SDKManager.

Thanks again.

[DaneLLL]

Hi,
Please try

1. Boot AGX Xavier developer kit
2. Backup /boot/Image
3. Overwrite it with self-built kernel image and reboot

[WayneWWW]

Hi,

Some methods to update kernel

1. First, if you don’t want to reflash the board, you could just put your new kernel image to /boot/Image on your board. Reboot and new kernel will take effect.

2. Second, if you want to use flash to upgrade kernel, put your new kernel into Linux_for_Tegra/kernel/Image and then reflash.

As the kernel image is read from rootfs directly, those “-k” parameters in flash.sh won’t directly work. Also, flash.sh cannot directly flash things to external drive (nvme/usb ssd), thus, your attempt to use that will lead to obvious failure.

[infox22oo]

Wonderful!
This worked. with Wayne’s bug fix Wifi now reconnects almost instantly after the AGX Xavier wakes up after suspend, and with your additional info I was able to upgrade another AGX Xavier as well. We have many, so this was very helpful.

[infox22oo]

Excellent, thank you. I marked your bug fix as the solution, and thanks for the additional explanation.