Jetson Xavier NX - Enable ASPM

Robotics & Edge Computing Jetson & Embedded Systems Jetson Xavier NX
1

Please share comments on how to enable ASPM on Jetson Xavier NX. I am using L4T BSP v32.5.1. By default, the ASPM is disabled as seen under “lspci -vvv” output.

2

I don’t know the details, but typically this would be either via a device tree edit (see your device in the device tree to explore this), or else via a kernel command line parameter. See:
https://www.kernel.org/doc/html/v4.15/admin-guide/kernel-parameters.html

Note that drivers ignore parameters they don’t care about on the kernel command line, and that not all drivers support command line changes to ASPM. I do see PCIe supports this, but it is easy enough to test.

One “gotcha” you might find is that the PCI bus is disabled if no device is seen as attached in earlier boot. If you have a slow booting device on the PCIe bus, then the test for devices may occur prior to your end point device being “alive”, and so it could miss that device. The above should work though for most devices on PCI just via adding the kernel command line option (“extlinux.conf” has key/value pair “APPEND” which could for example accept “pcie_aspm=force”).

3

@linuxdev I have tried setting pcie_aspm=force in file /boot/extlinux/extlinux.conf at the end of APPEND field. Rebooted and checked, still it stays only with ASPM disabled.

Usually, for the Jetson TX2 Dev kit, as you mentioned, we used to make device tree changes to remove the below field.

        pcie-controller@10003000 {
                pci@1,0 {
                        nvidia,num-lanes = <4>;
-                       nvidia,disable-clock-request;
                };

We remove this field in all the occurrences in the DTS files. Similarly, for the Jetson Xavier NX dev kit, I am trying to do the same thing. For that, can you confirm the PCIe controller address for M.2 Key E interface pcie@14160000 - Is this correct ?

4

To enable ASPM after kernel boot, I have to use the below command. But then, i think i won’t have L1 CLKREQ enabled by the kernel.

$ sudo setpci -s 00:00.0 0x80.B=0x42

Looks like the ASPM is disabled (set with 0x40) in the PCIe configuration space by the Xavier bootloader. Is there any way i can send argument to bootloader during L4T BSP build so that i can enable ASPM L1 state with 0x42 value. I believe the kernel during boot will be able to enable L1 CLKREQ only when ASPM is set by the bootloader. Any thoughts?

5

Does the change show up in “cat /proc/cmdline”? If not, then it just might be that it is picking a different kernel command line. If it does show up here, then the command line argument is being ignored.

I couldn’t tell you what would be a correct way to deal with this from a device tree. I do not know off the top of my head what the m.2 address is. Someone from NVIDIA would have to confirm that.

One possible thing to check is the boot arguments in the device tree. What happens is that boot content prior to reaching the Linux kernel will use the device tree, and only upon reaching the Linux kernel does the “extlinux.conf” “APPEND” get added. It is possible that if the issue is a need for earlier load of any ASPM argument, then it needs to be in the device tree.

Notice in “extlinux.conf” that the start of the “APPEND” key/value pair is “${cbootargs}”. This is inherited from earlier bootloader stages. For the most part the content here is found in the device tree’s “chosen->bootargs” entry. You could also (or alternatively) put the “pcie_aspm=force” in that node of the device tree to get earlier load.

6

Yes, the /proc/cmdline had the changes in it (pcie_aspm=force) but still the ASPM was not enabled. Even I tried to have pcie_aspm=force configured by bootloader by modifying CMDLINE in file p3668.conf.common. It didn’t help.

I started changing the device-tree nodes for the PCIe controller pcie@14160000. There is a field disable-aspm-states which is set to 0xf by default. As per Nvidia dt-bindings notes, this means disable all ASPM states (L0 and L1). So, after I changed it to 0x1 (which means enable only L1 and disable only L0), I am now able to see ASPM L1 enabled under lspci.!

$ sudo lspci -vvv | grep ASPM
lspci: Unable to load libkmod resources: error -12
                LnkCap: Port #0, Speed 8GT/s, Width x1, ASPM L1, Exit Latency L0s <1us, L1 <64us
                        ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+

$ lsmod
Module                  Size  Used by

But as you can see from lsmod output, there are no drivers loaded. I am not sure how the above device-tree change can cause drivers to not load during boot. Without the device tree change, I can see all the modules were loaded by the kernel. The dmesg did not show any errors related to fail to load driver, so I don’t think providing dmesg logs would help anyways.

7

I re-built and flashed the entire image from scratch with ASPM DTS changes in. Now I can see ASPM enabled, and also all the modules loaded fine. Not sure, it could be intermittent issue with flashing mechanism.

So, I have now the ASPM enabled. I just want to test and confirm it’s working fine or not. I would like to take few more days just to confirm on this.

8

Just for follow-up, a failure to load modules might be influenced either by what is in an initrd, or by changing the kernel itself. The device tree should not have any effect on finding modules unless for some reason the storage mechanism itself can no longer load. Glad to hear you made progress though, not sure what happened prior to flashing again.

The normal mechanism for finding modules is via the kernel searching here:
/lib/modules/$(uname -r)/kernel

The “uname -r” command will change if:

  • The base kernel version changes, or
  • CONFIG_LOCALVERSION is not set correctly prior to kernel build.
1 Like
9

ASPM works fine. Thank you @linuxdev.

10

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.