TX1 PCIe relaxed ordering requirement

Autonomous Machines Jetson & Embedded Systems Jetson TX1
1

Hi,

PCIe host controller driver has this:

/* Tegra PCIE requires relaxed ordering */
static void tegra_pcie_relax_enable(struct pci_dev *dev)
{
        pcie_capability_set_word(dev, PCI_EXP_DEVCTL, PCI_EXP_DEVCTL_RELAX_EN);
}
DECLARE_PCI_FIXUP_FINAL(PCI_ANY_ID, PCI_ANY_ID, tegra_pcie_relax_enable);

I am wondering what the “requires” actually means for devices connected to the PCIe bus.

2

This is actually required only for Tegra-20 and Tegra-30 chipsets and not for ones came after Tegra-30. We will push a change upstream to fix this.

3

Thanks, that is good to know. That comment was a bit scary.

4

Hi,
I’m trying to disable relaxed ordering because my device does not support it. Turns out that tegra_pcie_relax_enable() is never run on my TX1 when my device is added. I can clear the relaxed bit on the bridge but it remains enabled on the device, if the results of “lspci -vvvxxx” are to be trusted. Is there an easy way to disable relaxed ordering for the whole PCIe bus?

5

Have you tried removing tegra_pcie_relax_enable() API altogether from the code? Even that is also not working??

6

Yeah, first I commented the whole thing out and lspci still said RlxdOrd+. This seemed suspicious so I added some printk() statements to pci-tegra.c. The printk from tegra_pcie_relax_enable() is never seen.

Do I need to explicitly enable CONFIG_PCI_QUIRKS in the kernel config? I just noticed that pci_fixup_device() is defined as an empty function without it, which would explain the behavior.

7

This seems to be caused by building pci_tegra as a module. I moved the fixups from pci-tegra.c to quirks.c and now they are being called.

8

I did as suggested: commented out
pcie_capability_set_word(dev, PCI_EXP_DEVCTL, PCI_EXP_DEVCTL_RELAX_EN);
However RlxdOrd+ in “sudo lspci -vv” output is still being set (Note that I verified that tegra_pcie_relax_enable is being called).
Then I tried
pcie_capability_clear_word(dev, PCI_EXP_DEVCTL, PCI_EXP_DEVCTL_RELAX_EN);
Now I see RlxdOrd- on both devices: “00:01.0 PCI bridge” and “01:00.0 Memory controller”.

However, when I double check registers T_PCIE2_RP_DEVICE_CONTROL_STATUS:

sudo busybox devmem 0x01000088
0x0000280F

  • OK, PCI_EXP_DEVCTL was cleared
    but on the memory controller:
    sudo busybox devmem 0x01001088
    0x00002810
  • here PCI_EXP_DEVCTL still remains set.

I am confused: when I see “RlxdOrd-” in “sudo lspci -vv” output, is it from the register value on the endpoint or the root (TX1)?
But registers T_PCIE2_RP_DEVICE_CONTROL_STATUS do control the root, right? And here PCI_EXP_DEVCTL_RELAX_EN is still enabled.

So, what should I do to reliably disable relaxed ordering on both root and endpoint?

9

You can remove tegra_pcie_relax_enable() API from the driver code and since PCIe host controller is being built as a module, you can just copy pci-tegra.ko to target and you should be good to go.

10

Actually, pci-tegra is not built as a module. There is “CONFIG_PCI_TEGRA=y” in .config.
Also, simply removing tegra_pcie_relax_enable does not make difference.
I found that “Device Control Register” is 0x2810 by default on my endpoint,
therefore, I had to explicitly call pcie_capability_clear_word(dev, PCI_EXP_DEVCTL, PCI_EXP_DEVCTL_RELAX_EN); to clear it.

Now my question is about T_PCIE2_RP_DEVICE_CONTROL_STATUS register on TX1.
When I call pcie_capability_clear_word(dev, PCI_EXP_DEVCTL, PCI_EXP_DEVCTL_RELAX_EN);
it only resets PCI_EXP_DEVCTL_RELAX_EN on the endpoint,
but T_PCIE2_RP_DEVICE_CONTROL_STATUS register at address 0x01001088 still remains 0x00002810.

Please, explain the meaning of bit [4] - T_PCIE2_RP_DEVICE_CONTROL_STATUS_ENABLE_RELAXED_ORDERING in T_PCIE2_RP_DEVICE_CONTROL_STATUS.
If this bit is set, would it allow TX1 to send transactions with Relaxed Ordering set?
I must ensure that no such relaxed transactions can be sent in either direction
because that would violate producer-consumer model and cause data corruption.

Please, tell me how to make sure that relaxed ordering is cleared in all necessary places.

11

I’m not sure if I understood your requirement correctly.
Please remove tegra_pcie_relax_enable() completely from pci-tegra.c file for all good reasons. It is required only for Tegra20 and Tegra30 and not for any other Tegra chip (certainly not for TX1).
With that being removed, you shouldn’t be seeing ‘Enable Relaxed Ordering’ bit getting set either in root port or endpoint.
Here is the snippet from PCIe spec on what setting ‘Enable Relaxed Ordering’ would do

Enable Relaxed Ordering – If this bit is Set, the Function is
permitted to set the Relaxed Ordering bit in the Attributes field of
transactions it initiates that do not require strong write ordering
(see Section 2.2.6.4 and Section 2.4).
A Function is permitted to hardwire this bit to 0b if it never sets
the Relaxed Ordering attribute in transactions it initiates as a
Requester.
Default value of this bit is 1b.

Based on this, it would only affect the packets that this particular port (be it root port or endpoint) initiates.

12

With that being removed, you shouldn’t be seeing ‘Enable Relaxed Ordering’ bit getting set either in root port or endpoint.

But this bit is is SET by default!
You can see TX1 TRM Bit “T_PCIE2_RP_DEVICE_CONTROL_STATUS_ENABLE_RELAXED_ORDERING”
reset value: 1
The same is on the endpoint: “Device Control Register” is 0x2810 on reset.

So, this bit is set initially and software must clear this bit explicitly.
I will take care to clear it on the end point.

But I would like you to explain the exact effect of bit T_PCIE2_RP_DEVICE_CONTROL_STATUS_ENABLE_RELAXED_ORDERING.

If this bit is 1 (which is on reset),
does this mean that TX1 will set Relaxed Ordering attribute on transactions that TX1 sends?

Thank you

13

How can I get an answer about T_PCIE2_RP_DEVICE_CONTROL_STATUS_ENABLE_RELAXED_ORDERING?
Should I submit a new question?
Thank you

14

I’m checking this internally and have an update shortly.

15

Even though Tegra has Relaxed Ordering bit set, it won’t set Relaxed-Ordering bit in PCIe packets going over the bus. So, I think you don’t have to worry about it at least from Tegra point of view.

16

OK
Thank you