PCIe Max Payload Size

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin NX
1

Dear Jetson Hardware Experts,

I am using the latest version of JetPack with a Jetson Orin NX 8GB and a custom carrier board.
This board includes a PCIe port, and I am able to communicate with another card.

However, I am experiencing bandwidth issues. The problem comes from the Max Payload Size.
By default, the value is set to 128, while the card supports up to 256.

After some research and several attempts, I modified the file /boot/extlinux/extlinux.conf by adding:

+ pci=pcie_bus_perf

startup_pcie_perf.txt (61.8 KB)

startup_pcie_safe.txt (55.9 KB)

This addition solves the issue, as shown in the dmesg output:

[    7.238541] pci 0001:00:00.0: Max Payload Size set to  256/ 256 (was  128), Max Read Rq  256  
[    7.238656] pci 0001:01:00.0: Max Payload Size set to  256/ 512 (was  128), Max Read Rq  256

With this change, I recover the expected bandwidth, and data transfers work correctly.
However, I lose the Ethernet port, it no longer functions.

Is it possible to modify only the Max Payload Size assigned to my card without affecting other components ?
And where exactly should I make these modifications ?

Could you please provide further support on this ?

Thanks and best regards.

2

Do you have full dmesg of current situation?

I am just curious what is going on with ethernet port.

3

it is already in the message.

4

I saw the realtek driver got crashed. Which Jetpack release are you using?

I think one way to prevent this issue here is to adjust only the MPS of PCIe C1 here instead of a overall system configuration.

5

I’m using JetPack 6.2.1.

Sure, but do you know how to do it ?

7

Some commands like below one should work

sudo setpci -s 0005:00:00.0 74.w (device capabilities register for x4)
sudo setpci -s 0005:00:00.0 78.w ( Device Control register for x4) and write value for bits 7:5 as (001b) for 256 Bytes MPS

Please be aware that you need to change 0005 to the device you are using.

8

I’ve already found similar commands in other forum topics.

Register 74 seems to be read-only:

sudo setpci -s 0004:00:00.0 74.w
    8001
sudo setpci -s 0004:00:00.0 74.w=8021
sudo setpci -s 0004:00:00.0 74.w
    8001

sudo setpci -s 0004:01:00.0 74.w
    0000
sudo setpci -s 0004:01:00.0 74.w=0020
sudo setpci -s 0004:01:00.0 74.w
    0000

The register that allows modifying the MaxPayload value displayed by lspci is the following:

sudo setpci -s 0004:01:00.0 68.w
    2810
sudo setpci -s 0004:01:00.0 68.w=2830
sudo setpci -s 0004:01:00.0 68.w
    2830

However, after making these modifications, I can no longer communicate with my board.
Do I need to reload something ?

Anyway, I noticed that these changes are cleared after shutting down and rebooting.
Is there a way to set the Max Payload Size at startup?
Do I need to modify the kernel or look for a specific file in the DTB files ?

9

Does dmesg give out anything after you adjust the MPS here?

Also, do you still see your device in lspci after you adjusted the MPS?

10

dmesg doesn’t show anything, and I can still see my device with lspci.

I managed to modify the MaxPayload value while keeping everything working. I used the following commands :

sudo setpci -s 0004:00:00.0 78.w  
    281f  
sudo setpci -s 0004:00:00.0 78.w=283f  
sudo setpci -s 0004:00:00.0 78.w  
    283f  
sudo setpci -s 0004:01:00.0 68.w  
    2810  
sudo setpci -s 0004:01:00.0 68.w=2830  
sudo setpci -s 0004:01:00.0 68.w  
    2830

After that, I can load my module into the kernel (insmod) and communicate with my board with a sufficient data rate.

Now I just need to automate the configuration using setpci.

Thank you very much for your help.

For now, this seems to be the most effective method. If you know of a cleaner or more elegant way to do it, I’d be glad to hear it.