PCIe Root Complex DMA Test

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1

Hello,

I enabled DMA testing for PCIe RP by following the topic below
https://forums.developer.nvidia.com/t/what-is-the-actual-maximum-speed-of-jetson-agx-xavier-pcie-ethernet/212429/8 and I have some questions below:

The Jetpack version is v5.1(Jetson Linux v35.2.1)

  1. The default of data size for DMA test is 255MB(BAR0_SIZE - 1MB), and when I try to change the BAR0_SIZE to 1GB to do a larger data size for DMA testing, it fails. The largest BAR0_SIZE I’ve tried to do DMA test successfully is 512MB. Is there any way I can start DMA testing if the data size is greater than 1GB?

  2. How can I change the max payload size of PCIe?

  3. Does Jetson AGX Xavier can support the max payload size over 256 bytes?

BR,
Hermes Wu

3

Hi,

I launched a test that ran other commands to occupy the CPU usage(up to 100%) and then triggered the PCIe RP DMA transmission for testing.
However, the heavy CPU loading does not impact the DMA transmission(PCIe Gen.4 x2, all around 10Gbps ~ 11Gbps).
Do you have any idea why the heavy CPU loading has not impacted the PCIe DMA transmission?

BR,
Hermes

4

Please refer to Increasing size of BAR0 in Endpoint Mode

5

Hi @WayneWWW

Referring to the topic you posted, I should do:

  1. Modify kernel-dts/tegra194-soc/tegra194-soc-pcie.dtsi to be the same as the topic.
  2. Modify drivers/pci/dwc/pcie-tegra.c to be the same as the topic.
  3. Modify drivers/pci/endpoint/functions/pci-epf-nv-test.c to be the same as the topic but change BAR0_SIZE as per requirement (eg: SZ_1GB).

Are these steps correct?

BR,
Hermes Wu

6

Yes, correct.

7

For setting the max payload. Previously some other users using setpci to configure it.

example:

sudo setpci -s 0005:00:00.0 74.w (device capabilities register for x4)
sudo setpci -s 0005:00:00.0 78.w ( Device Control register for x4) and write value for bits 7:5 as (001b) for 256 Bytes MPS

But the actual MPS still need to check with lspci command.

And then check lspci -vvv

    Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 256 bytes, MaxReadReq 256 bytes
8

Hi @WayneWWW

I tried using the command “setpci” to configure the max payload for the PCIe root port (0005:00:00.0), but this only applied to the PCIe root port, not the endpoint. Do I also need to configure the max payload for the endpoint?

Before Configuration:

        Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
--
        Capabilities: [40] Express (v2) Downstream Port (Slot+), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag+ RBE+
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-

After configuration:

        Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
--
        Capabilities: [40] Express (v2) Downstream Port (Slot+), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag+ RBE+
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
9

Yes you need to do that.

10

Hi @WayneWWW

Do you have any idea about the issue I posted above?:

I have tried the Jetson AGX Orin platform, and the PCIe RP DMA transfer performance is almost twice more than Xavier.
If this situation has nothing to do with CPU usage or performance, what are your thoughts on what causes this result?

Best Regards,
Hermes Wu

11

Hi @WayneWWW

Is there any update?

12

Hi,

If you are using the DMA test driver developed by Nvidia. In this case, we don’t have any CPU intensive tasks, we are just doing DMA transfers and detecting DMA completion based on interrupt.

If same DMA driver is used in Xavier, then I expected same performance. Please compare PCIe link speed and width from lspci -vvv and see if they are same.