How to change AGX Orin PCIe RC flow control credit in "VC0 posted receive queue" registers

[wolf318]

Hi,
We have a FPGA frame grabber connected to AGX Orin PCIe C4 port, configured as Gen3 x4. Supposed data throughput should reach 30+Gbps. But it’s found that even 4K60 (~ 8Gbps) cannot properly get the full frame. First few hundred lines seem to be OK, but after that FIFO overflow. Chipscope in FPGA found that posed data flow control credit from AGX run out.

The initial value of fc_pd is 320, it matches Orin TRM register “PCIE_X4_RC_PF0_PORT_LOGIC_VC0_P_RX_Q_CTRL_OFF_0” setting.
It also mentioned in this post #210956.

We can read this also from AGX terminal. But the value doesn’t seem to be changeable. In TRM the note says “The access attributes of this field are as follows: - Wire: No access. - Dbi: R (sticky)”. Does it mean this is a HW fixed value? Is there any way to increase the number of credit, or the TLP rx buffer? Or a method to dump out the data to main memory faster?

Note that for C4 port IOMMU is disabled for our driver to work, but “dma-coherent” property is kept when building the kernel.
It’s mentioned in this post #283398.

Please help, appreciated.

[carolyuu]

Hi,

If you are designing a custom base board, then it means some adaptation configurations are needed.
Otherwise, your board may not work fine.

For Orin AGX series, you could refer to below document
https://docs.nvidia.com/jetson/archives/r36.3/DeveloperGuide/HR/JetsonModuleAdaptationAndBringUp/JetsonAgxOrinSeries.html?highlight=universal%20serial%20bus#jetson-agx-orin-platform-adaptation-and-bring-up
(please be aware that above link is for rel-36.3/jetpack6.0)

This document includes below configuration

1. pinmux change & GPIO configuration
2. EEPROM change as most custom boards do not have an EEPROM on it.
3. Kernel porting
4. PCIe configuration
5. USB configuration
6. MGBE configuration
7. RGMII configuration

Thanks!

[wolf318]

Hi carolyuu,
Thanks for the info. Yes we’re doing custom base board, using P3701-0008 SOM. It boots up and function normal so assume other configurations for adaptation were properly set. Just the PCIe performance in question.

We’re using Jetpack5 so I reference to r35.4.1 document,
https://docs.nvidia.com/jetson/archives/r35.4.1/DeveloperGuide/text/HR/JetsonModuleAdaptationAndBringUp.html

However, neither document mention buffer size is configurable in the “PCIe controller configuration” section.

What else shall we look at? Thanks.

[wolf318]

Update:
from Nvidia document “nvidia,tegra194-pcie.txt”, also found this device tree entry,
"Optional properties:

• nvidia,update-fc-fixup: This is a boolean property and needs to be present to
  improve performance when a platform is designed in such a way that it
  satisfies at least one of the following conditions thereby enabling root
  port to exchange optimum number of FC (Flow Control) credits with
  downstream devices
  1. If C0/C4/C5 run at x1/x2 link widths (irrespective of speed and MPS)
  2. If C0/C1/C2/C3/C4/C5 operate at their respective max link widths and
     a) speed is Gen-2 and MPS is 256B
     b) speed is >= Gen-3 with any MPS
     "
     I suspected this may change the fc default value or allow change the register. Built it in kernel but didn’t seem to help anything.