Changing Memory Regions For PCIe Devices

Hi there,

I am trying to connect an external PCIe x1 device to the Jetson AGX Xavier devkit using the PCIe x8 expansion slot. The PCIe device is recognized by the Jetson and shows up in “lspci”. The issue is that when trying to load the drivers, the BAR0 and BAR2 values are off. The driver expects 32-bit values, and reads 0x4000_0000 and 0x4000_0200 values for BAR0 and BAR2 respectively. When running “lspci -vv” the memory regions for the device are shown as 0x1f_4000_0000 and 0x1f_4000_0200 for Regions 0 and 2. My understanding is that the Regions should always have a 32-bit address, as specified by bits 1&2 of the register being 0. What I am wondering is:

  1. Why is the memory region starting at a 64-bit address when it should be fixed to 32-bits?
  2. How can I change the memory region start address so that it is addressable with 32-bits?

Thanks,
John

I won’t be able to answer, but what follows might be of interest for your case…

In the 32-bit systems the DMA was basically (by default) a physical address. Going to 64-bit also changed DMA and data transfers through the memory controller as a virtual address. Many PCI applications from 32-bit failed due to assuming a particular address. Once updated to query the host for the base address those applications once more worked. I won’t be able to give you details, but what I believe this comes down to is that you must query the host for the correct address based on going through a virtual address, not a physical address. Your device is probably still behaving as an older non-virtual 32-bit physical address.

I found a similar issue here: pcie ethernet i210 flash failed. From this I believe the driver is actually reading the bus address (0x4000_0000) rather than the system address (0x1f_4000_0000). The fix from that link seems to be to manually change the read address, rather than moving the BAR itself. Unfortunately the driver is still expecting a 32-bit value, so it won’t be able to store 0x1f_4000_0000. This is why I am trying to change the BAR to something with a 32-bit address.