Jetson Xavier + FPGA using PCIe, BAR assignment failure

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
1

I have a Xilinx FPGA for which I compiled and loaded a driver. But when I look into the enumeration of the device, it looks like there is a failure in assigning a BAR. This failure does not allow the driver to load correctly

The output of lspci and dmesg | grep BAR is as shown below. It can be seen that the PCIe device number for the Xilinx device fails to assign a BAR.

nvidia@jetson-0423018054808:~$ lspci
0001:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad2 (rev a1)
0001:01:00.0 SATA controller: Marvell Technology Group Ltd. Device 9171 (rev 13)
0003:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad0 (rev a1)
0003:01:00.0 Serial controller: Xilinx Corporation Device 6a90
0003:01:00.1 Serial controller: Xilinx Corporation Device 6a8f
nvidia@jetson-0423018054808:~$ dmesg | grep BAR
[    7.466510] pci 0001:00:00.0: BAR 14: assigned [mem 0x30200000-0x302fffff]
[    7.466532] pci 0001:00:00.0: BAR 13: assigned [io  0x100000-0x100fff]
[    7.466537] pci 0001:01:00.0: BAR 6: assigned [mem 0x30200000-0x3020ffff pref]
[    7.466540] pci 0001:01:00.0: BAR 5: assigned [mem 0x30210000-0x302101ff]
[    7.466616] pci 0001:01:00.0: BAR 4: assigned [io  0x100000-0x10000f]
[    7.466675] pci 0001:01:00.0: BAR 0: assigned [io  0x100010-0x100017]
[    7.466696] pci 0001:01:00.0: BAR 2: assigned [io  0x100018-0x10001f]
[    7.466721] pci 0001:01:00.0: BAR 1: assigned [io  0x100020-0x100023]
[    7.466746] pci 0001:01:00.0: BAR 3: assigned [io  0x100024-0x100027]
[    8.231142] pci 0003:00:00.0: BAR 14: no space for [mem size 0x05000000]
[    8.232455] pci 0003:00:00.0: BAR 14: failed to assign [mem size 0x05000000]
[    8.233771] pci 0003:01:00.0: BAR 0: no space for [mem size 0x02000000]
[    8.235079] pci 0003:01:00.0: BAR 0: failed to assign [mem size 0x02000000]
[    8.236376] pci 0003:01:00.1: BAR 0: no space for [mem size 0x02000000]
[    8.238696] pci 0003:01:00.1: BAR 0: failed to assign [mem size 0x02000000]
[    8.245700] pci 0003:01:00.1: BAR 1: no space for [mem size 0x00020000]
[    8.252086] pci 0003:01:00.1: BAR 1: failed to assign [mem size 0x00020000]
[    8.259275] pci 0003:01:00.0: BAR 1: no space for [mem size 0x00010000]
[    8.265921] pci 0003:01:00.0: BAR 1: failed to assign [mem size 0x00010000]
[   10.293606] xclmgmt 0003:01:00.1: map_bars: Could not map BAR #0
[   10.400940] xocl:map_bars: Failed to detect XDMA config BAR
2

I see that the BAR size is 32 MB here. Please note that if this BAR is a non-prefetchable BAR, it is possible that Tegra is not able to fit this as the available window to fit endpoint’s BAR is only 31 MB.
If you don’t have any specific requirement to keep BAR as non-prefetchable, please make it as prefetchable BAR and you should be good to go. (I’m not sure how this can be done for Xilinx FPGA endpoint, I think the documentation should have this information)

3

Thanks vidyas, I will try to find this information (making the BAR prefetchable) and update here.

4

Hi vidyas,

I am still trying to find out more information about the Xilinx driver. In the mean time I had the following questions on some basic topics:

I read the relevant sections of the Xavier_TRM, and so far I have understood that there are 2 PCIe apertures and 32 bit BARs get mapped to the the lower address range. Is that why you asked me to make it prefetchable so that, so that they can be mapped to the second, larger aperture?

Does changing the BAR size to 64 bits have the same effect?

Thanks,
Karthik

5

so that they can be mapped to the second, larger aperture?
Yes.

Does changing the BAR size to 64 bits have the same effect?
This is a different thing and to be able to map it to the larger aperture, it has to be a 64-bit BAR.

6

Hi Vidyas,

Please note that if this BAR is a non-prefetchable BAR, it is possible that Tegra is not able to fit this as the available window to fit endpoint’s BAR is only 31 MB.

Does it mean that the 32bit BAR for Xavire is up to 31MB?

7

Yes. But that was with old releases. Now with the new release, it can fit larger than 31MB 32-bit BARs (in fact, it can fit up to 3GB 32-bit BAR)

8

Vidyas

Thanks,

I would like to double check your comment.
Does Old release mean Jetpack4.1 DP?
Does New release mean Jetpack4.2 or more?

9

Yes, old release means Jetpack4.2 and 4.1 while the new release indicates later release like 4.2.1 EA.