Linux kernel on Jetson TX2 is somehow failing to create addresses for prefetchable pcie devices

Hey everyone,

My problem is:

I’m unable to get BAR addresing from prefetchable end-point devices behind a pcie bridge with my Jetson TX2. Can anyone help me with that?

Here is what I did until now:

1- I’m able to get addressing for a prefetchable device without a pcie bridge with my TX2. Although I cannot perform read/write operations to prefetchable devices, I do not suspect it is from the TX2 side of the equation. However, I can build a topic for this later.

2- I’m able to get addressing for non-prefetchable devices without a pcie bridge. I can also perform memory operations on them.

3- I’m unable to get addressing for a single prefetchable device behind a pcie bridge. I suspect that this is probably the main cause for my trouble.

4- I’m able to get addressing for multiple non-prefetchable devices behind a pcie bridge. I also can perform memory operations on both.

5- I’m unable to get addressing for both a prefetchable and a non-prefetchable device behind the same pcie bridge.

My deduction from my experiments:

Case 2 clears the suspicion on non-prefetchable devices. Case 4 clears the suspicion on the pcie bridge. Thus, I’m suspecting that kernel cannot use prefetchable memory behind a pcie bridge. I need it to.

Remaining things are on experiment 5
When I use lspci, I can get vendor IDs, device IDs, and device types of the both devices succesfully but their memory mapping is not done. And the files related to these devices are not appearing on /sys/bus/pci_express/devices (I guess this was expected in this condition)

Here is what I found on dmesg related to the devices:

Device 0000:03:00.0, which is another linux card with endpoint capability with 8 MB + 8 MB prefetchable memory, and a multimedia device.

[    2.496007] pci 0000:03:00.0: [104c:b005] type 00 class 0x048000
[    2.496065] pci 0000:03:00.0: reg 0x10: [mem 0x00000000-0x007fffff]
[    2.496099] pci 0000:03:00.0: reg 0x14: [mem 0x00000000-0x007fffff pref]
[    2.496268] pci 0000:03:00.0: Max Payload Size set to 128 (was 256, max 256)
[    2.496605] iommu: Adding device 0000:03:00.0 to group 59
[    2.496610] arm-smmu: forcing sodev map for 0000:03:00.0
[    2.510188] pci 0000:03:00.0: BAR 0: no space for [mem size 0x00800000]
[    2.510192] pci 0000:03:00.0: BAR 0: failed to assign [mem size 0x00800000]
[    2.510196] pci 0000:03:00.0: BAR 1: no space for [mem size 0x00800000 pref]
[    2.510200] pci 0000:03:00.0: BAR 1: failed to assign [mem size 0x00800000 pref]
[    2.510942] pci 0000:03:00.0: Signaling PME through PCIe PME interrupt

Device 0000:04:00.0, which is an FPGA with endpoint capability with 64 MB non-prefetchable memory, as an unassigned class device.

[    2.497054] pci 0000:04:00.0: [1172:e001] type 00 class 0xff0000
[    2.497114] pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x03ffffff]
[    2.497691] iommu: Adding device 0000:04:00.0 to group 60
[    2.497696] arm-smmu: forcing sodev map for 0000:04:00.0
[    2.510254] pci 0000:04:00.0: BAR 0: no space for [mem size 0x04000000]
[    2.510259] pci 0000:04:00.0: BAR 0: failed to assign [mem size 0x04000000]
[    2.510949] pci 0000:04:00.0: Signaling PME through PCIe PME interrupt

dmesg outputs pointed me out for searching memory capabilities of TX2, however I found out that it is capable of much more than what I’ve expected from it. 127MB for non-prefetchable, and much more that 16MB for prefetchable.

So, I’m stuck at that point. Anything you say can help me right now because I feel I’m out of things to do. Any suggestions are appreciated, an answer would be GREAT, and I can improve my question if you guys need any more information.

Could you please connect the same hierarchy of PCIe devices to another system (preferably to an x86 machine) and share the output of ‘sudo lspci -vvvv’?

Unfortunately, that seems impossible right now, sorry. To be more specific, this devices sit on a common card, and the pcie switch is not a pluggable device, but a chip on that card. I hope that does not block us from getting the problem fixed, though. Looking forward for your response.

TX2 didn’t have this issue with any of the off-the-shelf PCIe devices with a mixture of prefetchable/non-prefetchable BARs. So, it is very unlikely that it is an issue with TX2.
I’m wondering the correctness of the default config space values of these cards hence I was asking if it is possible to connect them to an x86 host and see if they get enumerated fine.
Have these cards been tried out with any other host in the past?
worst case, can we have the output of both ‘sudo lspci -vv’ and ‘sudo lspci -xxx’ ?