My problem is:
I’m unable to get BAR addresing from prefetchable end-point devices behind a pcie bridge with my Jetson TX2. Can anyone help me with that?
Here is what I did until now:
1- I’m able to get addressing for a prefetchable device without a pcie bridge with my TX2. Although I cannot perform read/write operations to prefetchable devices, I do not suspect it is from the TX2 side of the equation. However, I can build a topic for this later.
2- I’m able to get addressing for non-prefetchable devices without a pcie bridge. I can also perform memory operations on them.
3- I’m unable to get addressing for a single prefetchable device behind a pcie bridge. I suspect that this is probably the main cause for my trouble.
4- I’m able to get addressing for multiple non-prefetchable devices behind a pcie bridge. I also can perform memory operations on both.
5- I’m unable to get addressing for both a prefetchable and a non-prefetchable device behind the same pcie bridge.
My deduction from my experiments:
Case 2 clears the suspicion on non-prefetchable devices. Case 4 clears the suspicion on the pcie bridge. Thus, I’m suspecting that kernel cannot use prefetchable memory behind a pcie bridge. I need it to.
Remaining things are on experiment 5
When I use lspci, I can get vendor IDs, device IDs, and device types of the both devices succesfully but their memory mapping is not done. And the files related to these devices are not appearing on /sys/bus/pci_express/devices (I guess this was expected in this condition)
Here is what I found on dmesg related to the devices:
Device 0000:03:00.0, which is another linux card with endpoint capability with 8 MB + 8 MB prefetchable memory, and a multimedia device.
[ 2.496007] pci 0000:03:00.0: [104c:b005] type 00 class 0x048000 [ 2.496065] pci 0000:03:00.0: reg 0x10: [mem 0x00000000-0x007fffff] [ 2.496099] pci 0000:03:00.0: reg 0x14: [mem 0x00000000-0x007fffff pref] [ 2.496268] pci 0000:03:00.0: Max Payload Size set to 128 (was 256, max 256) [ 2.496605] iommu: Adding device 0000:03:00.0 to group 59 [ 2.496610] arm-smmu: forcing sodev map for 0000:03:00.0 [ 2.510188] pci 0000:03:00.0: BAR 0: no space for [mem size 0x00800000] [ 2.510192] pci 0000:03:00.0: BAR 0: failed to assign [mem size 0x00800000] [ 2.510196] pci 0000:03:00.0: BAR 1: no space for [mem size 0x00800000 pref] [ 2.510200] pci 0000:03:00.0: BAR 1: failed to assign [mem size 0x00800000 pref] [ 2.510942] pci 0000:03:00.0: Signaling PME through PCIe PME interrupt
Device 0000:04:00.0, which is an FPGA with endpoint capability with 64 MB non-prefetchable memory, as an unassigned class device.
[ 2.497054] pci 0000:04:00.0: [1172:e001] type 00 class 0xff0000 [ 2.497114] pci 0000:04:00.0: reg 0x10: [mem 0x00000000-0x03ffffff] [ 2.497691] iommu: Adding device 0000:04:00.0 to group 60 [ 2.497696] arm-smmu: forcing sodev map for 0000:04:00.0 [ 2.510254] pci 0000:04:00.0: BAR 0: no space for [mem size 0x04000000] [ 2.510259] pci 0000:04:00.0: BAR 0: failed to assign [mem size 0x04000000] [ 2.510949] pci 0000:04:00.0: Signaling PME through PCIe PME interrupt
dmesg outputs pointed me out for searching memory capabilities of TX2, however I found out that it is capable of much more than what I’ve expected from it. 127MB for non-prefetchable, and much more that 16MB for prefetchable.
So, I’m stuck at that point. Anything you say can help me right now because I feel I’m out of things to do. Any suggestions are appreciated, an answer would be GREAT, and I can improve my question if you guys need any more information.