PCIe bus rescanning after FPGA programming on Jetson Xavier

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
1

Hi,

I have a Jetson Xavier kit with a Hitech Global K800 PCIe development card. After I reprogram the UltraScale, the Xavier does not reliably detect the card’s PCIe link.

These are the steps that I take.

First, I remove the device entry in Linux using:

echo 1 | sudo tee /sys/bus/pci/devices/0005\:01\:00.0/remove

"dmesg" reports:
[255672.488706] iommu: Removing device 0005:01:00.0 from group 64

Then I use Vivado to reprogram the FPGA:

"dmesg" reports:
[255683.330101] pcieport 0005:00:00.0: AER: Corrected error received: id=0000
[255683.330129] pcieport 0005:00:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0000(Receiver ID)
[255683.330403] pcieport 0005:00:00.0:   device [10de:1ad0] error status/mask=00000001/0000e000
[255683.330629] pcieport 0005:00:00.0:    [ 0] Receiver Error         (First)
[255683.380511] pcieport 0005:00:00.0: AER: Uncorrected (Fatal) error received: id=0000
[255683.380537] pcieport 0005:00:00.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Transaction Layer, id=0000(Receiver ID)
[255683.380755] pcieport 0005:00:00.0:   device [10de:1ad0] error status/mask=00000020/00400000
[255683.380904] pcieport 0005:00:00.0:    [ 5] Surprise Down Error    (First)
[255684.410036] pcieport 0005:00:00.0: Root Port link has been reset
[255684.410056] pcieport 0005:00:00.0: broadcast mmio_enabled message
[255684.410064] pcieport 0005:00:00.0: broadcast resume message
[255684.410071] pcieport 0005:00:00.0: AER: Device recovery successful

However, when I rescan the PCI bus (using sysfs), no new entries appear. Additionally, the link status LEDs on the K800 indicate no link is present.

I have a few K800 boards (all configured the same) and there seems to be a difference in ability to be detected after FPGA reprogramming. One card is detected successfully hundreds of times while another card is never detected after FPGA reprogramming.

Is this expected behavior for this PCIe slot?

Is there a reliable way to force a PCIe rescan (or reset the root port link)?

TIA,
Jeremy

3

Most FPGAs do not boot in time for the initial PCIe bus scan. The Xavier itself does not have PCIe powered after boot if no PCIe was detected, and does not support hot plug for late boot. The following is from the TX1, and it might explain a few things:
https://devtalk.nvidia.com/default/topic/997845/jetson-tx1/force-rescan-of-pcie-bus-/post/5102006/#5102006

I don’t know if this is the same for Xavier or not so far as actual patch, but adjustments need to be made if the echo is going to work to rescan. This URL should be more specific to Xavier:
https://devtalk.nvidia.com/default/topic/1055962/jetson-agx-xavier/pcie-device-no-link/post/5354479/#5354479

Someone else may know more about late PCIe enumeration on Xavier.

NOTE: Conversely, if you fully boot the FPGA before applying power to the Jetson, then this too should work.