PCIe Bus Errors with JetPack 7.2.1

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
,
1

Dear Nvidia Team

With JetPack 7.2.1, we see PCIe errors after a reboot of our custom platform:

[ 33.729012] pcieport 0001:00:00.0: AER: Correctable error message received from 0001:01:00.0
[ 33.729090] pcieport 0001:01:00.0: PCIe Bus Error: severity=Correctable, type=Physical Layer, (Receiver ID)
[ 33.739830] pcieport 0001:01:00.0: device [12d8:b404] error status/mask=00000001/00002000
[ 33.749133] pcieport 0001:01:00.0: [ 0] RxErr (First)
[ 33.756982] pcieport 0001:00:00.0: AER: Correctable error message received from 0001:01:00.0
[ 33.757060] pcieport 0001:01:00.0: PCIe Bus Error: severity=Correctable, type=Physical Layer, (Receiver ID)
[ 33.767787] pcieport 0001:01:00.0: device [12d8:b404] error status/mask=00000001/00002000
[ 33.777082] pcieport 0001:01:00.0: [ 0] RxErr (First)
[ 33.784939] pcieport 0001:00:00.0: AER: Correctable error message received from 0001:01:00.0
[ 33.785017] pcieport 0001:01:00.0: PCIe Bus Error: severity=Correctable, type=Physical Layer, (Receiver ID)
[ 33.795767] pcieport 0001:01:00.0: device [12d8:b404] error status/mask=00000001/00002000
[ 33.805063] pcieport 0001:01:00.0: [ 0] RxErr (First)
[ 33.812906] pcieport 0001:00:00.0: AER: Correctable error message received from 0001:01:00.0
[ 33.812983] pcieport 0001:01:00.0: PCIe Bus Error: severity=Correctable, type=Physical Layer, (Receiver ID)
[ 33.823731] pcieport 0001:01:00.0: device [12d8:b404] error status/mask=00000001/00002000
[ 33.833029] pcieport 0001:01:00.0: [ 0] RxErr (First)
[ 33.840859] pcieport 0001:00:00.0: AER: Correctable error message received from 0001:01:00.0
[ 33.840937] pcieport 0001:01:00.0: PCIe Bus Error: severity=Correctable, type=Physical Layer, (Receiver ID)
[ 33.851683] pcieport 0001:01:00.0: device [12d8:b404] error status/mask=00000001/00002000
[ 33.860984] pcieport 0001:01:00.0: [ 0] RxErr (First)
[ 33.868775] pcieport 0001:00:00.0: AER: Correctable error message received from 0001:01:00.0
[ 33.868862] pcieport 0001:00:00.0: AER: Correctable error message received from 0001:01:00.0
[ 33.868914] pcieport 0001:00:00.0: AER: Correctable error message received from 0001:01:00.0

With JetPack 6.2.1, we have not seen such messages, therefore we do not think this is an issue of the hardware.

Please let us know how we can further debug this and solve it.

Thank you.

3

What device is in use here? Do you have the “lspci -vvv” result of this device between jp7 and jp6?

4

Hi WayneWWW

Thank you for the fast reply. Here are the lspci outputs:

lspci_jp621.txt (23.5 KB)

lspci_jp721.txt (23.8 KB)

The device is a PCIe Switch from Diodes (PI7C9X2G404SV).

5

Hi,

I compared the result here . But the result is not as your expectation.

JP6:
DevCtl: CorrErr-
DevSta: CorrErr+

JP7.2:
DevCtl: CorrErr+
DevSta: CorrErr+

It means this error is present on both jp6 and jp7. It is just the DevCtl is disabled on JP6, so it won’t report the error to kernel. But the error already happened.

The issue is also there for JP6. And Rx error indicates potential hardware signal quality issue.

You could downgrade to speed to only gen1 and see if the issue is gone first.

6

Hi WayneWWW

Thank you for the hint, we were not aware of that.

When changing the speed to gen1, we see many more error messages. We also tested it with setting the speed to gen2 and we see the error much less. However, lspci -vvv still reports DevSta: CorrErr+

Do you still think this is a hardware issue? We have not seen any failures in connected devices or the switch itself. Anything else we should test?

7

You could try to disable SSC on Orin side to check if that makes the error less.

The method is mentioned in the Jetson tuning guide as patch.

However, EMI may increase after you disable SSC.