TW686x FIFO Errors Followed by PCIe AER Fatal Error on NVIDIA Platform [ L4T 36.4 / JetPack 6.1 ]

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
, ,
1

Hi Team,
We are experiencing repeated FIFO errors from the TW686x video capture device, followed by a fatal PCIe AER error. The issue occurs while the TW686x device is actively capturing video.

Initially, the driver reports intermittent FIFO error messages and signal recovered events across multiple video channels. Subsequently, we also observe multiple unexpected p-b buffer! messages. The errors become persistent on video6, after which a fatal PCIe Transaction Layer error is reported by the PCIe Root Port.

The relevant kernel log is shown below:

[ 213.081612] tw686x 0001:01:00.0: video2: FIFO error
[ 213.231950] tw686x 0001:01:00.0: video2: FIFO error
[ 213.432400] tw686x 0001:01:00.0: video2: FIFO error
[ 216.659693] tw686x 0001:01:00.0: video3: signal recovered
[ 221.991744] tw686x 0001:01:00.0: video3: FIFO error
[ 222.262354] tw686x 0001:01:00.0: video3: FIFO error
[ 225.454571] tw686x 0001:01:00.0: video1: signal recovered
[ 227.233710] tw686x 0001:01:00.0: video1: FIFO error
[ 229.142905] tw686x 0001:01:00.0: video1: FIFO error
[ 229.363410] tw686x 0001:01:00.0: video1: FIFO error
[ 232.480452] tw686x 0001:01:00.0: video6: signal recovered
[ 232.656664] tw686x 0001:01:00.0: video3: unexpected p-b buffer!
[ 232.656680] tw686x 0001:01:00.0: video7: unexpected p-b buffer!
[ 232.661669] tw686x 0001:01:00.0: video4: unexpected p-b buffer!
[ 232.666679] tw686x 0001:01:00.0: video0: unexpected p-b buffer!
[ 232.666685] tw686x 0001:01:00.0: video5: unexpected p-b buffer!
[ 233.303564] tw686x 0001:01:00.0: video6: FIFO error
[ 233.453465] tw686x 0001:01:00.0: video6: FIFO error
[ 233.659122] tw686x 0001:01:00.0: video6: FIFO error
[ 233.814399] tw686x 0001:01:00.0: video6: FIFO error
[ 234.055032] tw686x 0001:01:00.0: video6: FIFO error
[ 234.235437] tw686x 0001:01:00.0: video6: FIFO error
[ 234.501157] tw686x 0001:01:00.0: video6: FIFO error
[ 234.631244] tw686x 0001:01:00.0: video6: FIFO error
[ 234.781569] tw686x 0001:01:00.0: video6: FIFO error
[ 234.951950] tw686x 0001:01:00.0: video6: FIFO error
[ 235.222735] tw686x 0001:01:00.0: video6: FIFO error
[ 235.418009] tw686x 0001:01:00.0: video6: FIFO error
[ 235.553309] tw686x 0001:01:00.0: video6: FIFO error
[ 235.713681] tw686x 0001:01:00.0: video6: FIFO error
[ 235.919032] tw686x 0001:01:00.0: video6: FIFO error
[ 236.049332] tw686x 0001:01:00.0: video6: FIFO error
[ 236.229742] tw686x 0001:01:00.0: video6: FIFO error
[ 236.375266] tw686x 0001:01:00.0: video6: FIFO error
[ 236.580723] tw686x 0001:01:00.0: video6: FIFO error
[ 236.781188] tw686x 0001:01:00.0: video6: FIFO error
[ 237.903982] tw686x 0001:01:00.0: video6: FIFO error
[ 239.662719] tw686x 0001:01:00.0: video6: FIFO error
[ 240.339215] tw686x 0001:01:00.0: video6: FIFO error
[ 243.095361] tw686x 0001:01:00.0: video7: signal recovered
[ 248.682991] tw686x 0001:01:00.0: video5: signal recovered
[ 248.863481] tw686x 0001:01:00.0: video5: FIFO error
[ 250.662475] tw686x 0001:01:00.0: video5: FIFO error
[ 255.202804] tw686x 0001:01:00.0: video7: signal recovered
[ 257.062039] tw686x 0001:01:00.0: video7: FIFO error
[ 259.071556] tw686x 0001:01:00.0: video7: FIFO error
[ 259.071900] pcieport 0001:00:00.0: AER: Uncorrected (Fatal) error received: 0001:00:00.0
[ 259.071920] pcieport 0001:00:00.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Transaction Layer, (Receiver ID)
[ 259.071923] pcieport 0001:00:00.0: device [10de:229e] error status/mask=00040000/00400000
[ 259.071926] pcieport 0001:00:00.0: [18] MalfTLP (First)
[ 259.071928] pcieport 0001:00:00.0: AER: TLP Header: 40000020 010075ff ff204ba0 00000000
[ 259.071941] tw686x 0001:01:00.0: AER: can't recover (no error_detected callback)
[ 259.222607] pcieport 0001:00:00.0: AER: Root Port link has been reset (0)
[ 259.222677] pcieport 0001:00:00.0: AER: device recovery failed

Observed Conditions and Reproduction Pattern

We observed that the issue is closely related to changing the camera connection between TW686x channels.

The behavior we observed is as follows:

  1. A camera is initially connected to one TW686x channel and operates normally.
  2. We physically disconnect the camera from that channel and connect it to another channel.
  3. After moving the camera to another channel, we start observing FIFO error, signal recovered, and occasionally unexpected p-b buffer! messages.
  4. However, if the camera remains connected to the same channel without being moved, the capture operates normally for an extended period without reproducing the issue.

Based on these observations, the issue appears to be reproducible when the camera is disconnected and subsequently connected to a different TW686x channel.

Could you please advise whether this behavior could be related to changes in the PCIe, DMA, buffer management, or TW686x driver implementation between JetPack 5.1.3 and JetPack 6.1?

We would also appreciate any guidance on additional diagnostics or PCIe/TW686x configuration that we can check to further isolate the root cause.