Kernel panic after static discharge

I use a FPGA-based PCIe-device connected to C5 port (141a0000) on Jetson AGX Orin. I observed kernel panics “Unable to handle kernel NULL pointer dereference at address…” at random moments and found out the reason is static discharges that affect the PCIe-device.

We use the same PCIe-device on x86 PC. We triggered and measured the same static discharges on the device on that PC and I confirm the PCIe-device faces the same discharges on PC and Orin. But on PC there are no panics (or oops) after discharges. RxErr correctable errors happen sometimes but the device continues working.

We have tried three different carrier boards: NVidia devkit, Osbourne and AverMedia, behavior is the same.

Is it possible to make the PCIe-device work on Jetson just like on x86 (reliable to static discharges)? The device’s driver is compiled from the same source for x86 and for Jetson so I don’t know what can we change on device side.

Could you share your ESD test conditions?

  1. Voltage level — how many kV? (e.g. 2kV, 4kV, 8kV…)
  2. Discharge method — contact or air discharge?
  3. Discharge point — where exactly on the FPGA device does the static hit? (bracket, connector, exposed pin, etc.)
  4. ESD gun model, or is this natural human-body static in a real environment?

Hello, thanks for response.

  1. I’ll tell the voltage level later.
  2. Discharge method in the tests is contact discharge, but we have observed air discharges too: Orin on antistatic mat, nobody touched it and ESD symptomas appeared (the same kernel panic).
    1. In our tests human touches the frame of the carrier board. GDN of the PCIe device is connected to GND of the carrier board and cannot be isolated.

Is this still an issue to support? Any result can be shared?

Hello. We investigated this error and found it’s reason.

  1. ESD causes unrecoverable link failure (SDES - Surprise Down Error Status).
  2. Device interrupt dereferences a pointer which is null after surprise down. As null pointer dereference occurs in device interrupt, it causes kernel panic.

Functions of pci_error_handlers struct were not implemented in the driver of our device so the driver kept running after link failure. We’ve implemented pci_error_handlers.error_detected and added pointer’s non-nullness check before dereferencing, it helped.

The topic Jetson AGX Orin AER: Uncorrected (Fatal). How to recover without reboot? describes the influence of ESD on PCIe devices on AGX Orin.