Gk20a and Jetson Nano crash

WayneWWW · September 8, 2020, 9:01am

Ok so there was a long backstory. But you don’t have new jetson nano at that time, right?

Please follow below steps and try to reproduce issue again…

Could you directly put your sdcards to the A02 and B01 boards, run the sdkmanager to flash your card and then run the deepstream application again and see if you can see the errors other than GPU error?

And no ssd is connected during this test.

borelli.g92 · September 8, 2020, 10:05am

Hi @WayneWWW,

ok I will follow those steps, and let you know ASAP.

Thanks!

borelli.g92 · September 9, 2020, 2:00pm

Hi @WayneWWW,

I have flashed a new SD card and reinstalled Deepstream.
I am now running Test3 on my RTSP source. I have had no crash so far. I have been running the inference for only 5 hours, thus I need to wait until tomorrow at least.

However, I forgot to mention one thing. I have edited a couple of line of test3 for the Jetson Nano struggles to process 640x440 images at 15fps. I have followed a suggestion here on the forum to use a fakesink and adapt the transform to Jetson Nano capability.

/* Finally render the osd output */ 
#ifdef PLATFORM_TEGRA
  //transform = gst_element_factory_make ("nvegltransform", "nvegl-transform");
  transform = gst_element_factory_make ("queue", "queue");
#endif
  //sink = gst_element_factory_make ("nveglglessink", "nvvideo-renderer");
//  sink = gst_element_factory_make ("nvoverlaysink", "nvvideo-renderer");
  sink = gst_element_factory_make ("fakesink", "nvvideo-renderer");

If I do not keep those changings, the Jetson Nano skips a lot of frames.

In any case, even with this new installation I am having continuosly the PCIe error in the serial log file:

[ 3694.486136] nvmap_alloc_handle: PID 25876: deepstream-test: WARNING: All NvMap Allocations must have a tag to identify the subsystem allocating memory.Please pass the tag to the API call NvRmMemHanldeAllocAttr() or relevant. 

[ 3829.505138] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 3829.515495] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 3829.525101] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 4301.401428] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 4301.411648] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 4301.420127] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 4699.923482] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 4699.933845] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 4699.942254] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 4704.380219] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 4704.390432] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 4704.398903] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 4984.462448] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 4984.472733] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 4984.481219] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 5239.496408] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 5239.506646] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 5239.515151] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 5244.856556] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 5244.866894] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 5244.876513] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 5383.112661] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 5383.125853] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 5383.135285] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 5555.630474] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 5555.640725] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 5555.649163] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 5582.934188] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 5582.945952] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 5582.955437] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 5986.752396] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 5986.762661] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 5986.771138] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 6001.066925] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 6001.077253] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 6001.085679] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 6385.286333] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 6385.296596] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 6385.305052] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 6518.078827] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 6518.090378] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 6518.098772] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 6827.179587] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 6827.189865] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 6827.198402] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 6845.122111] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 6845.133746] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 6845.142752] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7136.454764] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7136.465071] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7136.473680] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7315.957757] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7315.968028] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7315.976656] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7428.406574] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7428.416804] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7428.425196] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7594.376226] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7594.386526] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7594.395016] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7742.343007] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7742.353328] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7742.361910] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7805.350766] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7805.361027] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7805.369429] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7895.531107] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7895.541476] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7895.549871] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7934.418950] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7934.430483] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7934.438882] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 8143.524417] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 8143.534724] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 8143.543170] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 8740.552607] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 8740.562898] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 8740.571294] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 8747.308790] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 8747.319122] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 8747.327563] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 8983.357800] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 8983.368024] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 8983.376450] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9047.552739] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9047.564234] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9047.572712] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9083.811655] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9083.821941] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9083.830341] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9247.128331] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9247.140172] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9247.148636] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9348.844755] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9348.855164] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9348.863602] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9518.018461] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9518.028704] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9518.037089] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9599.143523] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9599.153844] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9599.162274] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9691.924645] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9691.936598] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9691.945096] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9825.917152] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9825.929158] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9825.938318] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9965.968999] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9965.979244] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9965.987681] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[10448.900720] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[10448.912456] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[10448.920869] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[10467.754846] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[10467.766459] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[10467.775570] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[10627.103412] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[10627.113638] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[10627.122112] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11032.099079] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11032.109421] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11032.117898] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11156.423722] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11156.434029] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11156.442409] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11432.059473] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11432.069908] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11432.078431] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11536.615924] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11536.626178] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11536.634563] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11663.731535] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11663.741866] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11663.750301] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11704.676000] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11704.686422] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11704.694817] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11715.507221] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11715.517517] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11715.525983] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11954.196226] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11954.206892] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11954.215543] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[12295.010337] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[12295.020945] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[12295.029448] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[12296.191527] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[12296.201792] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[12296.210250] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[12309.818186] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[12309.828488] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[12309.836925] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[12466.580244] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[12466.591948] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[12466.600777] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[12867.862794] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[12867.873095] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[12867.881513] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[12883.473557] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[12883.483868] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[12883.492307] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13233.060192] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13233.070480] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13233.078868] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13274.358051] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13274.369962] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13274.378423] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13314.398364] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13314.408633] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13314.417060] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13407.152802] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13407.163019] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13407.171431] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13803.947849] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13803.958431] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13803.966809] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13853.501711] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13853.512021] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13853.520503] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13863.916831] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13863.928419] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13863.936812] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[14179.851081] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[14179.861348] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[14179.870020] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[14195.625717] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[14195.635966] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[14195.644347] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[14567.343598] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[14567.353881] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[14567.362281] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[14611.627146] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[14611.637447] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[14611.645894] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[14640.574484] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[14640.584770] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[14640.593251] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[14951.978855] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[14951.990443] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[14951.998825] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[15136.815257] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[15136.825543] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[15136.833907] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[15220.585809] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[15220.596070] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[15220.604537] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[15525.006691] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[15525.016964] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[15525.025398] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[15658.621418] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[15658.631678] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[15658.640067] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[15796.143257] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[15796.153480] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[15796.163103] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16124.444871] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16124.455166] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16124.463596] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16261.341463] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16261.351939] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16261.360352] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16321.996622] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16322.007119] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16322.015519] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16511.657290] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16511.667621] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16511.676085] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16738.561125] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16738.571355] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16738.579798] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16748.539225] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16748.549443] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16748.557867] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16801.108411] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16801.120047] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16801.128428] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16870.459330] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16870.469611] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16870.478340] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16922.403772] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16922.414086] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16922.422538] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16995.767352] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16995.777701] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16995.786165] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17241.815603] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17241.827195] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17241.835591] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17262.487767] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17262.498005] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17262.506392] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17313.452181] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17313.462768] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17313.471133] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17347.509027] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17347.519619] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17347.528227] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17465.323200] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17465.333414] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17465.341794] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17676.517559] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17676.527797] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17676.536180] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17705.886292] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17705.896525] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17705.904911] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17763.490621] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17763.500873] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17763.509488] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[18008.554584] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[18008.566358] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[18008.574749] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

Thanks again for your support!!

WayneWWW · September 10, 2020, 2:33am

Hi borelli.g92,

Since you have 2 jetson nanos, please clarify

Does pcie issue happen on both devices? or only one of them?
Will you see pcie error if you run other DS sample that does not transfer data through ethernet?

borelli.g92 · September 10, 2020, 8:40pm

Hi @WayneWWW,

I start with answering your questions:

No, the PCIe error happens only with the B01 model. The new one that I have bought. It happens both with the new SD card and the system running from USB SSD.
Yes, it also happens with no application running. The serial monitor logs that error even with the board in idel condition. That does not happen with the model A02.

Update on the reboot test. I followed your suggestion and I have tested the B01 board with a virgin installation running the test3 (with modified sink). The system rebooted after 13 hours:

[ 2481.298310] nvmap_alloc_handle: PID 24629: deepstream-test: WARNING: All NvMap Allocations must have a tag to identify the subsystem allocating memory.Please pass the tag to the API call NvRmMemHanldeAllocAttr() or relevant. 

[ 2547.142540] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 2547.153127] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 2547.162748] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 3069.310332] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 3069.320567] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 3069.328951] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 3204.620069] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 3204.630348] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 3204.638751] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 3237.325623] nvmap_alloc_handle: PID 25504: deepstream-test: WARNING: All NvMap Allocations must have a tag to identify the subsystem allocating memory.Please pass the tag to the API call NvRmMemHanldeAllocAttr() or relevant. 

[ 3544.895179] nvmap_alloc_handle: PID 25762: deepstream-test: WARNING: All NvMap Allocations must have a tag to identify the subsystem allocating memory.Please pass the tag to the API call NvRmMemHanldeAllocAttr() or relevant. 

[ 3694.486136] nvmap_alloc_handle: PID 25876: deepstream-test: WARNING: All NvMap Allocations must have a tag to identify the subsystem allocating memory.Please pass the tag to the API call NvRmMemHanldeAllocAttr() or relevant. 

[ 3829.505138] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 3829.515495] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 3829.525101] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 4301.401428] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 4301.411648] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 4301.420127] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 4699.923482] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 4699.933845] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 4699.942254] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 4704.380219] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 4704.390432] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 4704.398903] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 4984.462448] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 4984.472733] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 4984.481219] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 5239.496408] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 5239.506646] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 5239.515151] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 5244.856556] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 5244.866894] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 5244.876513] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 5383.112661] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 5383.125853] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 5383.135285] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 5555.630474] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 5555.640725] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 5555.649163] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 5582.934188] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 5582.945952] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 5582.955437] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 5986.752396] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 5986.762661] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 5986.771138] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 6001.066925] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 6001.077253] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 6001.085679] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 6385.286333] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 6385.296596] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 6385.305052] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 6518.078827] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 6518.090378] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 6518.098772] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 6827.179587] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 6827.189865] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 6827.198402] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 6845.122111] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 6845.133746] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 6845.142752] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7136.454764] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7136.465071] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7136.473680] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7315.957757] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7315.968028] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7315.976656] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7428.406574] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7428.416804] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7428.425196] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7594.376226] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7594.386526] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7594.395016] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7742.343007] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7742.353328] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7742.361910] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7805.350766] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7805.361027] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7805.369429] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7895.531107] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7895.541476] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7895.549871] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7934.418950] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7934.430483] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7934.438882] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 8143.524417] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 8143.534724] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 8143.543170] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 8740.552607] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 8740.562898] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 8740.571294] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 8747.308790] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 8747.319122] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 8747.327563] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 8983.357800] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 8983.368024] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 8983.376450] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9047.552739] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9047.564234] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9047.572712] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9083.811655] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9083.821941] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9083.830341] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9247.128331] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9247.140172] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9247.148636] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9348.844755] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9348.855164] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9348.863602] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9518.018461] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9518.028704] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9518.037089] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9599.143523] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9599.153844] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9599.162274] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9691.924645] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9691.936598] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9691.945096] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9825.917152] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9825.929158] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9825.938318] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9965.968999] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9965.979244] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9965.987681] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[10448.900720] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[10448.912456] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[10448.920869] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[10467.754846] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[10467.766459] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[10467.775570] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[10627.103412] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[10627.113638] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[10627.122112] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11032.099079] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11032.109421] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11032.117898] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11156.423722] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11156.434029] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11156.442409] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11432.059473] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11432.069908] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11432.078431] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11536.615924] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11536.626178] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11536.634563] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11663.731535] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11663.741866] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11663.750301] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11704.676000] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11704.686422] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11704.694817] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11715.507221] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11715.517517] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11715.525983] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11954.196226] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11954.206892] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11954.215543] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[12295.010337] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[12295.020945] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[12295.029448] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[12296.191527] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[12296.201792] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[12296.210250] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[12309.818186] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[12309.828488] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[12309.836925] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[12466.580244] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[12466.591948] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[12466.600777] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[12867.862794] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[12867.873095] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[12867.881513] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[12883.473557] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[12883.483868] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[12883.492307] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13233.060192] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13233.070480] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13233.078868] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13274.358051] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13274.369962] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13274.378423] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13314.398364] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13314.408633] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13314.417060] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13407.152802] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13407.163019] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13407.171431] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13803.947849] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13803.958431] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13803.966809] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13853.501711] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13853.512021] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13853.520503] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13863.916831] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13863.928419] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13863.936812] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[14179.851081] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[14179.861348] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[14179.870020] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[14195.625717] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[14195.635966] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[14195.644347] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[14567.343598] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[14567.353881] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[14567.362281] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[14611.627146] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[14611.637447] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[14611.645894] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[14640.574484] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[14640.584770] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[14640.593251] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[14951.978855] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[14951.990443] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[14951.998825] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[15136.815257] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[15136.825543] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[15136.833907] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[15220.585809] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[15220.596070] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[15220.604537] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[15525.006691] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[15525.016964] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[15525.025398] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[15658.621418] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[15658.631678] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[15658.640067] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[15796.143257] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[15796.153480] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[15796.163103] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16124.444871] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16124.455166] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16124.463596] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16261.341463] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16261.351939] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16261.360352] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16321.996622] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16322.007119] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16322.015519] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16511.657290] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16511.667621] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16511.676085] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16738.561125] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16738.571355] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16738.579798] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16748.539225] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16748.549443] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16748.557867] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16801.108411] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16801.120047] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16801.128428] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16870.459330] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16870.469611] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16870.478340] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16922.403772] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16922.414086] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16922.422538] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16995.767352] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16995.777701] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16995.786165] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17241.815603] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17241.827195] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17241.835591] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17262.487767] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17262.498005] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17262.506392] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17313.452181] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17313.462768] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17313.471133] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17347.509027] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17347.519619] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17347.528227] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17465.323200] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17465.333414] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17465.341794] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17676.517559] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17676.527797] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17676.536180] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17705.886292] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17705.896525] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17705.904911] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17763.490621] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17763.500873] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17763.509488] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[18008.554584] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[18008.566358] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[18008.574749] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[18548.707962] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[18548.719432] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[18548.727830] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[18598.321547] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[18598.331846] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[18598.340362] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[18760.514887] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[18760.525142] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[18760.533538] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[18844.325806] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[18844.337380] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[18844.345822] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[19225.396102] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[19225.406374] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[19225.414769] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[19291.971921] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[19291.982148] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[19291.990532] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[19331.334222] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[19331.344500] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[19331.352907] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[19705.993990] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[19706.004215] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[19706.012615] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[19805.565941] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[19805.576166] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[19805.584574] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[19924.921335] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[19924.931569] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[19924.939981] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20089.184937] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20089.195346] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20089.203787] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20110.701762] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20110.711981] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20110.720371] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20525.956585] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20525.966967] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20525.975451] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20733.711903] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20733.722130] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20733.730540] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20753.976935] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20753.987403] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20753.995802] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20864.658986] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20864.669761] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20864.678353] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20868.316309] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20868.326591] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20868.334988] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20897.298449] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20897.308680] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20897.317065] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[21150.509562] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[21150.519831] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[21150.528261] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[21182.967692] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[21182.977939] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[21182.986345] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[21220.470230] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[21220.480448] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[21220.488849] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[21517.257859] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[21517.268060] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[21517.276438] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[21797.717797] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[21797.728360] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[21797.736827] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[21998.368518] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[21998.378742] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[21998.387130] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22184.790167] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22184.800753] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22184.809158] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22212.721739] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22212.732101] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22212.740540] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22321.441245] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22321.451479] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22321.459877] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22477.596700] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22477.608238] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22477.616791] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22571.813852] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22571.824117] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22571.832520] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23060.573037] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23060.583348] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23060.591796] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23110.707653] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23110.717914] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23110.726314] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23133.214266] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23133.224531] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23133.232985] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23209.655648] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23209.666031] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23209.674445] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23328.692313] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23328.702574] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23328.710998] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23467.437180] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23467.447402] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23467.455977] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23599.543630] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23599.553943] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23599.562368] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23774.274812] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23774.285118] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23774.293697] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[24011.380134] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[24011.390375] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[24011.398796] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[24499.914669] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[24499.924904] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[24499.933295] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[24727.044147] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[24727.054391] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[24727.062782] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25275.608911] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25275.619377] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25275.627832] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25292.497183] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25292.507452] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25292.515857] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25582.014403] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25582.024623] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25582.033024] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25603.346181] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25603.356471] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25603.364858] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25630.307934] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25630.318160] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25630.326557] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25756.848711] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25756.858977] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25756.867364] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25859.029691] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25859.040116] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25859.048517] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25883.022161] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25883.032424] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25883.040975] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[26160.635076] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[26160.645723] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[26160.654113] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[26688.427027] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[26688.437278] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[26688.445663] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[26908.111055] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[26908.121283] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[26908.129681] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[27022.564938] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[27022.575231] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[27022.583664] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[27302.925014] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[27302.935275] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[27302.943705] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[27386.357335] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[27386.367584] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[27386.375972] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[27703.346878] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[27703.357166] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[27703.365557] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[27797.052329] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[27797.062570] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[27797.070949] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[27832.301798] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[27832.312025] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[27832.320432] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[28416.734524] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[28416.744744] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[28416.753156] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[28498.299572] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[28498.309882] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[28498.318384] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[28856.161171] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[28856.171631] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[28856.180139] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[28913.888785] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[28913.899068] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[28913.907516] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[29541.895606] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[29541.905815] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[29541.914257] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[29890.008502] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[29890.018766] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[29890.027196] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[29890.675165] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[29890.685393] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[29890.693774] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[30330.044333] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[30330.054919] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[30330.063327] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[30423.596577] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[30423.606822] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[30423.615280] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[30428.548528] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[30428.558797] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[30428.567235] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[30450.350553] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[30450.361212] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[30450.369698] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[31214.340331] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[31214.350566] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[31214.359027] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[31666.237590] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[31666.249464] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[31666.257996] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[31870.334638] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[31870.345009] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[31870.353416] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[31933.213091] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[31933.223346] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[31933.231734] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[32004.082683] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[32004.093012] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[32004.101463] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[32026.049405] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[32026.059721] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[32026.068142] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[32076.148372] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[32076.158645] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[32076.167043] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[32225.181818] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[32225.192094] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[32225.200552] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[32238.586768] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[32238.597046] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[32238.605412] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[32420.971280] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[32420.981529] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[32420.989944] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[32575.841872] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[32575.852121] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[32575.860506] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[32595.284259] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[32595.294529] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[32595.302954] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[33140.371248] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[33140.381501] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[33140.389951] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[33155.418776] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[33155.429046] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[33155.437429] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[33330.931790] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[33330.942052] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[33330.950424] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[33561.599317] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[33561.610801] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[33561.619200] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[34011.702648] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[34011.712940] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[34011.721362] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[34122.325391] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[34122.335712] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[34122.344176] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[34516.586558] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[34516.596848] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[34516.605353] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[34535.018726] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[34535.029008] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[34535.037402] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[34633.137734] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[34633.153631] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[34633.162189] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[34649.813125] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[34649.823580] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[34649.832037] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[34770.949852] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[34770.960224] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[34770.968853] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[34910.976446] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[34910.986959] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[34910.995370] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[35390.603252] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[35390.614812] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[35390.623206] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[35594.083654] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[35594.093974] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[35594.102432] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[35739.552967] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[35739.563196] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[35739.571639] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[35789.335870] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[35789.347463] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[35789.356132] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[35806.349452] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[35806.359723] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[35806.368114] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[36084.152404] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[36084.164248] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[36084.172644] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[36145.437180] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[36145.447426] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[36145.455930] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[36467.507823] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[36467.518098] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[36467.526514] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[36655.382338] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[36655.392592] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[36655.401378] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[36798.425407] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[36798.435678] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[36798.444179] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[37051.418190] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[37051.428423] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[37051.436819] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[37384.454134] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[37384.464404] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[37384.472878] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[37570.192879] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[37570.203181] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[37570.211642] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[38031.805627] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[38031.815876] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[38031.824287] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[38177.649281] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[38177.659520] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[38177.667906] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[38338.215016] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[38338.225270] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[38338.233681] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[38361.513731] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[38361.523965] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[38361.532481] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[38388.143798] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[38388.154069] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[38388.162480] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[38633.004080] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[38633.015719] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[38633.024110] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[38692.730814] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[38692.741184] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[38692.749590] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[38709.676221] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[38709.686616] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[38709.695003] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[38783.036232] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[38783.046450] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[38783.054821] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[39082.197231] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[39082.207521] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[39082.215926] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[39265.577030] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[39265.587248] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[39265.595620] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[39732.205343] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[39732.215635] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[39732.224312] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[40054.557369] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[40054.567682] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[40054.576072] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[40239.488663] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[40239.499012] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[40239.507777] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[40756.789588] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[40756.799797] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[40756.808160] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[40936.637303] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[40936.647535] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[40936.655935] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[41152.091967] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[41152.102197] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[41152.110581] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[41202.394856] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[41202.405073] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[41202.413439] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[41509.572231] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[41509.583805] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[41509.592192] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[41722.072536] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[41722.082923] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[41722.091361] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[41777.066300] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[41777.076535] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[41777.084950] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[41962.874889] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[41962.885280] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[41962.893701] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[42347.166828] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[42347.177104] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[42347.185481] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[42731.891542] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[42731.901967] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[42731.910438] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[42978.317063] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[42978.327295] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[42978.335709] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[43197.077838] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[43197.088099] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[43197.096498] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[44145.328139] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[44145.339753] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[44145.348235] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[44145.578456] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[44145.588764] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[44145.597230] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[44375.308590] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[44375.318962] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[44375.327438] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[44587.570415] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[44587.580744] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[44587.589130] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[44591.294716] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[44591.304970] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[44591.313441] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[44661.288233] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[44661.299715] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[44661.308094] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[44669.727042] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[44669.737300] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[44669.745815] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[44765.203456] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[44765.213739] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[44765.222161] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[44777.337257] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[44777.347497] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[44777.355895] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[44817.406072] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[44817.416336] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[44817.424720] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[44875.292646] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[44875.302911] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[44875.311377] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[45050.573779] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[45050.584058] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[45050.592452] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[45299.804981] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[45299.815360] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[45299.824008] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[46113.326258] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[46113.336492] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[46113.344999] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[46229.678035] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[46229.688651] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[46229.697043] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[46299.465806] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[46299.476156] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[46299.484574] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[46704.144156] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[46704.154516] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[46704.163001] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[46860.066968] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[46860.077294] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[46860.085676] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[46935.732043] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[46935.742410] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[46935.750856] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[46972.950477] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[46972.962425] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[46972.971721] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[47004.617436] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[47004.627774] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[47004.636324] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[47426.169666] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[47426.179919] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[47426.188363] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[47485.894364] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[47485.904732] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[47485.913282] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[47525.332425] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[47525.342708] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[47525.351152] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[47565.060558] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[47565.070795] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[47565.079209] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[47604.344142] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[47604.354484] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[47604.362871] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[47639.617258] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[47639.627576] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[47639.636028] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[47644.736237] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[47644.746476] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[47644.755009] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[47692.641197] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[47692.651436] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[47692.659900] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[47939.692141] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[47939.702370] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[47939.710755] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

\00\00[0000.125] [L4T TegraBoot] (version 00.00.2018.01-l4t-80a468da)

[0000.131] Processing in cold boot mode Bootloader 2

[0000.135] A02 Bootrom Patch rev = 1023

[0000.139] Power-up reason: pmc por

[0000.142] No Battery Present

[0000.145] pmic max77620 reset reason

[0000.148] pmic max77620 NVERC : 0x50

[0000.151] RamCode = 0

[0000.154] Platform has DDR4 type RAM

The PCIe bus error seems to be a common error with certain hardware and Ubuntu: https://itsfoss.com/pcie-bus-error-severity-corrected/

In any case the system crashed anyway. This time it did not say anything about GPU disappearing.

Thanks again!!!

WayneWWW · September 11, 2020, 3:21am

Hi borelli.g92,

What is the test result of A02 module with pure installation?

As for pcie error over B01, could you share the result of lspci -vv?

borelli.g92 · September 11, 2020, 5:07am

Hi @WayneWWW,

I start now another session of testing with module A02 and I will let you know.

Here follows the result of lspci -vv on module B01:

00:02.0 PCI bridge: NVIDIA Corporation Device 0faf (rev a1) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 84
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: 00001000-00001fff
	Memory behind bridge: 13000000-130fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
	Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 407
	Region 0: I/O ports at 1000 [size=256]
	Region 2: Memory at 13004000 (64-bit, non-prefetchable) [size=4K]
	Region 4: Memory at 13000000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: r8168

Thank you!!

borelli.g92 · September 11, 2020, 4:05pm

Hi @WayneWWW,

I add to my previous message the latest log of today’s test.
As you have suggested I have run the application with a virgin SD installation on the A02 module.
The module rebooted after 9 hour.
With A02 there is no PCIe error appearing continously.

See the serial log below please:

Ubuntu 18.04.4 LTS jnano-desktop ttyS0



jnano-desktop login: [  224.674391] nvmap_alloc_handle: PID 6898: deepstream-test: WARNING: All NvMap Allocations must have a tag to identify the subsystem allocating memory.Please pass the tag to the API call NvRmMemHanldeAllocAttr() or relevant. 

[  276.701356] nvmap_alloc_handle: PID 7485: deepstream-test: WARNING: All NvMap Allocations must have a tag to identify the subsystem allocating memory.Please pass the tag to the API call NvRmMemHanldeAllocAttr() or relevant. 

[  316.528783] EXT4-fs (mmcblk0p1): error count since last fsck: 1

[  316.534716] EXT4-fs (mmcblk0p1): initial error at time 1517155098: htree_dirblock_to_tree:991: inode 285072: block 1059628

[  316.545825] EXT4-fs (mmcblk0p1): last error at time 1517155098: htree_dirblock_to_tree:991: inode 285072: block 1059628

[31998.894624] ------------[ cut here ]------------

[31998.899472] WARNING: CPU: 1 PID: 7579 at /dvs/git/dirty/git-master_linux/kernel/nvgpu/drivers/gpu/nvgpu/gk20a/gk20a.c:64 __gk20a_warn_on_no_regs+0x34/0x50 [nvgpu]

[31998.914557] ---[ end trace 6647317957369feb ]---

[31998.920633] nvgpu: 57000000.gpu           __nvgpu_check_gpu_state:56   [ERR]  GPU has disappeared from bus!!

[31998.930534] nvgpu: 57000000.gpu           __nvgpu_check_gpu_state:57   [ERR]  Rebooting system!!

[31998.981863] reboot: Restarting system

[0000.159] [L4T TegraBoot] (version 00.00.2018.01-l4t-80a468da)

[0000.165] Processing in cold boot mode Bootloader 2

[0000.170] A02 Bootrom Patch rev = 1023

[0000.173] Power-up reason: software reset

[0000.177] No Battery Present

[0000.179] pmic max77620 reset reason

[0000.183] pmic max77620 NVERC : 0x0

At this point the situation is the following:

A02 + SSD USB3.0 + test3-app = reboot with GPU disappeared
B01 + SSD USB3.0 + test3-app = reboot with GPU disappeared and PCIe error
A02 + new SD + test3-app = reboot with GPU disappeared
B01 + new SD + test3-app = reboot with GPU disappeared and PCIe error

Do you have any suggestion?

Thanks again for the support!!!

WayneWWW · September 14, 2020, 4:04am

Hi borelli.g92,

I will use A02 board to run this test for 1 day today and see if I can reproduce this issue.

As for B01, need to wait a few days before I finding a B01 module.

Also want to know, what is the spec of the power supply you are using?

borelli.g92 · September 14, 2020, 9:33am

Dear @WayneWWW,

Thank you very much!

I share with you the source file that I am using as test3.
As anticipated, you will see that I have disabled the standard sink and I have replaced it with a fakesink.
deepstream_test3_app.c (17.1 KB) dstest3_pgie_config.txt (3.5 KB)

I also thought that the problem might have been created by overheating.
I always had a fan active on the heatsink, and A0 temperature never exceeds 45 °C.
However, in order to be sure I have tried to run the system also with a 230V fan please see below pictures.

Unfortunately, nothing change. Thus, I do not think that heating is the problem.

Regarding power supply, I have tried to run the system through two different power supply:

As you can see the power in both cases is definitely higher than needed.

Thank you very much again for your support!!

WayneWWW · September 14, 2020, 9:38am

Hi,

Actually, I am not working on deepstream application so not very sure what is the difference between your case and the original sample.

Thus, I also want to know will you hit this issue even with original deepstream sample from our side?

We need to align the usecase we are running here.

borelli.g92 · September 14, 2020, 9:50am

Hi @WayneWWW,

The only difference in the source file is from line 382 to 389:

  /* Finally render the osd output */
#ifdef PLATFORM_TEGRA
  //transform = gst_element_factory_make ("nvegltransform", "nvegl-transform");
  transform = gst_element_factory_make ("queue", "queue");
#endif
  //sink = gst_element_factory_make ("nveglglessink", "nvvideo-renderer");
//  sink = gst_element_factory_make ("nvoverlaysink", "nvvideo-renderer");
  sink = gst_element_factory_make ("fakesink", "nvvideo-renderer");

Regarding the config file you can see that the difference is that I am using the models optimized for Jetson Nano by nvidia.

Both modifications have been done according to what @DaneLLL is suggesting here:

Those modifications are necessary otherwise the Jetson Nano is not able to process the RTSP video stream for it skips a lot of frames.

Thanks again!

ps. I am using only 1 RTSP video source 640*480 resolution at 15FPS.

borelli.g92 · September 15, 2020, 7:39pm

Dear @WayneWWW,

I have been testing the following configuration:
A02 model + new SD installation + test3-app
With a modification in the pipeline of test3:

#ifdef PLATFORM_TEGRA
  gst_bin_add_many (GST_BIN (pipeline), queue1, pgie, queue2, tiler, queue3,
      nvvidconv, queue4, transform, sink, NULL);  
  if (!gst_element_link_many (streammux, queue1, pgie, queue2, tiler, queue3,
        nvvidconv, queue4, transform, sink, NULL)) {
    g_printerr ("Elements could not be linked. Exiting.\n");
    return -1;
  }
#else

I removed nvosd from the pipeline.
The result is that I achieved a sensible longer time before the usual reboot time.
Almost 28 hours!!
Here follows the log:

NvRmMemHanldeAllocAttr() or relevant. 
[240320.269192] ------------[ cut here ]------------
[240320.274959] WARNING: CPU: 1 PID: 2140 at /dvs/git/dirty/git-master_linux/kernel/nvgpu/drivers/gpu/nvgpu/gk20a/gk20a.c:64 __gk20a_warn_on_no_regs+0x34/0x50 [nvgpu]
[240320.297296] ---[ end trace 6ca8f5afd7c1b41c ]---
[240320.321214] nvgpu: 57000000.gpu           __nvgpu_check_gpu_state:56   [ERR]  GPU has disappeared from bus!!
[240320.331202] nvgpu: 57000000.gpu           __nvgpu_check_gpu_state:57   [ERR]  Rebooting system!!
[240320.404744] reboot: Restarting system
[0000.159] [L4T TegraBoot] (version 00.00.2018.01-l4t-80a468da)
[0000.165] Processing in cold boot mode Bootloader 2
[0000.169] A02 Bootrom Patch rev = 1023
[0000.173] Power-up reason: software reset
[0000.177] No Battery Present
[0000.179] pmic max77620 reset reason
[0000.183] pmic max77620 NVERC : 0x0
[0000.186] RamCode = 0
[0000.188] Platform has DDR4 type RAM
[0000.192] max77620 disabling SD1 Remote Sense
[0000.196] Setting DDR voltage to 1125mv
[0000.200] Serial Number of Pmic Max77663: 0x1235e9
[0000.208] Entering ramdump check
[0000.211] Get RamDumpCarveOut = 0x0
[0000.214] RamDumpCarveOut=0x0,  RamDumperFlag=0xe59ff3f8
[0000.219] Last reboot was clean, booting normally!
[0000.224] Sdram initialization is successful

I believe that the problem is related to some sort of overheating of a component that is not monitored by system’s temperature sensors.
I have found the following post on the forum: Drive PX2 rebooting at high CPU load
The problem in the previous post was related to the fan NOT spinning, thus the GPU was reaching high temperatures.
In my case the fan is spinning very well and, as you have seen from my previous post, I have also tried a 230V high power fan. Same result.

In any case, today I was looking more closely the log that you can see above. I would like to focus on the following lines:

[0000.173] Power-up reason: software reset
[0000.177] No Battery Present
[0000.179] pmic max77620 reset reason

MAX77620 is a power management IC. Does the reboot might be linked to an overheating of a component?

Thanks again!!

WayneWWW · September 16, 2020, 2:44am

Hi,

I don’t think you should refer to posts on PX2 because it is using different SoC from Nano.

Actually, we put our A02 device with test3-app for 2 days: on first day, we use the original sample from jetpack and cannot reproduce. Then, we modify it based on your patch for another 19 hours, still cannot reproduce your issue.

Do you configure your board with nvpmodel and jetson_clocks?

borelli.g92 · September 16, 2020, 7:34am

Hi @WayneWWW,

I have checked the mode with: sudo nvpmodel -q –verbose
This is the result:

NVPM WARN: fan mode is not set!
NV Power Mode: MAXN
0

I also have checked the jetson_clocks with: sudo jetson_clocks --show
This is the result:

SOC family:tegra210  Machine:NVIDIA Jetson Nano Developer Kit
Online CPUs: 0-3
CPU Cluster Switching: Disabled
cpu0: Online=1 Governor=schedutil MinFreq=102000 MaxFreq=1479000 CurrentFreq=1132800 IdleStates: WFI=1 c7=1 
cpu1: Online=1 Governor=schedutil MinFreq=102000 MaxFreq=1479000 CurrentFreq=921600 IdleStates: WFI=1 c7=1 
cpu2: Online=1 Governor=schedutil MinFreq=102000 MaxFreq=1479000 CurrentFreq=1036800 IdleStates: WFI=1 c7=1 
cpu3: Online=1 Governor=schedutil MinFreq=102000 MaxFreq=1479000 CurrentFreq=825600 IdleStates: WFI=1 c7=1 
GPU MinFreq=76800000 MaxFreq=921600000 CurrentFreq=76800000
EMC MinFreq=204000000 MaxFreq=1600000000 CurrentFreq=1600000000 FreqOverride=0
Fan: speed=0
NV Power Mode: MAXN

Do you see some differences with respect to your power configurations?

Thanks again!!

WayneWWW · September 16, 2020, 7:35am

Hi,

I meant running jetson clock to push the clk to limit.

borelli.g92 · September 16, 2020, 9:21am

Hi @WayneWWW,

I do not really understand why rising the clock of the GPU might solve this problem in theory.
However, as it is explained here: https://www.jetsonhacks.com/2019/04/10/jetson-nano-use-more-power/
the Jetson_clocks utility influences the behaviour of the Dynamic Voltage and Frequency Scaling (DVFS).

Maybe after many hours of dynamic scaling of the GPU’s voltage that mechanism fail and some kind of brown-out reset intervenes. Just supposing here.
Now I am running the system with frequency-override on:

SOC family:tegra210  Machine:NVIDIA Jetson Nano Developer Kit
Online CPUs: 0-3
CPU Cluster Switching: Disabled
cpu0: Online=1 Governor=schedutil MinFreq=1479000 MaxFreq=1479000 CurrentFreq=1479000 IdleStates: WFI=0 c7=0 
cpu1: Online=1 Governor=schedutil MinFreq=1479000 MaxFreq=1479000 CurrentFreq=1479000 IdleStates: WFI=0 c7=0 
cpu2: Online=1 Governor=schedutil MinFreq=1479000 MaxFreq=1479000 CurrentFreq=1479000 IdleStates: WFI=0 c7=0 
cpu3: Online=1 Governor=schedutil MinFreq=1479000 MaxFreq=1479000 CurrentFreq=1479000 IdleStates: WFI=0 c7=0 
GPU MinFreq=921600000 MaxFreq=921600000 CurrentFreq=921600000
EMC MinFreq=204000000 MaxFreq=1600000000 CurrentFreq=1600000000 FreqOverride=1
Fan: speed=0
NV Power Mode: MAXN

I will let you know how it goes.

Thanks!!!

WayneWWW · September 16, 2020, 9:42am

Hi,

Frankly speaking, I would like you to RMA both of your devices and try to get a new one.
Since we cannot reproduce this issue with same application, then the reason is something else.

But before doing that, I still filed a ticket internally for our team to check this issue.
Maybe we should reproduce it too but just lucky enough to escape that.

How long is the average time to reproduce this issue? Is it >10 hours?

WayneWWW · September 16, 2020, 9:46am

Also, since we are out of ideas, it is worthy checking if this is related to release revision.

For example, the earliest release that supports A02 nano is r32.1, then maybe you can run the application on that release and see if you can still see issue.

If it cannot, then rel-32.2/rel-32.3.1…

borelli.g92 · September 16, 2020, 10:19am

Hi @WayneWWW,

what is the procedure for RMA.
Could you kinldy point it out?

Here follows a summary of the tests that I have done:

Date	Seconds	Hours	Model	Boot	Pwer Supply	PCIe error	What new
22/08/20	75069	21	A02	SD	ATX, 5V, 15A	No
23/08/20	95304	26	A02	SD	ATX, 5V, 15A	No
24/08/20	33548	9	A02	SD	ATX, 5V, 15A	No
26/08/20	5533	2	A02	USB3.0 SSD	ATX, 5V, 15A	No	I have booted from USB3.0 SSD
01/09/20	22163	6	A02	USB3.0 SSD	5V, 4A	No	I changed power supply
02/09/20	37080	10	A02	USB3.0 SSD	5V, 4A	No
06/09/20	76130	21	B01	USB3.0 SSD	5V, 4A	Yes	I bought a new board: B01 model
07/09/20	25316	7	B01	USB3.0 SSD	5V, 4A	Yes	Logged temperature 38-45°C
08/09/20	5705	2	B01	USB3.0 SSD	5V, 4A	Yes
09/09/20	44245	12	B01	SD	5V, 4A	Yes	Virgin SD installation
10/09/20	108365	30	B01	SD	5V, 4A	Few	No inference running, NO CRASH
11/09/20	31998	9	A02	SD	5V, 4A	No	Virgin SD installation
12/09/20	19152	5	A02	SD	5V, 4A	No	Big Fan 230V
14/09/20	96681	27	A02	SD	5V, 4A	No	No OSD in deepstream pipeline
15/09/20	Running	Running	A02	SD	5V, 4A	No	Only PGIE in deepstream pipeline, after 14 hours I run jetson_clocks

You can see the hours for every test that I have run.

Now I am waiting to see what happens in this last test.
I will keep you updated.

Thanks