Gk20a and Jetson Nano crash

Hi borelli.g92,

Could you try to remove the external storage and see if you can reproduce issue when booting from sdcard only?

Hi @WayneWWW,

yes, booting up from SD card was my starting point.
I have the same error there as well.
If you look in my previous message, 23rd of August log is referring to SD card.
Also I had a number of attempts prior to that with the SD card only.
I also had to repair the filesystem of the SD card a couple of time because of the reboots that were corrupting it.

I have tried both booting from SD card and booting from USB SSD.

Thanks again!

And do you have other sdcard to do the test? I mean are you using only one sdcard on either A02 or B01?

Hi @WayneWWW,
during my tests before 23rd of August I have tested 2 different SD cards on module A02. Same result.
However, I do not understand how SD card can be the problem. It happens with both:

  • booting from SD cards
  • booting from USB3.0 SSD drive

I believe that the origin of the error should be somewhere else right?
Thanks again

Hi,

I just hope we don’t get issues altogether. Could you point out one log that is pure sdcard based image without ssd?

According to your log during 8/24, some logs only have gk20a error. But suddenly, sda1 appeared so obviously it is log with booting from ssd.

Hi @WayneWWW,

Sure, the sull story of SD experiments and corruptions is here:

Here you can see the errors that were repeated in the logs.
However, at that time I was not logging with the serial monitor, thus kern.log is only available. The behaviour has not changed since then:

Thanks

Hi borelli.g92,

Ok so there was a long backstory. But you don’t have new jetson nano at that time, right?

Please follow below steps and try to reproduce issue again…

Could you directly put your sdcards to the A02 and B01 boards, run the sdkmanager to flash your card and then run the deepstream application again and see if you can see the errors other than GPU error?

And no ssd is connected during this test.

Hi @WayneWWW,

ok I will follow those steps, and let you know ASAP.

Thanks!

Hi @WayneWWW,

I have flashed a new SD card and reinstalled Deepstream.
I am now running Test3 on my RTSP source. I have had no crash so far. I have been running the inference for only 5 hours, thus I need to wait until tomorrow at least.

However, I forgot to mention one thing. I have edited a couple of line of test3 for the Jetson Nano struggles to process 640x440 images at 15fps. I have followed a suggestion here on the forum to use a fakesink and adapt the transform to Jetson Nano capability.

/* Finally render the osd output */ 
#ifdef PLATFORM_TEGRA
  //transform = gst_element_factory_make ("nvegltransform", "nvegl-transform");
  transform = gst_element_factory_make ("queue", "queue");
#endif
  //sink = gst_element_factory_make ("nveglglessink", "nvvideo-renderer");
//  sink = gst_element_factory_make ("nvoverlaysink", "nvvideo-renderer");
  sink = gst_element_factory_make ("fakesink", "nvvideo-renderer");

If I do not keep those changings, the Jetson Nano skips a lot of frames.

In any case, even with this new installation I am having continuosly the PCIe error in the serial log file:

[ 3694.486136] nvmap_alloc_handle: PID 25876: deepstream-test: WARNING: All NvMap Allocations must have a tag to identify the subsystem allocating memory.Please pass the tag to the API call NvRmMemHanldeAllocAttr() or relevant. 

[ 3829.505138] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 3829.515495] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 3829.525101] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 4301.401428] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 4301.411648] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 4301.420127] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 4699.923482] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 4699.933845] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 4699.942254] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 4704.380219] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 4704.390432] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 4704.398903] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 4984.462448] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 4984.472733] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 4984.481219] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 5239.496408] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 5239.506646] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 5239.515151] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 5244.856556] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 5244.866894] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 5244.876513] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 5383.112661] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 5383.125853] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 5383.135285] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 5555.630474] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 5555.640725] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 5555.649163] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 5582.934188] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 5582.945952] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 5582.955437] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 5986.752396] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 5986.762661] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 5986.771138] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 6001.066925] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 6001.077253] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 6001.085679] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 6385.286333] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 6385.296596] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 6385.305052] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 6518.078827] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 6518.090378] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 6518.098772] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 6827.179587] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 6827.189865] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 6827.198402] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 6845.122111] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 6845.133746] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 6845.142752] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7136.454764] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7136.465071] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7136.473680] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7315.957757] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7315.968028] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7315.976656] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7428.406574] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7428.416804] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7428.425196] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7594.376226] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7594.386526] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7594.395016] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7742.343007] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7742.353328] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7742.361910] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7805.350766] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7805.361027] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7805.369429] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7895.531107] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7895.541476] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7895.549871] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7934.418950] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7934.430483] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7934.438882] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 8143.524417] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 8143.534724] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 8143.543170] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 8740.552607] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 8740.562898] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 8740.571294] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 8747.308790] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 8747.319122] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 8747.327563] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 8983.357800] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 8983.368024] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 8983.376450] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9047.552739] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9047.564234] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9047.572712] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9083.811655] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9083.821941] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9083.830341] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9247.128331] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9247.140172] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9247.148636] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9348.844755] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9348.855164] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9348.863602] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9518.018461] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9518.028704] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9518.037089] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9599.143523] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9599.153844] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9599.162274] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9691.924645] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9691.936598] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9691.945096] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9825.917152] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9825.929158] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9825.938318] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9965.968999] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9965.979244] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9965.987681] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[10448.900720] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[10448.912456] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[10448.920869] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[10467.754846] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[10467.766459] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[10467.775570] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[10627.103412] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[10627.113638] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[10627.122112] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11032.099079] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11032.109421] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11032.117898] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11156.423722] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11156.434029] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11156.442409] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11432.059473] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11432.069908] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11432.078431] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11536.615924] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11536.626178] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11536.634563] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11663.731535] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11663.741866] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11663.750301] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11704.676000] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11704.686422] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11704.694817] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11715.507221] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11715.517517] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11715.525983] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11954.196226] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11954.206892] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11954.215543] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[12295.010337] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[12295.020945] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[12295.029448] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[12296.191527] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[12296.201792] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[12296.210250] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[12309.818186] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[12309.828488] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[12309.836925] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[12466.580244] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[12466.591948] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[12466.600777] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[12867.862794] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[12867.873095] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[12867.881513] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[12883.473557] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[12883.483868] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[12883.492307] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13233.060192] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13233.070480] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13233.078868] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13274.358051] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13274.369962] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13274.378423] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13314.398364] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13314.408633] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13314.417060] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13407.152802] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13407.163019] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13407.171431] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13803.947849] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13803.958431] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13803.966809] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13853.501711] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13853.512021] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13853.520503] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13863.916831] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13863.928419] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13863.936812] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[14179.851081] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[14179.861348] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[14179.870020] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[14195.625717] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[14195.635966] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[14195.644347] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[14567.343598] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[14567.353881] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[14567.362281] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[14611.627146] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[14611.637447] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[14611.645894] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[14640.574484] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[14640.584770] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[14640.593251] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[14951.978855] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[14951.990443] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[14951.998825] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[15136.815257] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[15136.825543] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[15136.833907] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[15220.585809] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[15220.596070] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[15220.604537] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[15525.006691] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[15525.016964] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[15525.025398] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[15658.621418] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[15658.631678] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[15658.640067] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[15796.143257] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[15796.153480] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[15796.163103] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16124.444871] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16124.455166] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16124.463596] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16261.341463] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16261.351939] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16261.360352] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16321.996622] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16322.007119] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16322.015519] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16511.657290] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16511.667621] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16511.676085] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16738.561125] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16738.571355] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16738.579798] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16748.539225] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16748.549443] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16748.557867] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16801.108411] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16801.120047] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16801.128428] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16870.459330] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16870.469611] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16870.478340] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16922.403772] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16922.414086] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16922.422538] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16995.767352] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16995.777701] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16995.786165] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17241.815603] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17241.827195] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17241.835591] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17262.487767] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17262.498005] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17262.506392] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17313.452181] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17313.462768] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17313.471133] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17347.509027] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17347.519619] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17347.528227] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17465.323200] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17465.333414] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17465.341794] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17676.517559] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17676.527797] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17676.536180] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17705.886292] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17705.896525] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17705.904911] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17763.490621] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17763.500873] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17763.509488] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[18008.554584] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[18008.566358] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[18008.574749] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

Thanks again for your support!!

Hi borelli.g92,

Since you have 2 jetson nanos, please clarify

  1. Does pcie issue happen on both devices? or only one of them?

  2. Will you see pcie error if you run other DS sample that does not transfer data through ethernet?

Hi @WayneWWW,

I start with answering your questions:

  1. No, the PCIe error happens only with the B01 model. The new one that I have bought. It happens both with the new SD card and the system running from USB SSD.
  2. Yes, it also happens with no application running. The serial monitor logs that error even with the board in idel condition. That does not happen with the model A02.

Update on the reboot test. I followed your suggestion and I have tested the B01 board with a virgin installation running the test3 (with modified sink). The system rebooted after 13 hours:

[ 2481.298310] nvmap_alloc_handle: PID 24629: deepstream-test: WARNING: All NvMap Allocations must have a tag to identify the subsystem allocating memory.Please pass the tag to the API call NvRmMemHanldeAllocAttr() or relevant. 

[ 2547.142540] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 2547.153127] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 2547.162748] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 3069.310332] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 3069.320567] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 3069.328951] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 3204.620069] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 3204.630348] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 3204.638751] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 3237.325623] nvmap_alloc_handle: PID 25504: deepstream-test: WARNING: All NvMap Allocations must have a tag to identify the subsystem allocating memory.Please pass the tag to the API call NvRmMemHanldeAllocAttr() or relevant. 

[ 3544.895179] nvmap_alloc_handle: PID 25762: deepstream-test: WARNING: All NvMap Allocations must have a tag to identify the subsystem allocating memory.Please pass the tag to the API call NvRmMemHanldeAllocAttr() or relevant. 

[ 3694.486136] nvmap_alloc_handle: PID 25876: deepstream-test: WARNING: All NvMap Allocations must have a tag to identify the subsystem allocating memory.Please pass the tag to the API call NvRmMemHanldeAllocAttr() or relevant. 

[ 3829.505138] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 3829.515495] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 3829.525101] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 4301.401428] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 4301.411648] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 4301.420127] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 4699.923482] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 4699.933845] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 4699.942254] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 4704.380219] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 4704.390432] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 4704.398903] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 4984.462448] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 4984.472733] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 4984.481219] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 5239.496408] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 5239.506646] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 5239.515151] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 5244.856556] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 5244.866894] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 5244.876513] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 5383.112661] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 5383.125853] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 5383.135285] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 5555.630474] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 5555.640725] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 5555.649163] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 5582.934188] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 5582.945952] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 5582.955437] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 5986.752396] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 5986.762661] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 5986.771138] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 6001.066925] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 6001.077253] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 6001.085679] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 6385.286333] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 6385.296596] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 6385.305052] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 6518.078827] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 6518.090378] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 6518.098772] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 6827.179587] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 6827.189865] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 6827.198402] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 6845.122111] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 6845.133746] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 6845.142752] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7136.454764] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7136.465071] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7136.473680] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7315.957757] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7315.968028] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7315.976656] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7428.406574] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7428.416804] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7428.425196] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7594.376226] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7594.386526] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7594.395016] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7742.343007] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7742.353328] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7742.361910] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7805.350766] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7805.361027] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7805.369429] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7895.531107] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7895.541476] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7895.549871] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 7934.418950] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 7934.430483] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 7934.438882] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 8143.524417] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 8143.534724] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 8143.543170] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 8740.552607] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 8740.562898] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 8740.571294] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 8747.308790] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 8747.319122] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 8747.327563] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 8983.357800] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 8983.368024] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 8983.376450] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9047.552739] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9047.564234] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9047.572712] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9083.811655] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9083.821941] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9083.830341] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9247.128331] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9247.140172] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9247.148636] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9348.844755] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9348.855164] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9348.863602] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9518.018461] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9518.028704] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9518.037089] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9599.143523] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9599.153844] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9599.162274] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9691.924645] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9691.936598] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9691.945096] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9825.917152] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9825.929158] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9825.938318] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[ 9965.968999] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[ 9965.979244] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[ 9965.987681] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[10448.900720] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[10448.912456] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[10448.920869] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[10467.754846] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[10467.766459] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[10467.775570] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[10627.103412] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[10627.113638] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[10627.122112] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11032.099079] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11032.109421] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11032.117898] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11156.423722] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11156.434029] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11156.442409] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11432.059473] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11432.069908] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11432.078431] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11536.615924] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11536.626178] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11536.634563] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11663.731535] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11663.741866] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11663.750301] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11704.676000] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11704.686422] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11704.694817] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11715.507221] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11715.517517] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11715.525983] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[11954.196226] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[11954.206892] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[11954.215543] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[12295.010337] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[12295.020945] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[12295.029448] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[12296.191527] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[12296.201792] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[12296.210250] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[12309.818186] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[12309.828488] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[12309.836925] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[12466.580244] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[12466.591948] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[12466.600777] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[12867.862794] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[12867.873095] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[12867.881513] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[12883.473557] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[12883.483868] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[12883.492307] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13233.060192] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13233.070480] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13233.078868] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13274.358051] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13274.369962] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13274.378423] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13314.398364] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13314.408633] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13314.417060] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13407.152802] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13407.163019] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13407.171431] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13803.947849] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13803.958431] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13803.966809] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13853.501711] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13853.512021] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13853.520503] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[13863.916831] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[13863.928419] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[13863.936812] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[14179.851081] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[14179.861348] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[14179.870020] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[14195.625717] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[14195.635966] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[14195.644347] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[14567.343598] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[14567.353881] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[14567.362281] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[14611.627146] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[14611.637447] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[14611.645894] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[14640.574484] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[14640.584770] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[14640.593251] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[14951.978855] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[14951.990443] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[14951.998825] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[15136.815257] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[15136.825543] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[15136.833907] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[15220.585809] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[15220.596070] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[15220.604537] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[15525.006691] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[15525.016964] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[15525.025398] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[15658.621418] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[15658.631678] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[15658.640067] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[15796.143257] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[15796.153480] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[15796.163103] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16124.444871] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16124.455166] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16124.463596] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16261.341463] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16261.351939] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16261.360352] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16321.996622] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16322.007119] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16322.015519] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16511.657290] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16511.667621] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16511.676085] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16738.561125] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16738.571355] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16738.579798] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16748.539225] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16748.549443] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16748.557867] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16801.108411] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16801.120047] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16801.128428] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16870.459330] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16870.469611] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16870.478340] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16922.403772] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16922.414086] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16922.422538] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[16995.767352] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[16995.777701] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[16995.786165] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17241.815603] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17241.827195] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17241.835591] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17262.487767] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17262.498005] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17262.506392] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17313.452181] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17313.462768] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17313.471133] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17347.509027] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17347.519619] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17347.528227] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17465.323200] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17465.333414] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17465.341794] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17676.517559] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17676.527797] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17676.536180] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17705.886292] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17705.896525] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17705.904911] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[17763.490621] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[17763.500873] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[17763.509488] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[18008.554584] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[18008.566358] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[18008.574749] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[18548.707962] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[18548.719432] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[18548.727830] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[18598.321547] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[18598.331846] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[18598.340362] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[18760.514887] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[18760.525142] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[18760.533538] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[18844.325806] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[18844.337380] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[18844.345822] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[19225.396102] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[19225.406374] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[19225.414769] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[19291.971921] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[19291.982148] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[19291.990532] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[19331.334222] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[19331.344500] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[19331.352907] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[19705.993990] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[19706.004215] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[19706.012615] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[19805.565941] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[19805.576166] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[19805.584574] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[19924.921335] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[19924.931569] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[19924.939981] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20089.184937] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20089.195346] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20089.203787] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20110.701762] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20110.711981] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20110.720371] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20525.956585] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20525.966967] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20525.975451] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20733.711903] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20733.722130] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20733.730540] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20753.976935] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20753.987403] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20753.995802] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20864.658986] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20864.669761] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20864.678353] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20868.316309] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20868.326591] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20868.334988] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20897.298449] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20897.308680] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20897.317065] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[21150.509562] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[21150.519831] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[21150.528261] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[21182.967692] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[21182.977939] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[21182.986345] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[21220.470230] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[21220.480448] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[21220.488849] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[21517.257859] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[21517.268060] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[21517.276438] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[21797.717797] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[21797.728360] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[21797.736827] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[21998.368518] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[21998.378742] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[21998.387130] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22184.790167] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22184.800753] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22184.809158] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22212.721739] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22212.732101] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22212.740540] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22321.441245] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22321.451479] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22321.459877] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22477.596700] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22477.608238] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22477.616791] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22571.813852] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22571.824117] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22571.832520] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23060.573037] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23060.583348] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23060.591796] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23110.707653] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23110.717914] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23110.726314] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23133.214266] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23133.224531] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23133.232985] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23209.655648] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23209.666031] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23209.674445] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23328.692313] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23328.702574] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23328.710998] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23467.437180] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23467.447402] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23467.455977] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23599.543630] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23599.553943] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23599.562368] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23774.274812] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23774.285118] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23774.293697] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[24011.380134] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[24011.390375] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[24011.398796] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[24499.914669] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[24499.924904] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[24499.933295] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[24727.044147] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[24727.054391] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[24727.062782] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25275.608911] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25275.619377] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25275.627832] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25292.497183] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25292.507452] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25292.515857] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25582.014403] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25582.024623] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25582.033024] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25603.346181] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25603.356471] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25603.364858] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25630.307934] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25630.318160] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25630.326557] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25756.848711] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25756.858977] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25756.867364] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25859.029691] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25859.040116] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25859.048517] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25883.022161] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25883.032424] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25883.040975] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[26160.635076] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[26160.645723] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[26160.654113] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[26688.427027] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[26688.437278] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[26688.445663] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[26908.111055] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[26908.121283] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[26908.129681] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[27022.564938] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[27022.575231] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[27022.583664] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[27302.925014] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[27302.935275] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[27302.943705] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[27386.357335] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[27386.367584] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[27386.375972] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[27703.346878] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[27703.357166] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[27703.365557] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[27797.052329] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[27797.062570] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[27797.070949] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[27832.301798] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[27832.312025] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[27832.320432] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[28416.734524] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[28416.744744] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[28416.753156] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[28498.299572] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[28498.309882] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[28498.318384] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[28856.161171] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[28856.171631] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[28856.180139] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[28913.888785] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[28913.899068] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[28913.907516] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[29541.895606] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[29541.905815] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[29541.914257] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[29890.008502] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[29890.018766] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[29890.027196] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[29890.675165] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[29890.685393] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[29890.693774] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[30330.044333] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[30330.054919] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[30330.063327] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[30423.596577] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[30423.606822] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[30423.615280] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[30428.548528] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[30428.558797] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[30428.567235] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[30450.350553] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[30450.361212] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[30450.369698] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[31214.340331] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[31214.350566] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[31214.359027] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[31666.237590] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[31666.249464] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[31666.257996] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[31870.334638] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[31870.345009] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[31870.353416] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[31933.213091] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[31933.223346] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[31933.231734] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[32004.082683] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[32004.093012] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[32004.101463] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[32026.049405] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[32026.059721] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[32026.068142] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[32076.148372] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[32076.158645] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[32076.167043] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[32225.181818] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[32225.192094] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[32225.200552] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[32238.586768] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[32238.597046] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[32238.605412] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[32420.971280] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[32420.981529] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[32420.989944] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[32575.841872] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[32575.852121] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[32575.860506] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[32595.284259] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[32595.294529] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[32595.302954] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[33140.371248] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[33140.381501] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[33140.389951] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[33155.418776] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[33155.429046] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[33155.437429] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[33330.931790] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[33330.942052] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[33330.950424] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[33561.599317] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[33561.610801] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[33561.619200] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[34011.702648] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[34011.712940] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[34011.721362] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[34122.325391] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[34122.335712] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[34122.344176] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[34516.586558] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[34516.596848] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[34516.605353] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[34535.018726] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[34535.029008] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[34535.037402] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[34633.137734] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[34633.153631] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[34633.162189] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[34649.813125] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[34649.823580] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[34649.832037] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[34770.949852] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[34770.960224] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[34770.968853] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[34910.976446] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[34910.986959] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[34910.995370] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[35390.603252] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[35390.614812] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[35390.623206] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[35594.083654] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[35594.093974] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[35594.102432] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[35739.552967] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[35739.563196] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[35739.571639] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[35789.335870] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[35789.347463] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[35789.356132] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[35806.349452] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[35806.359723] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[35806.368114] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[36084.152404] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[36084.164248] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[36084.172644] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[36145.437180] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[36145.447426] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[36145.455930] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[36467.507823] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[36467.518098] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[36467.526514] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[36655.382338] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[36655.392592] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[36655.401378] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[36798.425407] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[36798.435678] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[36798.444179] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[37051.418190] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[37051.428423] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[37051.436819] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[37384.454134] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[37384.464404] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[37384.472878] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[37570.192879] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[37570.203181] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[37570.211642] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[38031.805627] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[38031.815876] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[38031.824287] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[38177.649281] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[38177.659520] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[38177.667906] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[38338.215016] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[38338.225270] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[38338.233681] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[38361.513731] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[38361.523965] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[38361.532481] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[38388.143798] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[38388.154069] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[38388.162480] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[38633.004080] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[38633.015719] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[38633.024110] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[38692.730814] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[38692.741184] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[38692.749590] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[38709.676221] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[38709.686616] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[38709.695003] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[38783.036232] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[38783.046450] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[38783.054821] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[39082.197231] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[39082.207521] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[39082.215926] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[39265.577030] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[39265.587248] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[39265.595620] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[39732.205343] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[39732.215635] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[39732.224312] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[40054.557369] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[40054.567682] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[40054.576072] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[40239.488663] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[40239.499012] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[40239.507777] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[40756.789588] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[40756.799797] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[40756.808160] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[40936.637303] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[40936.647535] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[40936.655935] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[41152.091967] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[41152.102197] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[41152.110581] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[41202.394856] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[41202.405073] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[41202.413439] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[41509.572231] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[41509.583805] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[41509.592192] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[41722.072536] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[41722.082923] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[41722.091361] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[41777.066300] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[41777.076535] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[41777.084950] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[41962.874889] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[41962.885280] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[41962.893701] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[42347.166828] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[42347.177104] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[42347.185481] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[42731.891542] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[42731.901967] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[42731.910438] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[42978.317063] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[42978.327295] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[42978.335709] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[43197.077838] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[43197.088099] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[43197.096498] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[44145.328139] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[44145.339753] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[44145.348235] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[44145.578456] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[44145.588764] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[44145.597230] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[44375.308590] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[44375.318962] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[44375.327438] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[44587.570415] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[44587.580744] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[44587.589130] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[44591.294716] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[44591.304970] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[44591.313441] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[44661.288233] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[44661.299715] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[44661.308094] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[44669.727042] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[44669.737300] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[44669.745815] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[44765.203456] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[44765.213739] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[44765.222161] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[44777.337257] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[44777.347497] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[44777.355895] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[44817.406072] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[44817.416336] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[44817.424720] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[44875.292646] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[44875.302911] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[44875.311377] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[45050.573779] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[45050.584058] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[45050.592452] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[45299.804981] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[45299.815360] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[45299.824008] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[46113.326258] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[46113.336492] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[46113.344999] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[46229.678035] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[46229.688651] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[46229.697043] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[46299.465806] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[46299.476156] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[46299.484574] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[46704.144156] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[46704.154516] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[46704.163001] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[46860.066968] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[46860.077294] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[46860.085676] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[46935.732043] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[46935.742410] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[46935.750856] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[46972.950477] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[46972.962425] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[46972.971721] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[47004.617436] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[47004.627774] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[47004.636324] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[47426.169666] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[47426.179919] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[47426.188363] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[47485.894364] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[47485.904732] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[47485.913282] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[47525.332425] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[47525.342708] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[47525.351152] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[47565.060558] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[47565.070795] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[47565.079209] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[47604.344142] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[47604.354484] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[47604.362871] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[47639.617258] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[47639.627576] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[47639.636028] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[47644.736237] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[47644.746476] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[47644.755009] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[47692.641197] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[47692.651436] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[47692.659900] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[47939.692141] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[47939.702370] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[47939.710755] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

\00\00[0000.125] [L4T TegraBoot] (version 00.00.2018.01-l4t-80a468da)

[0000.131] Processing in cold boot mode Bootloader 2

[0000.135] A02 Bootrom Patch rev = 1023

[0000.139] Power-up reason: pmc por

[0000.142] No Battery Present

[0000.145] pmic max77620 reset reason

[0000.148] pmic max77620 NVERC : 0x50

[0000.151] RamCode = 0

[0000.154] Platform has DDR4 type RAM

The PCIe bus error seems to be a common error with certain hardware and Ubuntu: Troubleshooting PCIe Bus Error severity Corrected on Ubuntu and Linux Mint - It's FOSS

In any case the system crashed anyway. This time it did not say anything about GPU disappearing.

Thanks again!!!

Hi borelli.g92,

What is the test result of A02 module with pure installation?

As for pcie error over B01, could you share the result of lspci -vv?

Hi @WayneWWW,

I start now another session of testing with module A02 and I will let you know.

Here follows the result of lspci -vv on module B01:

00:02.0 PCI bridge: NVIDIA Corporation Device 0faf (rev a1) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 84
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: 00001000-00001fff
	Memory behind bridge: 13000000-130fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: <access denied>
	Kernel driver in use: pcieport

01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
	Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 407
	Region 0: I/O ports at 1000 [size=256]
	Region 2: Memory at 13004000 (64-bit, non-prefetchable) [size=4K]
	Region 4: Memory at 13000000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: r8168

Thank you!!

Hi @WayneWWW,

I add to my previous message the latest log of today’s test.
As you have suggested I have run the application with a virgin SD installation on the A02 module.
The module rebooted after 9 hour.
With A02 there is no PCIe error appearing continously.

See the serial log below please:

Ubuntu 18.04.4 LTS jnano-desktop ttyS0



jnano-desktop login: [  224.674391] nvmap_alloc_handle: PID 6898: deepstream-test: WARNING: All NvMap Allocations must have a tag to identify the subsystem allocating memory.Please pass the tag to the API call NvRmMemHanldeAllocAttr() or relevant. 

[  276.701356] nvmap_alloc_handle: PID 7485: deepstream-test: WARNING: All NvMap Allocations must have a tag to identify the subsystem allocating memory.Please pass the tag to the API call NvRmMemHanldeAllocAttr() or relevant. 

[  316.528783] EXT4-fs (mmcblk0p1): error count since last fsck: 1

[  316.534716] EXT4-fs (mmcblk0p1): initial error at time 1517155098: htree_dirblock_to_tree:991: inode 285072: block 1059628

[  316.545825] EXT4-fs (mmcblk0p1): last error at time 1517155098: htree_dirblock_to_tree:991: inode 285072: block 1059628

[31998.894624] ------------[ cut here ]------------

[31998.899472] WARNING: CPU: 1 PID: 7579 at /dvs/git/dirty/git-master_linux/kernel/nvgpu/drivers/gpu/nvgpu/gk20a/gk20a.c:64 __gk20a_warn_on_no_regs+0x34/0x50 [nvgpu]

[31998.914557] ---[ end trace 6647317957369feb ]---

[31998.920633] nvgpu: 57000000.gpu           __nvgpu_check_gpu_state:56   [ERR]  GPU has disappeared from bus!!

[31998.930534] nvgpu: 57000000.gpu           __nvgpu_check_gpu_state:57   [ERR]  Rebooting system!!

[31998.981863] reboot: Restarting system

[0000.159] [L4T TegraBoot] (version 00.00.2018.01-l4t-80a468da)

[0000.165] Processing in cold boot mode Bootloader 2

[0000.170] A02 Bootrom Patch rev = 1023

[0000.173] Power-up reason: software reset

[0000.177] No Battery Present

[0000.179] pmic max77620 reset reason

[0000.183] pmic max77620 NVERC : 0x0

At this point the situation is the following:

A02 + SSD USB3.0 + test3-app = reboot with GPU disappeared
B01 + SSD USB3.0 + test3-app = reboot with GPU disappeared and PCIe error
A02 + new SD + test3-app = reboot with GPU disappeared
B01 + new SD + test3-app = reboot with GPU disappeared and PCIe error

Do you have any suggestion?

Thanks again for the support!!!

Hi borelli.g92,

I will use A02 board to run this test for 1 day today and see if I can reproduce this issue.

As for B01, need to wait a few days before I finding a B01 module.

Also want to know, what is the spec of the power supply you are using?

Dear @WayneWWW,

Thank you very much!

I share with you the source file that I am using as test3.
As anticipated, you will see that I have disabled the standard sink and I have replaced it with a fakesink.
deepstream_test3_app.c (17.1 KB) dstest3_pgie_config.txt (3.5 KB)

I also thought that the problem might have been created by overheating.
I always had a fan active on the heatsink, and A0 temperature never exceeds 45 °C.
However, in order to be sure I have tried to run the system also with a 230V fan please see below pictures.

Unfortunately, nothing change. Thus, I do not think that heating is the problem.

Regarding power supply, I have tried to run the system through two different power supply:

As you can see the power in both cases is definitely higher than needed.

Thank you very much again for your support!!

Hi,

Actually, I am not working on deepstream application so not very sure what is the difference between your case and the original sample.

Thus, I also want to know will you hit this issue even with original deepstream sample from our side?

We need to align the usecase we are running here.

Hi @WayneWWW,

The only difference in the source file is from line 382 to 389:

  /* Finally render the osd output */
#ifdef PLATFORM_TEGRA
  //transform = gst_element_factory_make ("nvegltransform", "nvegl-transform");
  transform = gst_element_factory_make ("queue", "queue");
#endif
  //sink = gst_element_factory_make ("nveglglessink", "nvvideo-renderer");
//  sink = gst_element_factory_make ("nvoverlaysink", "nvvideo-renderer");
  sink = gst_element_factory_make ("fakesink", "nvvideo-renderer");

Regarding the config file you can see that the difference is that I am using the models optimized for Jetson Nano by nvidia.

Both modifications have been done according to what @DaneLLL is suggesting here:

Those modifications are necessary otherwise the Jetson Nano is not able to process the RTSP video stream for it skips a lot of frames.

Thanks again!

ps. I am using only 1 RTSP video source 640*480 resolution at 15FPS.

Dear @WayneWWW,

I have been testing the following configuration:
A02 model + new SD installation + test3-app
With a modification in the pipeline of test3:

#ifdef PLATFORM_TEGRA
  gst_bin_add_many (GST_BIN (pipeline), queue1, pgie, queue2, tiler, queue3,
      nvvidconv, queue4, transform, sink, NULL);  
  if (!gst_element_link_many (streammux, queue1, pgie, queue2, tiler, queue3,
        nvvidconv, queue4, transform, sink, NULL)) {
    g_printerr ("Elements could not be linked. Exiting.\n");
    return -1;
  }
#else 

I removed nvosd from the pipeline.
The result is that I achieved a sensible longer time before the usual reboot time.
Almost 28 hours!!
Here follows the log:

NvRmMemHanldeAllocAttr() or relevant. 
[240320.269192] ------------[ cut here ]------------
[240320.274959] WARNING: CPU: 1 PID: 2140 at /dvs/git/dirty/git-master_linux/kernel/nvgpu/drivers/gpu/nvgpu/gk20a/gk20a.c:64 __gk20a_warn_on_no_regs+0x34/0x50 [nvgpu]
[240320.297296] ---[ end trace 6ca8f5afd7c1b41c ]---
[240320.321214] nvgpu: 57000000.gpu           __nvgpu_check_gpu_state:56   [ERR]  GPU has disappeared from bus!!
[240320.331202] nvgpu: 57000000.gpu           __nvgpu_check_gpu_state:57   [ERR]  Rebooting system!!
[240320.404744] reboot: Restarting system
[0000.159] [L4T TegraBoot] (version 00.00.2018.01-l4t-80a468da)
[0000.165] Processing in cold boot mode Bootloader 2
[0000.169] A02 Bootrom Patch rev = 1023
[0000.173] Power-up reason: software reset
[0000.177] No Battery Present
[0000.179] pmic max77620 reset reason
[0000.183] pmic max77620 NVERC : 0x0
[0000.186] RamCode = 0
[0000.188] Platform has DDR4 type RAM
[0000.192] max77620 disabling SD1 Remote Sense
[0000.196] Setting DDR voltage to 1125mv
[0000.200] Serial Number of Pmic Max77663: 0x1235e9
[0000.208] Entering ramdump check
[0000.211] Get RamDumpCarveOut = 0x0
[0000.214] RamDumpCarveOut=0x0,  RamDumperFlag=0xe59ff3f8
[0000.219] Last reboot was clean, booting normally!
[0000.224] Sdram initialization is successful 

I believe that the problem is related to some sort of overheating of a component that is not monitored by system’s temperature sensors.
I have found the following post on the forum: Drive PX2 rebooting at high CPU load
The problem in the previous post was related to the fan NOT spinning, thus the GPU was reaching high temperatures.
In my case the fan is spinning very well and, as you have seen from my previous post, I have also tried a 230V high power fan. Same result.

In any case, today I was looking more closely the log that you can see above. I would like to focus on the following lines:

[0000.173] Power-up reason: software reset
[0000.177] No Battery Present
[0000.179] pmic max77620 reset reason

MAX77620 is a power management IC. Does the reboot might be linked to an overheating of a component?

Thanks again!!

Hi,

I don’t think you should refer to posts on PX2 because it is using different SoC from Nano.

Actually, we put our A02 device with test3-app for 2 days: on first day, we use the original sample from jetpack and cannot reproduce. Then, we modify it based on your patch for another 19 hours, still cannot reproduce your issue.

Do you configure your board with nvpmodel and jetson_clocks?