Gk20a and Jetson Nano crash

borelli.g92 · September 2, 2020, 4:39pm

Dear all,

I have been running the Jetson Nano with JetPack-4.4 GA to run deepstream sample app test3 overnight.
Test3 is processing 1 RTSP streaming 15fps, 640*480 resolution.
After a random number of hours (usually greater than 6) the Jetson Nano reboots.

Initially I thought that the problem was the SD card, thus I am now running the system from a USB3.0 SSD.

Then, I thought that the problem might be due to the power supply. I tried a different one, but the problem is still there. I have a fan on top of the heat sink, and I also monitor A0 temperature: it never goes over 40 celsius degree.

Sometime kern.log is not able to record anything. Sometime this is what has been recorded:

Aug 23 19:12:20 jnano-desktop kernel: [95304.891839] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:50   [ERR]  PRI timeout: ADR 0x00400120 READ  DATA 0x00000000
Aug 23 19:12:20 jnano-desktop kernel: [95304.903609] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:56   [ERR]  FECS_ERRCODE 0xbadf1301
Aug 23 19:51:58 jnano-desktop kernel: [    0.000000] Booting Linux on physical CPU 0x0
Aug 23 19:51:58 jnano-desktop kernel: [    0.000000] Linux version 4.9.140-tegra (buildbrain@mobile-u64-3456) (gcc version 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a424ecfbc167ef90065c0eeb7f91977701] (Linaro GCC 7.3-2018.05) ) #1 SMP PREEMPT Thu Jun 25 21:25:44 PDT 2020
Aug 23 19:51:58 jnano-desktop kernel: [    0.000000] Boot CPU: AArch64 Processor [411fd071]

I also have setup a serial logger to read from Jetson Nano serial console. From there I am always able to get some log:

Ubuntu 18.04.4 LTS jnano-desktop ttyS0

jnano-desktop login: [   31.659552] EXT4-fs (mmcblk0p1): warning: mounting fs with errors, running e2fsck is recommended
[  147.962522] nvmap_alloc_handle: PID 4722: deepstream-test: WARNING: All NvMap Allocations must have a tag to identify the subsystem allocating memory.Please pass the tag to the API call NvRmMemHanldeAllocAttr() or relevant. 
[22163.067150] ------------[ cut here ]------------
[22163.072812] WARNING: CPU: 2 PID: 4905 at /dvs/git/dirty/git-master_linux/kernel/nvgpu/drivers/gpu/nvgpu/gk20a/gk20a.c:64 __gk20a_warn_on_no_regs+0x34/0x50 [nvgpu]
[22163.093261] ---[ end trace 91b6de834d66ac90 ]---
[22163.100131] nvgpu: 57000000.gpu           __nvgpu_check_gpu_state:56   [ERR]  GPU has disappeared from bus!!
[22163.109978] nvgpu: 57000000.gpu           __nvgpu_check_gpu_state:57   [ERR]  Rebooting system!!
[22163.121761] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 11141527 (offset 4096 size 4096 starting block 44640094)
[22163.134959] Buffer I/O error on device sda1, logical block 44639836
[22163.141218] Buffer I/O error on device sda1, logical block 44639837
[22163.147506] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 11141571 (offset 180224 size 4096 starting block 9712)
[22163.160524] Buffer I/O error on device sda1, logical block 9454
[22163.166436] Buffer I/O error on device sda1, logical block 9455
[22163.172608] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 11141527 (offset 0 size 0 starting block 44640094)
[22163.185281] Buffer I/O error on device sda1, logical block 44639837
[22163.191766] JBD2: Detected IO errors while flushing file data on sda1-8
[22163.198515] Aborting journal on device sda1-8.
[22163.203030] JBD2: Error -5 detected when updating journal superblock for sda1-8.
[22163.203142] EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected aborted journal
[22163.203145] EXT4-fs (sda1): Remounting filesystem read-only
[22163.203150] EXT4-fs (sda1): previous I/O error to superblock detected
[22163.299469] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm (spawn): error -5 reading directory block
[22163.312052] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block
[22163.323953] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #529938: comm colord: reading directory lblock 0
[22163.323968] EXT4-fs (sda1): previous I/O error to superblock detected
[22163.331417] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #14155777: comm (umount): reading directory lblock 0
[22163.338942] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo
[22163.338946] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg
[22163.350039] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #7340034: comm bash: reading directory lblock 0
[22163.384784] nvgpu: 57000000.gpu             gk20a_channel_release:455  [ERR]  failed to release a channel!
[22163.384800] nvgpu: 57000000.gpu             gk20a_channel_release:455  [ERR]  failed to release a channel!
[22163.418723] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm gdm-session-wor: error -5 reading directory block
[22163.440653] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm gvfs-udisks2-vo: error -5 reading directory block
[22163.441591] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm gvfs-udisks2-vo: error -5 reading directory block
[22163.441614] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm gvfs-udisks2-vo: error -5 reading directory block
[22163.441630] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm gvfs-udisks2-vo: error -5 reading directory block
[22163.443379] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #2: comm pool: reading directory lblock 0
[22163.443490] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #2: comm pool: reading directory lblock 0
[22163.446133] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #14417921: comm gnome-session-b: reading directory lblock 0
[22163.447900] reboot: Restarting system
[0000.159] [L4T TegraBoot] (version 00.00.2018.01-l4t-80a468da)
[0000.165] Processing in cold boot mode Bootloader 2
[0000.170] A02 Bootrom Patch rev = 1023
[0000.173] Power-up reason: software reset
[0000.177] No Battery Present

I have also been running the Jetson Nano for more than 48 hours without doing any task to see if that was happening also with the system in idle. No crash happened. The crash seems to happen when you use the GPU for long time.
In my case last night happened after 6 hours only.

I hope that you could give me some advice on this error.

Thank you!

WayneWWW · September 3, 2020, 2:53am

Hi,

Thanks for reporting this.

Do you write anything into your disk when running this application? Besides the gpu error, I also see some I/O error from sda1.

As for the gpu error, could you also try few tests and see if you can always see same error?

[22163.100131] nvgpu: 57000000.gpu __nvgpu_check_gpu_state:56 [ERR] GPU has disappeared from bus!!

This error seems rare.

borelli.g92 · September 3, 2020, 6:24am

Dear @WayneWWW,

Thanks for your message!

First, I do not write anything into the disk. The only files that are written to the disk are the logs. Maybe those errors are related to the fact that the logs cannot be written. Most of the time the error is not reported in the logs, but only thorugh the serial console.

Second, I have tried many times, and I always have the same error. I will report here the logs of my test:

23rd of August, kern.log

Aug 23 07:49:57 jnano-desktop kernel: [54363.034977] nvmap_alloc_handle: PID 29292: deepstream-test: WARNING: All NvMap Allocations must have a tag to identify the subsystem allocating memory.Please pass the tag to the API call NvRmMemHanldeAllocAttr() or relevant. 
Aug 23 07:54:40 jnano-desktop kernel: [54646.510702] tegradc tegradc.0: blank - powerdown
Aug 23 07:54:40 jnano-desktop kernel: [54646.572106] extcon-disp-state extcon:disp-state: cable 47 state 0
Aug 23 07:54:40 jnano-desktop kernel: [54646.572112] Extcon AUX1(HDMI) disable
Aug 23 10:30:03 jnano-desktop kernel: [63969.578713] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:50   [ERR]  PRI timeout: ADR 0x00400120 READ  DATA 0x00000000
Aug 23 10:30:03 jnano-desktop kernel: [63969.590461] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:56   [ERR]  FECS_ERRCODE 0xbadf1301
Aug 23 18:44:38 jnano-desktop kernel: [93643.601731] extcon-disp-state extcon:disp-state: cable 51 state 0
Aug 23 18:44:38 jnano-desktop kernel: [93643.601779] Extcon HDMI: HPD disabled
Aug 23 18:44:38 jnano-desktop kernel: [93643.602075] tegradc tegradc.0: hdmi: unplugged
Aug 23 18:44:38 jnano-desktop kernel: [93643.650657] tegradc tegradc.0: blank - powerdown
Aug 23 18:44:38 jnano-desktop kernel: [93643.650670] tegradc tegradc.0: unblank
Aug 23 18:44:38 jnano-desktop kernel: [93643.655476] tegradc tegradc.0: nominal-pclk:148500000 parent:148500000 div:1.0 pclk:148500000 147015000~161865000
Aug 23 18:44:38 jnano-desktop kernel: [93643.655553] tegradc tegradc.0: hdmi: tmds rate:148500K prod-setting:prod_c_hdmi_75m_150m
Aug 23 18:44:38 jnano-desktop kernel: [93643.656615] tegradc tegradc.0: hdmi: get RGB quant from EDID.
Aug 23 18:44:38 jnano-desktop kernel: [93643.656622] tegradc tegradc.0: hdmi: get YCC quant from EDID.
Aug 23 18:44:38 jnano-desktop kernel: [93643.696070] extcon-disp-state extcon:disp-state: cable 47 state 1
Aug 23 18:44:38 jnano-desktop kernel: [93643.696079] Extcon AUX1(HDMI) enable
Aug 23 18:44:38 jnano-desktop kernel: [93643.702528] tegradc tegradc.0: unblank
Aug 23 18:44:38 jnano-desktop kernel: [93643.702536] tegradc tegradc.1: blank - powerdown
Aug 23 18:44:38 jnano-desktop kernel: [93643.780019] tegradc tegradc.0: blank - powerdown
Aug 23 18:44:38 jnano-desktop kernel: [93643.840249] extcon-disp-state extcon:disp-state: cable 47 state 0
Aug 23 18:44:38 jnano-desktop kernel: [93643.840252] Extcon AUX1(HDMI) disable
Aug 23 18:44:38 jnano-desktop kernel: [93643.860684] tegradc tegradc.0: unblank
Aug 23 18:44:38 jnano-desktop kernel: [93643.871260] tegradc tegradc.0: nominal-pclk:148500000 parent:148500000 div:1.0 pclk:148500000 147015000~161865000
Aug 23 18:44:38 jnano-desktop kernel: [93643.871351] tegradc tegradc.0: hdmi: tmds rate:148500K prod-setting:prod_c_hdmi_75m_150m
Aug 23 18:44:38 jnano-desktop kernel: [93643.872878] tegradc tegradc.0: hdmi: get RGB quant from EDID.
Aug 23 18:44:38 jnano-desktop kernel: [93643.872886] tegradc tegradc.0: hdmi: get YCC quant from EDID.
Aug 23 18:44:38 jnano-desktop kernel: [93643.907659] extcon-disp-state extcon:disp-state: cable 47 state 1
Aug 23 18:44:38 jnano-desktop kernel: [93643.907662] Extcon AUX1(HDMI) enable
Aug 23 18:44:38 jnano-desktop kernel: [93643.907799] extcon-disp-state extcon:disp-state: cable 51 state 1
Aug 23 18:44:38 jnano-desktop kernel: [93643.907802] Extcon HDMI: HPD enabled
Aug 23 18:44:38 jnano-desktop kernel: [93643.907823] tegradc tegradc.0: hdmi: plugged
Aug 23 18:44:39 jnano-desktop kernel: [93643.923794] tegradc tegradc.0: blank - powerdown
Aug 23 18:44:39 jnano-desktop kernel: [93643.989879] extcon-disp-state extcon:disp-state: cable 47 state 0
Aug 23 18:44:39 jnano-desktop kernel: [93643.989881] Extcon AUX1(HDMI) disable
Aug 23 18:44:39 jnano-desktop kernel: [93644.010277] tegradc tegradc.0: unblank
Aug 23 18:44:39 jnano-desktop kernel: [93644.021030] tegradc tegradc.0: nominal-pclk:148500000 parent:148500000 div:1.0 pclk:148500000 147015000~161865000
Aug 23 18:44:39 jnano-desktop kernel: [93644.021102] tegradc tegradc.0: hdmi: tmds rate:148500K prod-setting:prod_c_hdmi_75m_150m
Aug 23 18:44:39 jnano-desktop kernel: [93644.022113] tegradc tegradc.0: hdmi: get RGB quant from EDID.
Aug 23 18:44:39 jnano-desktop kernel: [93644.022120] tegradc tegradc.0: hdmi: get YCC quant from EDID.
Aug 23 18:44:39 jnano-desktop kernel: [93644.059660] extcon-disp-state extcon:disp-state: cable 47 state 1
Aug 23 18:44:39 jnano-desktop kernel: [93644.059663] Extcon AUX1(HDMI) enable
Aug 23 18:44:39 jnano-desktop kernel: [93644.060146] tegradc tegradc.0: unblank
Aug 23 18:44:39 jnano-desktop kernel: [93644.060155] tegradc tegradc.1: blank - powerdown
Aug 23 18:44:39 jnano-desktop kernel: [93644.230186] tegradc tegradc.0: unblank
Aug 23 18:44:39 jnano-desktop kernel: [93644.230196] tegradc tegradc.1: blank - powerdown
Aug 23 18:44:39 jnano-desktop kernel: [93644.239098] extcon-disp-state extcon:disp-state: cable 47 state 0
Aug 23 18:44:39 jnano-desktop kernel: [93644.239102] Extcon AUX1(HDMI) disable
Aug 23 18:44:39 jnano-desktop kernel: [93644.259593] tegradc tegradc.0: blank - powerdown
Aug 23 18:44:39 jnano-desktop kernel: [93644.259655] extcon-disp-state extcon:disp-state: cable 51 state 0
Aug 23 18:44:39 jnano-desktop kernel: [93644.259658] Extcon HDMI: HPD disabled
Aug 23 18:44:39 jnano-desktop kernel: [93644.259677] tegradc tegradc.0: hdmi: unplugged
Aug 23 18:44:39 jnano-desktop kernel: [93644.263794] tegradc tegradc.0: blank - powerdown
Aug 23 18:44:39 jnano-desktop kernel: [93644.263806] tegradc tegradc.0: unblank
Aug 23 18:44:39 jnano-desktop kernel: [93644.268638] tegradc tegradc.0: nominal-pclk:148500000 parent:148500000 div:1.0 pclk:148500000 147015000~161865000
Aug 23 18:44:39 jnano-desktop kernel: [93644.268714] tegradc tegradc.0: hdmi: tmds rate:148500K prod-setting:prod_c_hdmi_75m_150m
Aug 23 18:44:39 jnano-desktop kernel: [93644.269828] tegradc tegradc.0: hdmi: get RGB quant from EDID.
Aug 23 18:44:39 jnano-desktop kernel: [93644.269833] tegradc tegradc.0: hdmi: get YCC quant from EDID.
Aug 23 18:44:39 jnano-desktop kernel: [93644.304572] extcon-disp-state extcon:disp-state: cable 47 state 1
Aug 23 18:44:39 jnano-desktop kernel: [93644.304574] Extcon AUX1(HDMI) enable
Aug 23 18:44:39 jnano-desktop kernel: [93644.304742] tegradc tegradc.0: unblank
Aug 23 18:44:39 jnano-desktop kernel: [93644.304749] tegradc tegradc.1: blank - powerdown
Aug 23 18:44:39 jnano-desktop kernel: [93644.359816] tegradc tegradc.0: blank - powerdown
Aug 23 18:44:39 jnano-desktop kernel: [93644.420130] extcon-disp-state extcon:disp-state: cable 47 state 0
Aug 23 18:44:39 jnano-desktop kernel: [93644.420133] Extcon AUX1(HDMI) disable
Aug 23 18:44:39 jnano-desktop kernel: [93644.440522] tegradc tegradc.0: unblank
Aug 23 18:44:39 jnano-desktop kernel: [93644.451010] tegradc tegradc.0: nominal-pclk:148500000 parent:148500000 div:1.0 pclk:148500000 147015000~161865000
Aug 23 18:44:39 jnano-desktop kernel: [93644.451079] tegradc tegradc.0: hdmi: tmds rate:148500K prod-setting:prod_c_hdmi_75m_150m
Aug 23 18:44:39 jnano-desktop kernel: [93644.452090] tegradc tegradc.0: hdmi: get RGB quant from EDID.
Aug 23 18:44:39 jnano-desktop kernel: [93644.452097] tegradc tegradc.0: hdmi: get YCC quant from EDID.
Aug 23 18:44:39 jnano-desktop kernel: [93644.486807] extcon-disp-state extcon:disp-state: cable 47 state 1
Aug 23 18:44:39 jnano-desktop kernel: [93644.486809] Extcon AUX1(HDMI) enable
Aug 23 18:44:39 jnano-desktop kernel: [93644.486938] extcon-disp-state extcon:disp-state: cable 51 state 1
Aug 23 18:44:39 jnano-desktop kernel: [93644.486940] Extcon HDMI: HPD enabled
Aug 23 18:44:39 jnano-desktop kernel: [93644.486959] tegradc tegradc.0: hdmi: plugged
Aug 23 18:44:39 jnano-desktop kernel: [93644.495701] tegradc tegradc.0: blank - powerdown
Aug 23 18:44:39 jnano-desktop kernel: [93644.552453] extcon-disp-state extcon:disp-state: cable 47 state 0
Aug 23 18:44:39 jnano-desktop kernel: [93644.552456] Extcon AUX1(HDMI) disable
Aug 23 18:44:39 jnano-desktop kernel: [93644.572859] tegradc tegradc.0: unblank
Aug 23 18:44:39 jnano-desktop kernel: [93644.583423] tegradc tegradc.0: nominal-pclk:148500000 parent:148500000 div:1.0 pclk:148500000 147015000~161865000
Aug 23 18:44:39 jnano-desktop kernel: [93644.583510] tegradc tegradc.0: hdmi: tmds rate:148500K prod-setting:prod_c_hdmi_75m_150m
Aug 23 18:44:39 jnano-desktop kernel: [93644.584557] tegradc tegradc.0: hdmi: get RGB quant from EDID.
Aug 23 18:44:39 jnano-desktop kernel: [93644.584565] tegradc tegradc.0: hdmi: get YCC quant from EDID.
Aug 23 18:44:39 jnano-desktop kernel: [93644.620433] extcon-disp-state extcon:disp-state: cable 47 state 1
Aug 23 18:44:39 jnano-desktop kernel: [93644.620435] Extcon AUX1(HDMI) enable
Aug 23 18:44:39 jnano-desktop kernel: [93644.621011] tegradc tegradc.0: unblank
Aug 23 18:44:39 jnano-desktop kernel: [93644.621082] tegradc tegradc.1: blank - powerdown
Aug 23 18:44:39 jnano-desktop kernel: [93644.783810] tegradc tegradc.0: unblank
Aug 23 18:44:39 jnano-desktop kernel: [93644.783821] tegradc tegradc.1: blank - powerdown
Aug 23 18:50:55 jnano-desktop kernel: [94020.371575] tegradc tegradc.0: blank - powerdown
Aug 23 18:50:55 jnano-desktop kernel: [94020.434106] extcon-disp-state extcon:disp-state: cable 47 state 0
Aug 23 18:50:55 jnano-desktop kernel: [94020.434109] Extcon AUX1(HDMI) disable
Aug 23 18:51:19 jnano-desktop kernel: [94044.179683] extcon-disp-state extcon:disp-state: cable 51 state 0
Aug 23 18:51:19 jnano-desktop kernel: [94044.179734] Extcon HDMI: HPD disabled
Aug 23 18:51:19 jnano-desktop kernel: [94044.180014] tegradc tegradc.0: hdmi: unplugged
Aug 23 18:51:19 jnano-desktop kernel: [94044.197679] tegradc tegradc.0: blank - powerdown
Aug 23 18:51:19 jnano-desktop kernel: [94044.197694] tegradc tegradc.0: unblank
Aug 23 18:51:19 jnano-desktop kernel: [94044.197725] tegradc tegradc.0: unblank
Aug 23 18:51:19 jnano-desktop kernel: [94044.197736] tegradc tegradc.1: blank - powerdown
Aug 23 18:51:19 jnano-desktop kernel: [94044.361986] tegradc tegradc.0: blank - powerdown
Aug 23 18:51:19 jnano-desktop kernel: [94044.362003] tegradc tegradc.0: unblank
Aug 23 18:51:19 jnano-desktop kernel: [94044.372660] tegradc tegradc.0: nominal-pclk:148500000 parent:148500000 div:1.0 pclk:148500000 147015000~161865000
Aug 23 18:51:19 jnano-desktop kernel: [94044.372753] tegradc tegradc.0: hdmi: tmds rate:148500K prod-setting:prod_c_hdmi_75m_150m
Aug 23 18:51:19 jnano-desktop kernel: [94044.376410] tegradc tegradc.0: hdmi: get RGB quant from EDID.
Aug 23 18:51:19 jnano-desktop kernel: [94044.376419] tegradc tegradc.0: hdmi: get YCC quant from EDID.
Aug 23 18:51:19 jnano-desktop kernel: [94044.411243] extcon-disp-state extcon:disp-state: cable 47 state 1
Aug 23 18:51:19 jnano-desktop kernel: [94044.411246] Extcon AUX1(HDMI) enable
Aug 23 18:51:19 jnano-desktop kernel: [94044.411374] extcon-disp-state extcon:disp-state: cable 51 state 1
Aug 23 18:51:19 jnano-desktop kernel: [94044.411377] Extcon HDMI: HPD enabled
Aug 23 18:51:19 jnano-desktop kernel: [94044.411396] tegradc tegradc.0: hdmi: plugged
Aug 23 18:51:19 jnano-desktop kernel: [94044.445639] tegradc tegradc.0: blank - powerdown
Aug 23 18:51:19 jnano-desktop kernel: [94044.510098] extcon-disp-state extcon:disp-state: cable 47 state 0
Aug 23 18:51:19 jnano-desktop kernel: [94044.510100] Extcon AUX1(HDMI) disable
Aug 23 18:51:19 jnano-desktop kernel: [94044.530652] tegradc tegradc.0: unblank
Aug 23 18:51:19 jnano-desktop kernel: [94044.541153] tegradc tegradc.0: nominal-pclk:148500000 parent:148500000 div:1.0 pclk:148500000 147015000~161865000
Aug 23 18:51:19 jnano-desktop kernel: [94044.541226] tegradc tegradc.0: hdmi: tmds rate:148500K prod-setting:prod_c_hdmi_75m_150m
Aug 23 18:51:19 jnano-desktop kernel: [94044.542235] tegradc tegradc.0: hdmi: get RGB quant from EDID.
Aug 23 18:51:19 jnano-desktop kernel: [94044.542242] tegradc tegradc.0: hdmi: get YCC quant from EDID.
Aug 23 18:51:19 jnano-desktop kernel: [94044.578465] extcon-disp-state extcon:disp-state: cable 47 state 1
Aug 23 18:51:19 jnano-desktop kernel: [94044.578467] Extcon AUX1(HDMI) enable
Aug 23 18:51:19 jnano-desktop kernel: [94044.578636] tegradc tegradc.0: unblank
Aug 23 18:51:19 jnano-desktop kernel: [94044.578913] tegradc tegradc.1: blank - powerdown
Aug 23 18:51:19 jnano-desktop kernel: [94044.641305] tegradc tegradc.0: unblank
Aug 23 18:51:19 jnano-desktop kernel: [94044.641462] tegradc tegradc.1: blank - powerdown
Aug 23 19:05:22 jnano-desktop kernel: [94887.450038] nvmap_alloc_handle: PID 4509: deepstream-test: WARNING: All NvMap Allocations must have a tag to identify the subsystem allocating memory.Please pass the tag to the API call NvRmMemHanldeAllocAttr() or relevant. 
Aug 23 19:10:05 jnano-desktop kernel: [95170.678791] tegradc tegradc.0: blank - powerdown
Aug 23 19:10:05 jnano-desktop kernel: [95170.739795] extcon-disp-state extcon:disp-state: cable 47 state 0
Aug 23 19:10:05 jnano-desktop kernel: [95170.739801] Extcon AUX1(HDMI) disable
Aug 23 19:10:29 jnano-desktop kernel: [95194.490356] extcon-disp-state extcon:disp-state: cable 51 state 0
Aug 23 19:10:29 jnano-desktop kernel: [95194.490372] Extcon HDMI: HPD disabled
Aug 23 19:10:29 jnano-desktop kernel: [95194.490471] tegradc tegradc.0: hdmi: unplugged
Aug 23 19:10:29 jnano-desktop kernel: [95194.519762] tegradc tegradc.0: blank - powerdown
Aug 23 19:10:29 jnano-desktop kernel: [95194.519778] tegradc tegradc.0: unblank
Aug 23 19:10:29 jnano-desktop kernel: [95194.519816] tegradc tegradc.0: unblank
Aug 23 19:10:29 jnano-desktop kernel: [95194.519826] tegradc tegradc.1: blank - powerdown
Aug 23 19:10:29 jnano-desktop kernel: [95194.669559] tegradc tegradc.0: blank - powerdown
Aug 23 19:10:29 jnano-desktop kernel: [95194.669578] tegradc tegradc.0: unblank
Aug 23 19:10:29 jnano-desktop kernel: [95194.680221] tegradc tegradc.0: nominal-pclk:148500000 parent:148500000 div:1.0 pclk:148500000 147015000~161865000
Aug 23 19:10:29 jnano-desktop kernel: [95194.680320] tegradc tegradc.0: hdmi: tmds rate:148500K prod-setting:prod_c_hdmi_75m_150m
Aug 23 19:10:29 jnano-desktop kernel: [95194.681337] tegradc tegradc.0: hdmi: get RGB quant from EDID.
Aug 23 19:10:29 jnano-desktop kernel: [95194.681344] tegradc tegradc.0: hdmi: get YCC quant from EDID.
Aug 23 19:10:29 jnano-desktop kernel: [95194.716215] extcon-disp-state extcon:disp-state: cable 47 state 1
Aug 23 19:10:29 jnano-desktop kernel: [95194.716217] Extcon AUX1(HDMI) enable
Aug 23 19:10:29 jnano-desktop kernel: [95194.716345] extcon-disp-state extcon:disp-state: cable 51 state 1
Aug 23 19:10:29 jnano-desktop kernel: [95194.716348] Extcon HDMI: HPD enabled
Aug 23 19:10:29 jnano-desktop kernel: [95194.716367] tegradc tegradc.0: hdmi: plugged
Aug 23 19:10:29 jnano-desktop kernel: [95194.736457] tegradc tegradc.0: blank - powerdown
Aug 23 19:10:29 jnano-desktop kernel: [95194.798227] extcon-disp-state extcon:disp-state: cable 47 state 0
Aug 23 19:10:29 jnano-desktop kernel: [95194.798229] Extcon AUX1(HDMI) disable
Aug 23 19:10:29 jnano-desktop kernel: [95194.818755] tegradc tegradc.0: unblank
Aug 23 19:10:29 jnano-desktop kernel: [95194.829271] tegradc tegradc.0: nominal-pclk:148500000 parent:148500000 div:1.0 pclk:148500000 147015000~161865000
Aug 23 19:10:29 jnano-desktop kernel: [95194.829344] tegradc tegradc.0: hdmi: tmds rate:148500K prod-setting:prod_c_hdmi_75m_150m
Aug 23 19:10:29 jnano-desktop kernel: [95194.830364] tegradc tegradc.0: hdmi: get RGB quant from EDID.
Aug 23 19:10:29 jnano-desktop kernel: [95194.830371] tegradc tegradc.0: hdmi: get YCC quant from EDID.
Aug 23 19:10:29 jnano-desktop kernel: [95194.865987] extcon-disp-state extcon:disp-state: cable 47 state 1
Aug 23 19:10:29 jnano-desktop kernel: [95194.865989] Extcon AUX1(HDMI) enable
Aug 23 19:10:29 jnano-desktop kernel: [95194.866549] tegradc tegradc.0: unblank
Aug 23 19:10:29 jnano-desktop kernel: [95194.866557] tegradc tegradc.1: blank - powerdown
Aug 23 19:10:30 jnano-desktop kernel: [95194.922884] tegradc tegradc.0: unblank
Aug 23 19:10:30 jnano-desktop kernel: [95194.923042] tegradc tegradc.1: blank - powerdown
Aug 23 19:12:20 jnano-desktop kernel: [95304.891839] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:50   [ERR]  PRI timeout: ADR 0x00400120 READ  DATA 0x00000000
Aug 23 19:12:20 jnano-desktop kernel: [95304.903609] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:56   [ERR]  FECS_ERRCODE 0xbadf1301
Aug 23 19:51:58 jnano-desktop kernel: [    0.000000] Booting Linux on physical CPU 0x0

24th of August, kern.log:

Aug 23 19:57:48 jnano-desktop kernel: [  368.032038] Extcon HDMI: HPD disabled
Aug 23 19:57:48 jnano-desktop kernel: [  368.032315] tegradc tegradc.0: hdmi: unplugged
Aug 23 19:57:48 jnano-desktop kernel: [  368.047702] tegradc tegradc.0: blank - powerdown
Aug 23 19:57:48 jnano-desktop kernel: [  368.047718] tegradc tegradc.0: unblank
Aug 23 19:57:48 jnano-desktop kernel: [  368.047752] tegradc tegradc.0: unblank
Aug 23 19:57:48 jnano-desktop kernel: [  368.047763] tegradc tegradc.1: blank - powerdown
Aug 23 19:57:48 jnano-desktop kernel: [  368.210433] tegradc tegradc.0: blank - powerdown
Aug 23 19:57:48 jnano-desktop kernel: [  368.210450] tegradc tegradc.0: unblank
Aug 23 19:57:48 jnano-desktop kernel: [  368.221087] tegradc tegradc.0: nominal-pclk:148500000 parent:148500000 div:1.0 pclk:148500000 147015000~161865000
Aug 23 19:57:48 jnano-desktop kernel: [  368.221178] tegradc tegradc.0: hdmi: tmds rate:148500K prod-setting:prod_c_hdmi_75m_150m
Aug 23 19:57:48 jnano-desktop kernel: [  368.222231] tegradc tegradc.0: hdmi: get RGB quant from EDID.
Aug 23 19:57:48 jnano-desktop kernel: [  368.222238] tegradc tegradc.0: hdmi: get YCC quant from EDID.
Aug 23 19:57:48 jnano-desktop kernel: [  368.256957] extcon-disp-state extcon:disp-state: cable 47 state 1
Aug 23 19:57:48 jnano-desktop kernel: [  368.256959] Extcon AUX1(HDMI) enable
Aug 23 19:57:48 jnano-desktop kernel: [  368.257088] extcon-disp-state extcon:disp-state: cable 51 state 1
Aug 23 19:57:48 jnano-desktop kernel: [  368.257091] Extcon HDMI: HPD enabled
Aug 23 19:57:48 jnano-desktop kernel: [  368.257110] tegradc tegradc.0: hdmi: plugged
Aug 23 19:57:48 jnano-desktop kernel: [  368.285198] tegradc tegradc.0: blank - powerdown
Aug 23 19:57:48 jnano-desktop kernel: [  368.339249] extcon-disp-state extcon:disp-state: cable 47 state 0
Aug 23 19:57:48 jnano-desktop kernel: [  368.339252] Extcon AUX1(HDMI) disable
Aug 23 19:57:48 jnano-desktop kernel: [  368.359687] tegradc tegradc.0: unblank
Aug 23 19:57:48 jnano-desktop kernel: [  368.370262] tegradc tegradc.0: nominal-pclk:148500000 parent:148500000 div:1.0 pclk:148500000 147015000~161865000
Aug 23 19:57:48 jnano-desktop kernel: [  368.370337] tegradc tegradc.0: hdmi: tmds rate:148500K prod-setting:prod_c_hdmi_75m_150m
Aug 23 19:57:48 jnano-desktop kernel: [  368.371322] tegradc tegradc.0: hdmi: get RGB quant from EDID.
Aug 23 19:57:48 jnano-desktop kernel: [  368.371330] tegradc tegradc.0: hdmi: get YCC quant from EDID.
Aug 23 19:57:48 jnano-desktop kernel: [  368.406159] extcon-disp-state extcon:disp-state: cable 47 state 1
Aug 23 19:57:48 jnano-desktop kernel: [  368.406161] Extcon AUX1(HDMI) enable
Aug 23 19:57:48 jnano-desktop kernel: [  368.406353] tegradc tegradc.0: unblank
Aug 23 19:57:48 jnano-desktop kernel: [  368.406361] tegradc tegradc.1: blank - powerdown
Aug 23 19:57:48 jnano-desktop kernel: [  368.449711] tegradc tegradc.0: unblank
Aug 23 19:57:48 jnano-desktop kernel: [  368.449722] tegradc tegradc.1: blank - powerdown
Aug 23 20:40:19 jnano-desktop kernel: [ 2919.385046] nvmap_alloc_handle: PID 8140: deepstream-test: WARNING: All NvMap Allocations must have a tag to identify the subsystem allocating memory.Please pass the tag to the API call NvRmMemHanldeAllocAttr() or relevant. 
Aug 23 20:44:23 jnano-desktop kernel: [ 3163.071088] nvmap_alloc_handle: PID 8581: deepstream-test: WARNING: All NvMap Allocations must have a tag to identify the subsystem allocating memory.Please pass the tag to the API call NvRmMemHanldeAllocAttr() or relevant. 
Aug 23 20:51:38 jnano-desktop kernel: [ 3597.837236] tegradc tegradc.0: blank - powerdown
Aug 23 20:51:38 jnano-desktop kernel: [ 3597.896846] extcon-disp-state extcon:disp-state: cable 47 state 0
Aug 23 20:51:38 jnano-desktop kernel: [ 3597.896854] Extcon AUX1(HDMI) disable
Aug 23 21:54:11 jnano-desktop kernel: [ 7350.815505] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:50   [ERR]  PRI timeout: ADR 0x00400120 READ  DATA 0x00000000
Aug 23 21:54:11 jnano-desktop kernel: [ 7350.827125] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:56   [ERR]  FECS_ERRCODE 0xbadf1301
Aug 24 00:55:37 jnano-desktop kernel: [18236.750629] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:50   [ERR]  PRI timeout: ADR 0x00400120 READ  DATA 0x00000000
Aug 24 00:55:37 jnano-desktop kernel: [18236.762376] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:56   [ERR]  FECS_ERRCODE 0xbadf1301
Aug 24 01:24:53 jnano-desktop kernel: [19992.919393] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:50   [ERR]  PRI timeout: ADR 0x00400120 READ  DATA 0x00000000
Aug 24 01:24:53 jnano-desktop kernel: [19992.931149] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:56   [ERR]  FECS_ERRCODE 0xbadf1301
Aug 24 03:05:01 jnano-desktop kernel: [26000.704886] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:50   [ERR]  PRI timeout: ADR 0x00400120 READ  DATA 0x00000000
Aug 24 03:05:01 jnano-desktop kernel: [26000.716523] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:56   [ERR]  FECS_ERRCODE 0xbadf1301
Aug 24 03:50:46 jnano-desktop kernel: [28745.698754] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:50   [ERR]  PRI timeout: ADR 0x00400120 READ  DATA 0x00000000
Aug 24 03:50:46 jnano-desktop kernel: [28745.710504] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:56   [ERR]  FECS_ERRCODE 0xbadf1301
Aug 24 05:07:18 jnano-desktop kernel: [    0.000000] Booting Linux on physical CPU 0x0

24th of August, serial log:

[15952.473434] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:50   [ERR]  PRI timeout: ADR 0x00400120 READ  DATA 0x00000000

[15952.485185] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:56   [ERR]  FECS_ERRCODE 0xbadf1301

[21466.007576] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:50   [ERR]  PRI timeout: ADR 0x00400120 READ  DATA 0x00000000

[21466.019349] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:56   [ERR]  FECS_ERRCODE 0xbadf1301

[23897.931833] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:50   [ERR]  PRI timeout: ADR 0x00400120 READ  DATA 0x00000000

[23897.943592] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:56   [ERR]  FECS_ERRCODE 0xbadf1301

[24864.196782] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:50   [ERR]  PRI timeout: ADR 0x00400120 READ  DATA 0x00000000

[24864.208677] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:56   [ERR]  FECS_ERRCODE 0xbadf1301

[26605.713341] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:50   [ERR]  PRI timeout: ADR 0x00400120 READ  DATA 0x00000000

[26605.725247] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:56   [ERR]  FECS_ERRCODE 0xbadf1301

[28471.093470] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:50   [ERR]  PRI timeout: ADR 0x00400120 READ  DATA 0x00000000

[28471.105101] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:56   [ERR]  FECS_ERRCODE 0xbadf1301

[33548.939731] ------------[ cut here ]------------

[33548.944551] WARNING: CPU: 1 PID: 4859 at /dvs/git/dirty/git-master_linux/kernel/nvgpu/drivers/gpu/nvgpu/gk20a/gk20a.c:64 __gk20a_warn_on_no_regs+0x34/0x50 [nvgpu]

[33548.960799] ---[ end trace 6731045601169df0 ]---

[33548.970369] nvgpu: 57000000.gpu           __nvgpu_check_gpu_state:56   [ERR]  GPU has disappeared from bus!!

[33548.980210] nvgpu: 57000000.gpu           __nvgpu_check_gpu_state:57   [ERR]  Rebooting system!!

[33548.990629] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 11141551 (offset 618496 size 4096 starting block 29526112)

[33549.003999] Buffer I/O error on device sda1, logical block 29525854

[33549.010260] Buffer I/O error on device sda1, logical block 29525855

[33549.016547] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 11141515 (offset 9150464 size 4096 starting block 29585851)

[33549.029996] Buffer I/O error on device sda1, logical block 29585593

[33549.036250] Buffer I/O error on device sda1, logical block 29585594

[33549.042868] JBD2: Detected IO errors while flushing file data on sda1-8

[33549.049576] Aborting journal on device sda1-8.

[33549.054354] JBD2: Error -5 detected when updating journal superblock for sda1-8.

[33549.056453] EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected aborted journal

[33549.056457] EXT4-fs (sda1): Remounting filesystem read-only

[33549.056462] EXT4-fs (sda1): previous I/O error to superblock detected

[33549.162549] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm (spawn): error -5 reading directory block

[33549.174818] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[33549.185662] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #529938: comm colord: reading directory lblock 0

[33549.185677] EXT4-fs (sda1): previous I/O error to superblock detected

[33549.201914] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #14155777: comm (umount): reading directory lblock 0

[33549.212764] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[33549.212767] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[33549.216156] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[33549.216159] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  compiz

[33549.216220] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[33549.216222] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  compiz

[33549.216569] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[33549.216571] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  compiz

[33549.231994] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[33549.231997] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[33549.252360] reboot: Restarting system

[0000.159] [L4T TegraBoot] (version 00.00.2018.01-l4t-80a468da)

[0000.165] Processing in cold boot mode Bootloader 2

26th of August, serial log:

[  128.097698] nvmap_alloc_handle: PID 4717: deepstream-test: WARNING: All NvMap Allocations must have a tag to identify the subsystem allocating memory.Please pass the tag to the API call NvRmMemHanldeAllocAttr() or relevant. 

[ 5533.578835] ------------[ cut here ]------------

[ 5533.585041] WARNING: CPU: 3 PID: 4918 at /dvs/git/dirty/git-master_linux/kernel/nvgpu/drivers/gpu/nvgpu/gk20a/gk20a.c:64 __gk20a_warn_on_no_regs+0x34/0x50 [nvgpu]

[ 5533.607280] ---[ end trace a8c2b4d37b753354 ]---

[ 5533.631065] nvgpu: 57000000.gpu           __nvgpu_check_gpu_state:56   [ERR]  GPU has disappeared from bus!!

[ 5533.640990] nvgpu: 57000000.gpu           __nvgpu_check_gpu_state:57   [ERR]  Rebooting system!!

[ 5533.652303] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 11141551 (offset 913408 size 4096 starting block 29526184)

[ 5533.665681] Buffer I/O error on device sda1, logical block 29525926

[ 5533.671956] Buffer I/O error on device sda1, logical block 29525927

[ 5533.678318] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 11141527 (offset 2703360 size 4096 starting block 29585301)

[ 5533.691789] Buffer I/O error on device sda1, logical block 29585043

[ 5533.698065] Buffer I/O error on device sda1, logical block 29585044

[ 5533.704731] JBD2: Detected IO errors while flushing file data on sda1-8

[ 5533.711476] Aborting journal on device sda1-8.

[ 5533.716028] JBD2: Error -5 detected when updating journal superblock for sda1-8.

[ 5533.718298] EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected aborted journal

[ 5533.718300] EXT4-fs (sda1): Remounting filesystem read-only

[ 5533.718305] EXT4-fs (sda1): previous I/O error to superblock detected

[ 5533.804221] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm (spawn): error -5 reading directory block

[ 5533.816642] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[ 5533.826443] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #14155777: comm (umount): reading directory lblock 0

[ 5533.828026] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[ 5533.828028] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[ 5533.839797] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[ 5533.839800] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  compiz

[ 5533.839855] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[ 5533.839858] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  compiz

[ 5533.840121] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[ 5533.840123] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  compiz

[ 5533.841617] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #11141128: comm python3: reading directory lblock 0

[ 5533.842262] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #2: comm python3: reading directory lblock 0

[ 5533.843270] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[ 5533.843273] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[ 5533.845248] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm unity-panel-ser: error -5 reading directory block

[ 5533.845272] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #11141128: comm unity-panel-ser: reading directory lblock 0

[ 5533.845290] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm unity-panel-ser: error -5 reading directory block

[ 5533.845316] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #524295: comm unity-panel-ser: reading directory lblock 0

[ 5533.845331] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm unity-panel-ser: error -5 reading directory block

[ 5533.845358] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm unity-panel-ser: error -5 reading directory block

[ 5533.845382] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #7340034: comm unity-panel-ser: reading directory lblock 0

[ 5533.845404] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #7340246: comm unity-panel-ser: reading directory lblock 0

[ 5533.845418] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm unity-panel-ser: error -5 reading directory block

[ 5533.845434] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm unity-panel-ser: error -5 reading directory block

[ 5533.845696] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #1313044: comm unity-panel-ser: reading directory lblock 0

[ 5533.846482] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #1836810: comm unity-panel-ser: reading directory lblock 0

[ 5533.846668] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #1705731: comm unity-panel-ser: reading directory lblock 0

[ 5533.879162] reboot: Restarting system

[0000.159] [L4T TegraBoot] (version 00.00.2018.01-l4t-80a468da)

1st September, serial log:

[  147.962522] nvmap_alloc_handle: PID 4722: deepstream-test: WARNING: All NvMap Allocations must have a tag to identify the subsystem allocating memory.Please pass the tag to the API call NvRmMemHanldeAllocAttr() or relevant. 
[22163.067150] ------------[ cut here ]------------
[22163.072812] WARNING: CPU: 2 PID: 4905 at /dvs/git/dirty/git-master_linux/kernel/nvgpu/drivers/gpu/nvgpu/gk20a/gk20a.c:64 __gk20a_warn_on_no_regs+0x34/0x50 [nvgpu]
[22163.093261] ---[ end trace 91b6de834d66ac90 ]---
[22163.100131] nvgpu: 57000000.gpu           __nvgpu_check_gpu_state:56   [ERR]  GPU has disappeared from bus!!
[22163.109978] nvgpu: 57000000.gpu           __nvgpu_check_gpu_state:57   [ERR]  Rebooting system!!
[22163.121761] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 11141527 (offset 4096 size 4096 starting block 44640094)
[22163.134959] Buffer I/O error on device sda1, logical block 44639836
[22163.141218] Buffer I/O error on device sda1, logical block 44639837
[22163.147506] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 11141571 (offset 180224 size 4096 starting block 9712)
[22163.160524] Buffer I/O error on device sda1, logical block 9454
[22163.166436] Buffer I/O error on device sda1, logical block 9455
[22163.172608] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 11141527 (offset 0 size 0 starting block 44640094)
[22163.185281] Buffer I/O error on device sda1, logical block 44639837
[22163.191766] JBD2: Detected IO errors while flushing file data on sda1-8
[22163.198515] Aborting journal on device sda1-8.
[22163.203030] JBD2: Error -5 detected when updating journal superblock for sda1-8.
[22163.203142] EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected aborted journal
[22163.203145] EXT4-fs (sda1): Remounting filesystem read-only
[22163.203150] EXT4-fs (sda1): previous I/O error to superblock detected
[22163.299469] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm (spawn): error -5 reading directory block
[22163.312052] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block
[22163.323953] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #529938: comm colord: reading directory lblock 0
[22163.323968] EXT4-fs (sda1): previous I/O error to superblock detected
[22163.331417] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #14155777: comm (umount): reading directory lblock 0
[22163.338942] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo
[22163.338946] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg
[22163.350039] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #7340034: comm bash: reading directory lblock 0
[22163.384784] nvgpu: 57000000.gpu             gk20a_channel_release:455  [ERR]  failed to release a channel!
[22163.384800] nvgpu: 57000000.gpu             gk20a_channel_release:455  [ERR]  failed to release a channel!
[22163.418723] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm gdm-session-wor: error -5 reading directory block
[22163.440653] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm gvfs-udisks2-vo: error -5 reading directory block
[22163.441591] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm gvfs-udisks2-vo: error -5 reading directory block
[22163.441614] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm gvfs-udisks2-vo: error -5 reading directory block
[22163.441630] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm gvfs-udisks2-vo: error -5 reading directory block
[22163.443379] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #2: comm pool: reading directory lblock 0
[22163.443490] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #2: comm pool: reading directory lblock 0
[22163.446133] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #14417921: comm gnome-session-b: reading directory lblock 0
[22163.447900] reboot: Restarting system
[0000.159] [L4T TegraBoot] (version 00.00.2018.01-l4t-80a468da)
[0000.165] Processing in cold boot mode Bootloader 2

2nd September, serial log:

[24229.694559] nvmap_alloc_handle: PID 14589: deepstream-test: WARNING: All NvMap Allocations must have a tag to identify the subsystem allocating memory.Please pass the tag to the API call NvRmMemHanldeAllocAttr() or relevant. 
[61309.937742] ------------[ cut here ]------------
[61309.943451] WARNING: CPU: 1 PID: 14783 at /dvs/git/dirty/git-master_linux/kernel/nvgpu/drivers/gpu/nvgpu/gk20a/gk20a.c:64 __gk20a_warn_on_no_regs+0x34/0x50 [nvgpu]
[61309.960582] ---[ end trace 110f25ae884fd831 ]---
[61309.972912] nvgpu: 57000000.gpu           __nvgpu_check_gpu_state:56   [ERR]  GPU has disappeared from bus!!
[61309.982846] nvgpu: 57000000.gpu           __nvgpu_check_gpu_state:57   [ERR]  Rebooting system!!
[61309.993645] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 11141527 (offset 327680 size 4096 starting block 44640173)
[61310.007023] Buffer I/O error on device sda1, logical block 44639915
[61310.013299] Buffer I/O error on device sda1, logical block 44639916
[61310.019656] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 11141571 (offset 299008 size 4096 starting block 9741)
[61310.032692] Buffer I/O error on device sda1, logical block 9483
[61310.038620] Buffer I/O error on device sda1, logical block 9484
[61310.045781] JBD2: Detected IO errors while flushing file data on sda1-8
[61310.052674] Aborting journal on device sda1-8.
[61310.057301] EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected aborted journal
[61310.057336] EXT4-fs (sda1): previous I/O error to superblock detected
[61310.057374] EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected aborted journal
[61310.057376] EXT4-fs (sda1): Remounting filesystem read-only
[61310.057380] EXT4-fs (sda1): previous I/O error to superblock detected
[61310.058187] JBD2: Error -5 detected when updating journal superblock for sda1-8.
[61310.100554] EXT4-fs (sda1): ext4_writepages: jbd2_start: 13312 pages, ino 11141527; err -30
[61310.179650] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm (spawn): error -5 reading directory block
[61310.191886] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block
[61310.204426] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block
[61310.205279] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #529938: comm colord: reading directory lblock 0
[61310.205293] EXT4-fs (sda1): previous I/O error to superblock detected
[61310.205312] ------------[ cut here ]------------
[61310.205322] WARNING: CPU: 2 PID: 21949 at /dvs/git/dirty/git-master_linux/kernel/kernel-4.9/fs/fs-writeback.c:2162 __mark_inode_dirty+0x188/0x4a0
[61310.205401] ---[ end trace 110f25ae884fd832 ]---
[61310.207406] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #14155777: comm (umount): reading directory lblock 0
[61310.207420] EXT4-fs (sda1): previous I/O error to superblock detected
[61310.207791] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo
[61310.207793] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg
[61310.208808] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo
[61310.208811] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  compiz
[61310.216526] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo
[61310.216529] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  compiz
[61310.216865] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo
[61310.216871] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  compiz
[61310.231329] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo
[61310.231332] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg
[61310.285936] reboot: Restarting system
[0000.159] [L4T TegraBoot] (version 00.00.2018.01-l4t-80a468da)
[0000.165] Processing in cold boot mode Bootloader 2

As I said, I also have tried to leave the Jetson Nano on for more than 48hours without any deepstream application running, that created NO crashes.

I hope that you could help me to solve this issue.

Thank you!!

WayneWWW · September 3, 2020, 6:27am

Hi,

This may not be resolved soon. We will check this internally.

Also, want to know if you have other jetson nano modules that can try this test too?

borelli.g92 · September 3, 2020, 6:31am

Hi @WayneWWW,

Unfortunately I have only one module.
Do you think that the error might be due to a faulty board?

Thanks again for the support, it is very much appreciated!

WayneWWW · September 3, 2020, 6:33am

Hi,

The test case seems a simple one, but so far we didn’t receive other people have such " GPU has disappeared from bus" error. Thus, I would like to check if device itself is the cause.

Anyway, we will use our device and run same sample here.

borelli.g92 · September 3, 2020, 9:46am

Thanks again @WayneWWW,

I want to highlight that I have found a similar error in the following post also:

The log is not highlighting the error of " GPU has disappeared from bus", but it is related to a long term usage and the error is also related to gk20a as well.

Your colleague @JerryChang suggested me to open a new topic in order to give the right focus to the issue.

Thank you!

borelli.g92 · September 7, 2020, 6:32pm

Hi @WayneWWW and @JerryChang,

I have bought a second Jetson Nano: the new model B01.
I previously had a Jetson Nano A02.

I tested the system in the exact same conditions. And the crash occurred after 21 hours of inference running (deepstream app test 3). Always with the same source: 1RTSP source 640x480 @15fps.

Here you can see the serial log:

[117860.885252] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[117860.897518] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[117860.906266] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[117987.591885] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[117987.608049] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[117987.616613] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[118570.239878] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[118570.251590] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[118570.260167] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[118931.695244] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[118931.706750] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[118931.715433] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[119748.697565] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[119748.708784] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[119748.717664] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[119850.635686] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[119850.646828] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[119850.655514] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[120150.036823] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[120150.048499] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[120150.057118] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[120466.894002] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[120466.904760] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[120466.913913] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[120559.016088] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[120559.027562] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[120559.036085] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[120658.211805] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[120658.223910] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[120658.232616] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[121346.928006] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[121346.940325] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[121346.949112] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[121490.238491] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[121490.249940] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[121490.258956] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[122454.480142] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[122454.496657] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[122454.507912] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[123525.636684] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[123525.647818] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[123525.656594] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[123570.484091] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[123570.495649] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[123570.504554] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[123901.755682] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[123901.766590] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[123901.775247] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[124062.525697] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[124062.537329] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[124062.545931] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[124333.747961] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[124333.762297] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[124333.771120] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[125608.144875] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[125608.160870] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[125608.170243] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[125666.708893] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[125666.719939] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[125666.728497] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[125906.522334] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[125906.533980] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[125906.542511] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[126262.932645] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[126262.943832] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[126262.952842] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[126333.815890] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[126333.827311] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[126333.836302] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[126793.284016] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[126793.301854] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[126793.310970] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[126822.608967] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[126822.619953] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[126822.628571] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[127092.894259] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[127092.905008] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[127092.913644] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[127501.667535] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[127501.683439] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[127501.692414] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[127570.532116] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[127570.543229] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[127570.552077] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[127797.469087] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[127797.480091] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[127797.488755] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[127934.781120] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[127934.794395] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[127934.803332] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[127990.977416] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[127990.988831] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[127990.997465] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[128237.794414] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[128237.805640] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[128237.814655] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[128846.523234] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[128846.534543] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[128846.543564] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[129592.679286] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[129592.690953] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[129592.700072] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[130369.642510] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[130369.657974] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[130369.666866] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[130905.947873] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[130905.960073] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[130905.968903] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[131266.053428] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[131266.067500] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[131266.076786] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[131389.944600] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[131389.960001] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[131389.968809] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[131421.838808] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[131421.850494] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[131421.859689] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[131471.574264] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[131471.585537] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[131471.594124] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[132895.681967] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[132895.692540] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[132895.701286] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[133071.120073] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[133071.131445] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[133071.140396] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[133667.616820] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[133667.627900] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[133667.639022] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[133789.850757] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[133789.862135] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[133789.871476] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[134779.698209] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[134779.709320] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[134779.717960] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[135682.013888] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[135682.025665] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[135682.034392] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[136481.775569] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[136481.786756] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[136481.795343] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[138673.437386] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[138673.448763] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[138673.457546] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[139384.278818] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[139384.289908] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[139384.298819] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[140166.520103] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[140166.534147] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[140166.545442] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[141016.342600] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[141016.354212] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[141016.362905] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[141290.298983] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[141290.310065] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[141290.318719] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[142641.391158] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[142641.402434] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[142641.411601] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[142841.604569] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[142841.615808] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[142841.624566] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[143376.654535] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[143376.664901] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[143376.673588] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[144008.055412] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[144008.066405] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[144008.075128] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[144088.193652] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[144088.206632] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[144088.215934] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[144273.538336] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[144273.549537] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[144273.558636] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[145848.042356] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[145848.052763] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[145848.061269] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[146043.973637] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[146043.985143] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[146043.993775] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[146659.566652] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[146659.579113] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[146659.587841] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[147159.289691] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[147159.301692] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[147159.310668] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[147673.529807] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[147673.543047] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[147673.552017] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[148091.811908] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[148091.828035] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[148091.837280] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[148590.541064] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[148590.553439] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[148590.562708] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[149329.373309] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[149329.383658] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[149329.392313] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[149558.800976] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[149558.811489] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[149558.820066] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[149632.343346] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[149632.353810] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[149632.362295] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[150133.876190] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[150133.890153] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[150133.899522] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[151022.767794] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[151022.779172] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[151022.787976] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[151239.310763] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[151239.322311] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[151239.331317] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[151749.464350] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[151749.478423] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[151749.487111] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[151815.764843] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[151815.775256] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[151815.783804] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[152195.615539] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[152195.626750] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[152195.635996] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[152201.898567] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[152201.908979] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[152201.917542] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[152964.747952] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[152964.759678] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[152964.768595] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[153338.352750] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[153338.363974] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[153338.372931] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[153387.314118] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[153387.325335] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[153387.333942] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[153843.185097] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[153843.196158] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[153843.204971] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[154392.657724] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[154392.668057] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[154392.676638] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[154669.399593] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[154669.410823] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[154669.419786] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[155438.413494] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[155438.425206] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[155438.433764] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[155492.806100] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[155492.817286] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[155492.826425] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[155596.476710] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[155596.487495] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[155596.496199] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[155800.320667] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[155800.332423] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[155800.341395] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[156113.909973] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[156113.920835] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[156113.929374] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[156598.258731] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[156598.269585] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[156598.278428] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[157493.574176] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[157493.585691] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[157493.594377] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[157720.051082] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[157720.062115] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[157720.070758] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[158300.129026] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[158300.140182] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[158300.149055] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[158811.147015] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[158811.158131] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[158811.166957] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[159017.681129] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[159017.693688] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[159017.702440] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[160085.014369] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[160085.025864] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[160085.034603] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[160352.164369] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[160352.175814] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[160352.184356] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[160614.521214] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[160614.538546] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[160614.547654] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[160705.263310] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[160705.276325] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[160705.286682] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[160872.009192] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[160872.020349] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[160872.029001] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[161172.176470] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[161172.187887] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[161172.196519] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[161213.209699] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[161213.220911] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[161213.229976] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[161557.623624] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[161557.635219] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[161557.650599] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[162530.066018] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[162530.084013] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[162530.092591] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[162780.015046] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[162780.028227] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[162780.037894] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[162835.404239] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[162835.415425] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[162835.424664] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[163169.469759] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[163169.481238] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[163169.490181] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[163356.858978] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[163356.870714] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[163356.879453] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[163401.090865] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[163401.102585] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[163401.111173] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[164000.619114] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[164000.630354] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[164000.639364] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[164491.453641] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[164491.465266] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[164491.473877] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[164506.013526] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[164506.024132] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[164506.032858] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[165623.813412] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[165623.826759] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[165623.836484] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[166045.054168] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[166045.064626] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[166045.073106] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[166786.416749] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[166786.428719] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[166786.437764] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[167111.870287] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[167111.881461] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[167111.890104] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[167371.928376] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[167371.942818] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[167371.951768] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[167525.344363] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[167525.354798] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[167525.363320] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[167543.776186] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[167543.787767] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[167543.796844] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[167717.555045] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[167717.565684] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[167717.574267] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[167773.198243] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[167773.209343] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[167773.217904] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[167815.290338] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[167815.301494] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[167815.310379] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[167887.645932] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[167887.657055] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[167887.665657] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[167952.834110] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[167952.845577] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[167952.854471] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[168021.687487] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[168021.698389] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[168021.707066] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[168312.732806] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[168312.744149] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[168312.752969] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[168410.891743] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[168410.903360] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[168410.912432] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[168516.516168] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[168516.527659] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[168516.536494] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[169191.769619] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[169191.787136] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[169191.795670] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[169286.216832] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[169286.228110] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[169286.236670] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[169348.269538] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:50   [ERR]  PRI timeout: ADR 0x00400120 READ  DATA 0x00000000

[169348.281401] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:56   [ERR]  FECS_ERRCODE 0xbadf1301

[169592.646278] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[169592.661005] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[169592.669708] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[169965.550081] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[169965.560391] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[169965.569150] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[170598.465653] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[170598.476774] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[170598.485474] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[170668.880110] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[170668.892049] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[170668.901454] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[170876.876478] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[170876.887949] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[170876.896836] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[172506.394144] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[172506.405443] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[172506.414325] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[172516.899350] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[172516.916192] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[172516.925496] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[172799.490662] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[172799.501917] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[172799.510793] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[172820.777567] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[172820.788798] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[172820.797961] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[173267.316650] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[173267.327974] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[173267.336650] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[174055.987586] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[174055.999191] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[174056.008591] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[174354.608503] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[174354.619879] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[174354.628517] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[174451.364090] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[174451.375368] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[174451.384417] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[174657.157798] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[174657.168275] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[174657.176840] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[175827.984902] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[175828.002475] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[175828.011571] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[176428.045698] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[176428.057397] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[176428.066180] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[176535.845239] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[176535.856836] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[176535.865869] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[177153.128441] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[177153.140803] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[177153.149568] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[177745.610556] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[177745.621557] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[177745.630220] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[177820.735735] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[177820.753273] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[177820.762233] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[177822.407987] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[177822.419169] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[177822.427860] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[177886.941081] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[177886.953640] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[177886.962889] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[178794.377178] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[178794.388424] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[178794.397697] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[178972.823207] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[178972.834689] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[178972.844286] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[179316.905910] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[179316.917214] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[179316.925828] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[179388.190284] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[179388.204164] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[179388.213349] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[179412.676257] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[179412.688007] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[179412.696966] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[179469.758319] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[179469.769225] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[179469.777973] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[179561.485071] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[179561.502090] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[179561.511426] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[179565.261629] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[179565.272638] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[179565.281334] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[179645.078188] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[179645.089818] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[179645.098477] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[179790.084012] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[179790.095670] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[179790.104200] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[179790.321601] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[179790.333144] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[179790.341990] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[180067.857363] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[180067.868541] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[180067.877651] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[180247.573925] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[180247.585070] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[180247.594117] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[180754.766190] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[180754.777446] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[180754.786188] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[180990.633995] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[180990.645325] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[180990.654326] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[181076.703322] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[181076.715012] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[181076.723913] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[181133.476401] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[181133.488172] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[181133.496811] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[181503.506163] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[181503.518277] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[181503.526902] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[182379.806628] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[182379.817954] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[182379.827219] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[182615.157168] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[182615.168905] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[182615.177861] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[182757.408959] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[182757.423816] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[182757.433472] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[182856.647429] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[182856.658568] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[182856.667275] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[183123.882195] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[183123.893035] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[183123.901721] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[183151.239057] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[183151.251255] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[183151.259867] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[183316.108282] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[183316.119701] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[183316.128433] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[183554.363441] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[183554.375525] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[183554.384103] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[183606.118376] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[183606.133381] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[183606.142075] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[184169.031476] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[184169.043296] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[184169.052146] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[184235.400471] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[184235.418015] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[184235.426573] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[184328.941476] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[184328.951913] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[184328.960489] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[184380.876046] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[184380.889026] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[184380.898431] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[184397.657942] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[184397.669294] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[184397.678074] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[184569.575030] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[184569.586592] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[184569.595741] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[184844.059309] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[184844.071920] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[184844.081013] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[184845.100459] EXT4-fs (mmcblk0p1): error count since last fsck: 6

[184845.106517] EXT4-fs (mmcblk0p1): initial error at time 1597765637: htree_dirblock_to_tree:991: inode 464593: block 1592264

[184845.117797] EXT4-fs (mmcblk0p1): last error at time 1597767749: htree_dirblock_to_tree:991: inode 285072: block 1059628

[185440.981368] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[185440.992684] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[185441.001534] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[185600.469463] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[185600.479815] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[185600.488304] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[186220.544501] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[186220.557299] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[186220.565953] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[186477.959584] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[186477.973293] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[186477.982468] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[186690.487576] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[186690.498416] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[186690.507780] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[186911.541413] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[186911.552840] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[186911.562205] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[186931.415083] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[186931.428430] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[186931.439428] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[187071.277171] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[187071.288220] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[187071.296818] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[187194.543992] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[187194.560521] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[187194.570094] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[187818.567353] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)



[187818.578908] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[187818.588020] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[188409.024060] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[188409.035589] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[188409.044229] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[188752.922429] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[188752.934903] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[188752.943599] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[189429.094133] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[189429.105534] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[189429.114337] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[189734.355433] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[189734.367767] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[189734.376710] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[189908.996009] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[189909.008206] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[189909.016932] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[189967.739371] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[189967.750622] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[189967.759606] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[190252.806631] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[190252.817793] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[190252.826465] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[190549.780899] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[190549.791403] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[190549.800478] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[190627.899648] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[190627.912196] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[190627.920800] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[191168.006513] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[191168.018099] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[191168.026759] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[192511.623023] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[192511.634386] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[192511.643620] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[193202.941460] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:50   [ERR]  PRI timeout: ADR 0x00400120 READ  DATA 0x00000000

[193202.953247] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:56   [ERR]  FECS_ERRCODE 0xbadf1301

[193403.625731] ------------[ cut here ]------------

[193403.632019] WARNING: CPU: 0 PID: 27361 at /dvs/git/dirty/git-master_linux/kernel/nvgpu/drivers/gpu/nvgpu/gk20a/gk20a.c:64 __gk20a_warn_on_no_regs+0x34/0x50 [nvgpu]

[193403.650473] ---[ end trace 751d547912e31108 ]---

[193403.667033] nvgpu: 57000000.gpu           __nvgpu_check_gpu_state:56   [ERR]  GPU has disappeared from bus!!

[193403.677647] nvgpu: 57000000.gpu           __nvgpu_check_gpu_state:57   [ERR]  Rebooting system!!

[193403.688827] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 11141555 (offset 151552 size 4096 starting block 2800870)

[193403.702203] Buffer I/O error on device sda1, logical block 2800612

[193403.708467] Buffer I/O error on device sda1, logical block 2800613

[193403.714804] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 11141515 (offset 46870528 size 4096 starting block 2814900)

[193403.728357] Buffer I/O error on device sda1, logical block 2814642

[193403.734641] Buffer I/O error on device sda1, logical block 2814643

[193403.741885] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 11141555 (offset 0 size 0 starting block 2800870)

[193403.754566] Buffer I/O error on device sda1, logical block 2800613

[193403.760878] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 11141515 (offset 0 size 0 starting block 2814900)

[193403.773557] Buffer I/O error on device sda1, logical block 2814643

[193403.780394] JBD2: Detected IO errors while flushing file data on sda1-8

[193403.787360] Aborting journal on device sda1-8.

[193403.790746] systemd: 31 output lines suppressed due to ratelimiting

[193403.792398] EXT4-fs (sda1): ext4_writepages: jbd2_start: 3072 pages, ino 11141515; err -30

[193403.792406] EXT4-fs (sda1): ext4_writepages: jbd2_start: 3072 pages, ino 11141515; err -30

[193403.792416] EXT4-fs (sda1): ext4_writepages: jbd2_start: 9223372036854775807 pages, ino 11141515; err -30

[193403.825155] JBD2: Error -5 detected when updating journal superblock for sda1-8.

[193403.855148] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #661491: comm async_44: reading directory lblock 0

[193403.896997] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #661475: comm async_44: reading directory lblock 0

[193403.907928] EXT4-fs (sda1): previous I/O error to superblock detected

[193403.914851] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #524295: comm async_44: reading directory lblock 0

[193403.925779] EXT4-fs (sda1): previous I/O error to superblock detected

[193403.933476] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #524295: comm async_44: reading directory lblock 0

[193403.944541] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #1441826: comm cron: reading directory lblock 0

[193403.944551] EXT4-fs (sda1): previous I/O error to superblock detected

[193403.961921] EXT4-fs (sda1): previous I/O error to superblock detected

[193403.970329] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm async_44: error -5 reading directory block

[193403.982619] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm async_44: error -5 reading directory block

[193403.996591] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #10747905: comm async_44: reading directory lblock 0

[193404.007690] EXT4-fs (sda1): previous I/O error to superblock detected

[193404.008201] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm (spawn): error -5 reading directory block

[193404.022551] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #529938: comm colord: reading directory lblock 0

[193404.036637] EXT4-fs (sda1): previous I/O error to superblock detected

[193404.037195] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #14155777: comm async_44: reading directory lblock 0

[193404.055336] EXT4-fs (sda1): previous I/O error to superblock detected

[193404.063010] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #2: comm async_44: reading directory lblock 0

[193404.115410] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm gvfs-udisks2-vo: error -5 reading directory block

[193404.130309] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #11141128: comm nautilus: reading directory lblock 0

[193404.130342] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nautilus: error -5 reading directory block

[193404.130377] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nautilus: error -5 reading directory block

[193404.217643] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[193404.228981] reboot: Restarting system

\00[0000.159] [L4T TegraBoot] (version 00.00.2018.01-l4t-80a468da)

There are a number of errors in the log. It is always the same error repeated over and over.
However, at the end the crash is anticipated by the same error:

[193202.941460] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:50   [ERR]  PRI timeout: ADR 0x00400120 READ  DATA 0x00000000

[193202.953247] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:56   [ERR]  FECS_ERRCODE 0xbadf1301

Do you think that this might be related to some overheating of one component that is not monitored by a sensor and that is not connected to the heatsink?
Or is it some software issue?

Thanks,
Gabriele

WayneWWW · September 8, 2020, 2:11am

Hi,

tegrastats should tell you the thermal status of your board. But you need to run this along when you run the deepstream test.

I don’t get why there is pcie error showing up. Do you connect any pcie device on your board?

borelli.g92 · September 8, 2020, 5:20am

Hi @WayneWWW,

tonight I had the same error after 7 hours. Here follows the serial log:

[19895.643751] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[19895.657718] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[19895.666546] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[19967.070178] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[19967.081558] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[19967.090293] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20113.420838] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20113.432191] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20113.441362] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20337.885381] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20337.896445] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20337.905476] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20394.005901] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20394.016882] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20394.025391] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20436.562193] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20436.573369] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20436.581875] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20601.128126] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20601.140846] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20601.149737] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20609.563844] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20609.574880] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20609.583452] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20623.216156] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20623.227260] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20623.235792] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20627.889885] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20627.902719] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20627.911360] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20646.414077] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20646.425111] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20646.433642] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20650.853661] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20650.864740] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20650.873225] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20712.110019] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20712.123946] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20712.132641] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20737.383190] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20737.394208] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20737.402822] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20814.859116] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20814.870356] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20814.878871] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20856.500992] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20856.511655] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20856.520124] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20859.661274] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20859.672300] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20859.680759] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20875.463048] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20875.474139] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20875.482630] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[20943.275652] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[20943.286228] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[20943.294686] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[21017.724579] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[21017.735393] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[21017.744013] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[21268.873526] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[21268.885171] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[21268.893956] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[21336.806656] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[21336.817667] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[21336.826669] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[21350.308284] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[21350.323485] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[21350.331980] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[21553.286841] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[21553.298281] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[21553.307146] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[21620.458593] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[21620.469777] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[21620.478536] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[21623.117735] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[21623.128780] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[21623.137647] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[21798.136130] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[21798.147115] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[21798.155540] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[21825.029318] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[21825.041150] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[21825.049923] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[21943.724538] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[21943.735621] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[21943.745801] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22017.638721] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22017.651203] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22017.659651] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22048.906015] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22048.916998] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22048.925437] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22096.287752] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22096.299108] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22096.308193] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22142.402718] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22142.414484] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22142.423455] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22222.976181] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22222.986686] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22222.995458] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22285.638288] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22285.648980] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22285.657602] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22363.533439] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22363.544406] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22363.553000] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22472.910071] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22472.923586] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22472.932313] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22476.878569] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22476.889346] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22476.897923] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22534.932666] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22534.943565] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22534.952190] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22535.239268] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22535.250359] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22535.258966] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22582.338843] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22582.349773] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22582.358321] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22612.605589] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22612.617235] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22612.626005] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22633.681205] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:50   [ERR]  PRI timeout: ADR 0x00400120 READ  DATA 0x00000000

[22633.692959] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:56   [ERR]  FECS_ERRCODE 0xbadf1301

[22634.209946] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22634.221021] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22634.229829] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22748.873900] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22748.885048] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22748.893783] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22767.966560] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22767.977587] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22767.986313] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22796.081647] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22796.092542] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22796.101052] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[22840.761618] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[22840.772792] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[22840.781336] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23050.876161] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23050.887989] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23050.896820] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23205.761907] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23205.773056] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23205.781954] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23221.169800] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23221.180872] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23221.190058] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23297.627877] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23297.638882] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23297.648042] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23463.272601] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23463.284023] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23463.292859] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23539.818161] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23539.829072] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23539.838880] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23544.841808] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23544.854952] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23544.863461] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23599.592087] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23599.603525] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23599.612667] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23721.272453] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23721.282825] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23721.291600] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23772.462208] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23772.474319] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23772.482981] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23795.919840] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23795.930915] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23795.939834] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23806.951656] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23806.962228] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23806.970838] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23911.509936] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23911.521767] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23911.530337] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23912.803075] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23912.814091] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23912.822603] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23946.603289] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23946.614002] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23946.622461] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[23959.981979] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[23959.992954] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[23960.002147] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[24137.527837] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[24137.538907] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[24137.547523] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[24160.803156] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[24160.814258] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[24160.822906] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[24359.482377] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[24359.492799] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[24359.501277] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[24407.506691] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[24407.518259] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[24407.527474] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[24448.070773] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[24448.081712] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[24448.090248] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[24487.236197] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[24487.247284] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[24487.255766] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[24577.562260] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[24577.573299] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[24577.581928] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[24638.089415] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[24638.100514] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[24638.109429] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[24759.240072] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:50   [ERR]  PRI timeout: ADR 0x00400120 READ  DATA 0x00000000

[24759.251904] nvgpu: 57000000.gpu                  gk20a_ptimer_isr:56   [ERR]  FECS_ERRCODE 0xbadf1301

[24858.184744] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[24858.195809] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[24858.204361] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[24994.576175] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[24994.587415] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[24994.596744] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25003.014188] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25003.025469] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25003.034343] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25009.784653] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25009.796066] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25009.805024] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25014.454527] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25014.466678] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25014.475389] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25186.937706] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25186.948844] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25186.957724] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25328.368552] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25328.379598] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25328.388106] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25492.535332] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25492.547064] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25492.556352] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25530.912889] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25530.924110] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25530.932663] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25620.181333] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25620.192520] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25620.201240] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25782.984575] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25782.996372] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25783.005046] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[25994.633581] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[25994.644465] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[25994.653090] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[26032.988942] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[26033.000045] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[26033.008542] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[26048.938233] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[26048.949761] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[26048.958178] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[26081.909220] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[26081.920202] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[26081.929239] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[26280.912848] pcieport 0000:00:02.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=0010(Receiver ID)

[26280.923952] pcieport 0000:00:02.0:   device [10de:0faf] error status/mask=00000001/00002000

[26280.932591] pcieport 0000:00:02.0:    [ 0] Receiver Error         (First)

[26291.918776] ------------[ cut here ]------------

[26291.923667] WARNING: CPU: 0 PID: 6702 at /dvs/git/dirty/git-master_linux/kernel/nvgpu/drivers/gpu/nvgpu/gk20a/gk20a.c:64 __gk20a_warn_on_no_regs+0x34/0x50 [nvgpu]

[26291.938819] ---[ end trace bd264815883b0175 ]---

[26291.945132] nvgpu: 57000000.gpu           __nvgpu_check_gpu_state:56   [ERR]  GPU has disappeared from bus!!

[26291.955032] nvgpu: 57000000.gpu           __nvgpu_check_gpu_state:57   [ERR]  Rebooting system!!

[26291.965318] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 11141527 (offset 106496 size 4096 starting block 2800859)

[26291.978599] Buffer I/O error on device sda1, logical block 2800601

[26291.984769] Buffer I/O error on device sda1, logical block 2800602

[26291.990976] EXT4-fs warning (device sda1): ext4_end_bio:313: I/O error -5 writing to inode 11141555 (offset 413696 size 4096 starting block 2815846)

[26292.004251] Buffer I/O error on device sda1, logical block 2815588

[26292.010419] Buffer I/O error on device sda1, logical block 2815589

[26292.017489] JBD2: Detected IO errors while flushing file data on sda1-8

[26292.024189] Aborting journal on device sda1-8.

[26292.028684] JBD2: Error -5 detected when updating journal superblock for sda1-8.

[26292.028824] EXT4-fs error (device sda1): ext4_journal_check_start:56: Detected aborted journal

[26292.028827] EXT4-fs (sda1): Remounting filesystem read-only

[26292.028832] EXT4-fs (sda1): previous I/O error to superblock detected

[26292.163632] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm (spawn): error -5 reading directory block

[26292.175976] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26292.189744] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26292.189747] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[26292.193165] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26292.193169] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  compiz

[26292.193507] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26292.193510] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  compiz

[26292.195329] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #14155777: comm (umount): reading directory lblock 0

[26292.199948] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #529938: comm colord: reading directory lblock 0

[26292.203662] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #11141128: comm python3: reading directory lblock 0

[26292.204224] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #2: comm python3: reading directory lblock 0

[26292.209443] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26292.209446] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[26292.209899] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm unity-panel-ser: error -5 reading directory block

[26292.209923] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #11141128: comm unity-panel-ser: reading directory lblock 0

[26292.209939] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm unity-panel-ser: error -5 reading directory block

[26292.209959] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #524295: comm unity-panel-ser: reading directory lblock 0

[26292.209973] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm unity-panel-ser: error -5 reading directory block

[26292.209999] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm unity-panel-ser: error -5 reading directory block

[26292.210022] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #7340034: comm unity-panel-ser: reading directory lblock 0

[26292.210043] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #7340246: comm unity-panel-ser: reading directory lblock 0

[26292.210057] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm unity-panel-ser: error -5 reading directory block

[26292.210072] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm unity-panel-ser: error -5 reading directory block

[26292.210333] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #1313044: comm unity-panel-ser: reading directory lblock 0

[26292.211169] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #1836810: comm unity-panel-ser: reading directory lblock 0

[26292.331175] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26292.331179] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[26292.333768] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26292.333771] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  compiz

[26292.333939] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26292.333942] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  compiz

[26292.333949] nvgpu: 57000000.gpu         nvgpu_ioctl_tsg_dev_ioctl:656  [ERR]  failed to host gk20a for ioctl cmd: 0x40045402

[26292.333962] nvgpu: 57000000.gpu             gk20a_channel_release:455  [ERR]  failed to release a channel!

[26292.344634] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26292.344637] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[26292.349695] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26292.349698] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[26292.350307] ------------[ cut here ]------------

[26292.350319] WARNING: CPU: 0 PID: 4089 at /dvs/git/dirty/git-master_linux/kernel/kernel-4.9/fs/fs-writeback.c:2162 __mark_inode_dirty+0x188/0x4a0

[26292.350398] ---[ end trace bd264815883b0176 ]---

[26292.355602] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26292.355605] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[26292.355618] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26292.355620] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[26297.535512] EXT4-fs warning: 10026 callbacks suppressed

[26297.535517] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm gmain: error -5 reading directory block

[26298.224631] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26298.257250] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26298.266697] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm NetworkManager: error -5 reading directory block

[26298.268191] ------------[ cut here ]------------

[26298.268202] WARNING: CPU: 1 PID: 4459 at /dvs/git/dirty/git-master_linux/kernel/kernel-4.9/fs/fs-writeback.c:2162 __mark_inode_dirty+0x188/0x4a0

[26298.268285] ---[ end trace bd264815883b0177 ]---

[26298.272108] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm dbus-daemon: error -5 reading directory block

[26298.316542] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm gnome-software: error -5 reading directory block

[26298.332144] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm NetworkManager: error -5 reading directory block

[26298.344504] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26298.344531] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-resolve: error -5 reading directory block

[26298.345910] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-resolve: error -5 reading directory block

[26298.357764] EXT4-fs error: 47 callbacks suppressed

[26298.357770] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #10747905: comm systemd: reading directory lblock 0

[26298.360136] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #11141179: comm NetworkManager: reading directory lblock 0

[26298.360185] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #11141179: comm NetworkManager: reading directory lblock 0

[26298.372113] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #14417921: comm dbus-daemon: reading directory lblock 0

[26298.403754] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #524294: comm update-manager: reading directory lblock 0

[26299.765830] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #1313090: comm update-manager: reading directory lblock 0

[26299.777733] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #2624237: comm update-manager: reading directory lblock 0

[26299.789331] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #1313090: comm update-manager: reading directory lblock 0

[26299.800862] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #2624152: comm update-manager: reading directory lblock 0

[26302.539441] EXT4-fs warning: 179779 callbacks suppressed

[26302.539448] EXT4-fs warning (device sda1): dx_probe:743: inode #11142514: lblock 0: comm update-manager: error -5 reading directory block

[26302.557425] EXT4-fs warning (device sda1): dx_probe:743: inode #11142514: lblock 0: comm update-manager: error -5 reading directory block

[26302.569798] EXT4-fs warning (device sda1): dx_probe:743: inode #11142514: lblock 0: comm update-manager: error -5 reading directory block

[26302.582289] EXT4-fs warning (device sda1): dx_probe:743: inode #11142514: lblock 0: comm update-manager: error -5 reading directory block

[26302.594680] EXT4-fs warning (device sda1): dx_probe:743: inode #11142514: lblock 0: comm update-manager: error -5 reading directory block

[26302.607088] EXT4-fs warning (device sda1): dx_probe:743: inode #11142514: lblock 0: comm update-manager: error -5 reading directory block

[26302.619450] EXT4-fs warning (device sda1): dx_probe:743: inode #11142514: lblock 0: comm update-manager: error -5 reading directory block

[26302.631812] EXT4-fs warning (device sda1): dx_probe:743: inode #11142514: lblock 0: comm update-manager: error -5 reading directory block

[26302.644173] EXT4-fs warning (device sda1): dx_probe:743: inode #11142514: lblock 0: comm update-manager: error -5 reading directory block

[26302.656534] EXT4-fs warning (device sda1): dx_probe:743: inode #11142514: lblock 0: comm update-manager: error -5 reading directory block

[26308.476450] EXT4-fs warning: 11796 callbacks suppressed

[26308.476455] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm gmain: error -5 reading directory block

[26308.493102] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm gmain: error -5 reading directory block

[26308.504510] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm gmain: error -5 reading directory block

[26309.535291] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm gmain: error -5 reading directory block

[26310.665757] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26310.665984] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26310.666003] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26310.666020] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #10747905: comm nvmemwarning.sh: reading directory lblock 0

[26310.666039] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #14155777: comm nvmemwarning.sh: reading directory lblock 0

[26310.666139] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #1313090: comm nvmemwarning.sh: reading directory lblock 0

[26310.666220] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #1313090: comm nvmemwarning.sh: reading directory lblock 0

[26310.666237] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #2624237: comm nvmemwarning.sh: reading directory lblock 0

[26310.666256] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #1313090: comm nvmemwarning.sh: reading directory lblock 0

[26310.666272] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #2624152: comm nvmemwarning.sh: reading directory lblock 0

[26310.666289] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26310.666304] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26310.666317] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26310.666948] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #14417940: comm systemd-udevd: reading directory lblock 0

[26310.666976] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #1048701: comm systemd-udevd: reading directory lblock 0

[26310.667004] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #524294: comm systemd-udevd: reading directory lblock 0

[26312.357145] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26312.367612] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[26312.375282] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26312.385746] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[26313.479068] EXT4-fs warning: 9011 callbacks suppressed

[26313.479074] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26313.496581] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26313.508875] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26313.521155] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26313.533453] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26313.545786] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26313.545790] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm gmain: error -5 reading directory block

[26313.569736] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26313.571574] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26313.571589] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26318.483547] EXT4-fs warning: 17178 callbacks suppressed

[26318.483553] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26318.501083] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26318.513374] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26318.525697] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26318.537992] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26318.550276] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26318.562827] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26318.576017] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26318.576034] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26318.576298] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26322.394013] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26322.404465] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[26323.281227] EXT4-fs error: 10 callbacks suppressed

[26323.281233] EXT4-fs error (device sda1): ext4_find_entry:1441: inode #11145572: comm cron: reading directory lblock 0

[26323.486906] EXT4-fs warning: 17149 callbacks suppressed

[26323.486912] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26323.504433] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26323.516718] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26323.528995] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26323.541363] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26323.553681] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26323.566134] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26323.567684] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26323.567700] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26323.578937] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26328.490697] EXT4-fs warning: 17462 callbacks suppressed

[26328.490704] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26328.493036] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26328.493053] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26328.508663] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26328.508679] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26328.510800] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26328.510815] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26328.510828] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26328.510841] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26328.510854] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26332.412790] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26332.423247] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[26333.495090] EXT4-fs warning: 17465 callbacks suppressed

[26333.495097] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26333.495738] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26333.495753] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26333.495768] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26333.495781] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26333.495796] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26333.495809] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26333.497301] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26333.497317] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26333.497574] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26338.498801] EXT4-fs warning: 17558 callbacks suppressed

[26338.498807] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26338.516903] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26338.516919] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26338.516933] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26338.516946] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26338.516960] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26338.516973] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26338.518724] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26338.518741] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26338.518999] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26342.431543] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26342.442005] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[26343.502480] EXT4-fs warning: 17454 callbacks suppressed

[26343.502486] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26343.520030] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26343.532468] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26343.544756] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26343.557044] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26343.569318] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26343.581766] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26343.583578] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26343.583594] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26343.594596] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26348.507250] EXT4-fs warning: 17338 callbacks suppressed

[26348.507256] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26348.507462] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26348.507478] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26348.507672] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26348.561729] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26348.574735] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26348.587960] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26348.601132] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26348.613434] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26348.625712] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26352.450683] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26352.461146] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[26353.510292] EXT4-fs warning: 17126 callbacks suppressed

[26353.510299] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26353.527848] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26353.534108] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm gmain: error -5 reading directory block

[26353.552908] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26353.565185] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26353.577463] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26353.589737] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26353.602223] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26353.615462] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26353.615901] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26358.515024] EXT4-fs warning: 17267 callbacks suppressed

[26358.515031] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26358.515674] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26358.515691] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26358.515705] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26358.515719] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26358.515732] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26358.515745] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26358.517279] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26358.517295] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26358.517587] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26362.469007] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26362.479470] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[26363.517839] EXT4-fs warning: 17607 callbacks suppressed

[26363.517846] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26363.517915] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26363.517929] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26363.536956] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26363.536971] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26363.536984] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26363.536997] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26363.537010] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26363.537023] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26363.538598] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26368.522558] EXT4-fs warning: 17608 callbacks suppressed

[26368.522565] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26368.522789] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26368.522806] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26368.523003] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26368.577024] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26368.590467] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26368.591557] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26368.591573] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26368.591588] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26368.591602] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26372.487719] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26372.498180] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[26373.526880] EXT4-fs warning: 17194 callbacks suppressed

[26373.526886] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26373.527102] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26373.527118] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26373.527316] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26373.533666] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm gmain: error -5 reading directory block

[26373.592772] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26373.605825] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26373.618698] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26373.618959] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26373.618975] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26378.529892] EXT4-fs warning: 17326 callbacks suppressed

[26378.529898] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26378.530544] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26378.530561] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26378.530575] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26378.530589] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26378.530602] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26378.530615] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26378.532102] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26378.532119] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26378.532387] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26382.506485] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26382.516948] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[26383.533559] EXT4-fs warning: 17607 callbacks suppressed

[26383.533566] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26383.534219] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26383.534236] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26383.534251] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26383.534264] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26383.534278] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26383.534291] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26383.535807] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26383.535823] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26383.537652] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26388.537880] EXT4-fs warning: 17738 callbacks suppressed

[26388.537887] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26388.555475] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26388.567755] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26388.580044] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26388.592317] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26388.604609] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26388.617016] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26388.630344] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26388.630361] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26388.630629] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26392.525262] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26392.535721] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[26393.541191] EXT4-fs warning: 17293 callbacks suppressed

[26393.541197] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26393.558714] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26393.570993] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26393.583330] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26393.595610] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26393.607881] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26393.620285] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26393.621927] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26393.621943] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26393.633057] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26398.546059] EXT4-fs warning: 17302 callbacks suppressed

[26398.546065] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26398.546292] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26398.546307] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26398.546508] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26398.600567] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26398.613611] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26398.626884] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26398.627582] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26398.627598] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26398.627612] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26402.543993] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26402.554454] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[26403.549384] EXT4-fs warning: 17195 callbacks suppressed

[26403.549391] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26403.550032] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26403.550049] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26403.550063] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26403.550076] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26403.550089] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26403.550101] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26403.551590] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26403.551606] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26403.552408] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26408.553765] EXT4-fs warning: 17670 callbacks suppressed

[26408.553772] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26408.553998] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26408.554014] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26408.554206] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26408.608208] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26408.621211] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26408.633936] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26408.634203] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26408.634218] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26408.634231] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26412.562812] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26412.573263] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[26413.556678] EXT4-fs warning: 17352 callbacks suppressed

[26413.556685] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26413.574215] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26413.586543] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26413.598825] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26413.611102] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26413.623399] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26413.635812] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26413.637620] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26413.637636] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26413.648688] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26418.561938] EXT4-fs warning: 18596 callbacks suppressed

[26418.561944] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26418.562160] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26418.562176] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26418.562381] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26418.616356] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26418.629328] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26418.642587] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26418.642690] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26418.642706] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26418.642719] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26422.581521] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26422.591978] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[26423.565753] EXT4-fs warning: 17346 callbacks suppressed

[26423.565759] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26423.565966] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26423.565983] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26423.566183] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26423.620155] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26423.633077] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26423.646338] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26423.646556] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26423.646572] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26423.646586] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26428.568592] EXT4-fs warning: 17052 callbacks suppressed

[26428.568597] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26428.586150] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26428.598429] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26428.610736] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26428.623013] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26428.635297] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26428.647702] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26428.649510] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26428.649526] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26428.660517] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26432.600300] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26432.610768] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[26433.572440] EXT4-fs warning: 17251 callbacks suppressed

[26433.572446] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26433.572635] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26433.572651] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26433.572851] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26433.626879] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26433.639812] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26433.641423] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26433.641440] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26433.641453] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26433.641468] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26438.576169] EXT4-fs warning: 17281 callbacks suppressed

[26438.576175] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26438.576447] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26438.576463] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26438.576653] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26438.630648] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26438.643599] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26438.656473] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26438.656620] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26438.656636] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26438.656650] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26442.619031] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26442.629491] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[26443.580194] EXT4-fs warning: 17299 callbacks suppressed

[26443.580200] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26443.580853] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26443.580870] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26443.580883] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26443.580897] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26443.580910] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26443.580923] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26443.582423] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26443.582440] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26443.582707] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26448.584612] EXT4-fs warning: 17660 callbacks suppressed

[26448.584619] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26448.585266] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26448.585282] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26448.585296] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26448.585310] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26448.585323] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26448.585336] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26448.586876] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26448.586892] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26448.587184] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26452.637808] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26452.648269] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[26453.587827] EXT4-fs warning: 17641 callbacks suppressed

[26453.587834] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26453.588072] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26453.588087] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26453.588286] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26453.642244] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26453.655222] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26453.668234] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26453.668381] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26453.668397] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26453.668410] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26458.592255] EXT4-fs warning: 17369 callbacks suppressed

[26458.592262] EXT4-fs warning (device sda1): dx_probe:743: inode #1048577: lblock 0: comm systemd-journal: error -5 reading directory block

[26458.592896] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26458.592912] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26458.592926] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26458.592940] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26458.592954] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26458.592967] EXT4-fs warning (device sda1): dx_probe:743: inode #524298: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26458.594466] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26458.594482] EXT4-fs warning (device sda1): dx_probe:743: inode #524291: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26458.594749] EXT4-fs warning (device sda1): dx_probe:743: inode #524297: lblock 0: comm nvmemwarning.sh: error -5 reading directory block

[26462.656598] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:463  [ERR]  failed to host gk20a to submit gpfifo

[26462.667051] nvgpu: 57000000.gpu       nvgpu_submit_channel_gpfifo:464  [ERR]  Xorg

[26462.727555] INFO: task deepstream-test:6679 blocked for more than 120 seconds.

[26462.734822]       Tainted: G        W       4.9.140-tegra #1

[26462.740509] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

[26462.751247] Kernel panic - not syncing: hung_task: blocked tasks

[26462.757249] CPU: 0 PID: 584 Comm: khungtaskd Tainted: G        W       4.9.140-tegra #1

[26462.765236] Hardware name: NVIDIA Jetson Nano Developer Kit (DT)

[26462.771229] Call trace:

[26462.773671] [<ffffff800808bdb8>] dump_backtrace+0x0/0x198

[26462.779059] [<ffffff800808c37c>] show_stack+0x24/0x30

[26462.784101] [<ffffff800845c7a0>] dump_stack+0x98/0xc0

[26462.789144] [<ffffff80081c1438>] panic+0x11c/0x298

[26462.793924] [<ffffff8008181240>] watchdog+0x300/0x3b8

[26462.798965] [<ffffff80080dbe64>] kthread+0xec/0xf0

[26462.803744] [<ffffff80080838a0>] ret_from_fork+0x10/0x30

[26462.809054] Kernel Offset: disabled

[26462.812534] Memory Limit: none

[26462.821357] Rebooting in 5 seconds..

\00[0000.159] [L4T TegraBoot] (version 00.00.2018.01-l4t-80a468da)

[0000.165] Processing in cold boot mode Bootloader 2

I am also monitoring the temperature in a chart by readgin tegrastats with a python script. The temperature of A0 is always in the range of 38-45 celsius (it changes according to room temperature also).

I do not have any additional device connected to the board.

I hope that you could help me in solving this.

Thanks again!!

WayneWWW · September 8, 2020, 5:46am

Actually we already tried the same setup locally but cannot reproduce any issue.

It sounds like your device has some defect. Maybe you can consider RMA it and get a new one.

The error you have is not only gpu but also pcie and file system.

If you still want to check it, maybe you can check what application would reproduce this issue faster and efficiently.

borelli.g92 · September 8, 2020, 5:59am

Hi @WayneWWW,

I have bought a NEW one a couple of days ago.
Now I tried on two different devices: an A02 model and a B01 model.

The results are the same as you can see from the log.
The previous logs are related to the NEW device.

The application that I use to reproduce the error is the stock “deepstream-test3”, I modified nothing.
Have you had the chance to leave the Jetson Nano with deepstream-test3 running for 20 hours or more?

Thanks again for your support @WayneWWW, it is very much appreciated!!!

WayneWWW · September 8, 2020, 6:30am

Hi borelli.g92,

Are these two using same SDcard?

Our test was based on emmc (production) module. We will try A02 sdcard module later.

borelli.g92 · September 8, 2020, 8:00am

Hi @WayneWWW,

Yes, it is the exact same card.
I have tried both with running the module from the SD card, and also booting the Jetson Nano from a USB 3.0 SSD 240GB drive.
AS explained here: Jetson Nano - Run From USB Drive - JetsonHacks

During my first tests I thought that the problem was due to the SD card that I know to be quite delicate. However, the same happens with booting from sda1; where sda1 is an SSD drive as I just mentioned.

Thanks again!!!

WayneWWW · September 8, 2020, 8:09am

Hi borelli.g92,

Could you try to remove the external storage and see if you can reproduce issue when booting from sdcard only?

borelli.g92 · September 8, 2020, 8:19am

Hi @WayneWWW,

yes, booting up from SD card was my starting point.
I have the same error there as well.
If you look in my previous message, 23rd of August log is referring to SD card.
Also I had a number of attempts prior to that with the SD card only.
I also had to repair the filesystem of the SD card a couple of time because of the reboots that were corrupting it.

I have tried both booting from SD card and booting from USB SSD.

Thanks again!

WayneWWW · September 8, 2020, 8:23am

And do you have other sdcard to do the test? I mean are you using only one sdcard on either A02 or B01?

borelli.g92 · September 8, 2020, 8:30am

Hi @WayneWWW,
during my tests before 23rd of August I have tested 2 different SD cards on module A02. Same result.
However, I do not understand how SD card can be the problem. It happens with both:

booting from SD cards
booting from USB3.0 SSD drive

I believe that the origin of the error should be somewhere else right?
Thanks again

WayneWWW · September 8, 2020, 8:36am

Hi,

I just hope we don’t get issues altogether. Could you point out one log that is pure sdcard based image without ssd?

According to your log during 8/24, some logs only have gk20a error. But suddenly, sda1 appeared so obviously it is log with booting from ssd.

borelli.g92 · September 8, 2020, 8:50am

Hi @WayneWWW,

Sure, the sull story of SD experiments and corruptions is here:

Here you can see the errors that were repeated in the logs.
However, at that time I was not logging with the serial monitor, thus kern.log is only available. The behaviour has not changed since then:

Thanks