How to deal with the RmInitAdapter failed? The GPU always fell off the bus after that occurred.

The log is following:
Nov 15 16:19:18 dell-PowerEdge-T640 kernel: [ 759.389921] NVRM: GPU at PCI:0000:b1:00: GPU-2c48c2eb-cdc7-02bf-ac64-aa3ef8c88e96
Nov 15 16:19:18 dell-PowerEdge-T640 kernel: [ 759.389924] NVRM: GPU Board Serial Number: 0324218061405
Nov 15 16:19:18 dell-PowerEdge-T640 kernel: [ 759.389930] NVRM: Xid (PCI:0000:b1:00): 61, pid=4354, 0cde(308c) 00000000 00000000
Nov 15 16:19:18 dell-PowerEdge-T640 kernel: [ 759.407084] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 0, TPC 0, SM 0): Out Of Range Register
Nov 15 16:19:18 dell-PowerEdge-T640 kernel: [ 759.409649] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x504730=0x4000d 0x504734=0x0 0x504728=0x4c1eb72 0x50472c=0x174
Nov 15 16:19:18 dell-PowerEdge-T640 kernel: [ 759.412539] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 0, TPC 0, SM 1): Out Of Range Register
Nov 15 16:19:18 dell-PowerEdge-T640 kernel: [ 759.415110] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x5047b0=0x3000d 0x5047b4=0x20 0x5047a8=0x4c1eb72 0x5047ac=0x174
Nov 15 16:19:18 dell-PowerEdge-T640 kernel: [ 759.418021] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 0, TPC 1, SM 0): Out Of Range Register
Nov 15 16:19:18 dell-PowerEdge-T640 kernel: [ 759.420589] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x504f30=0xd 0x504f34=0x20 0x504f28=0x4c1eb72 0x504f2c=0x174
Nov 15 16:19:18 dell-PowerEdge-T640 kernel: [ 759.423418] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 0, TPC 1, SM 1): Out Of Range Register
Nov 15 16:19:18 dell-PowerEdge-T640 kernel: [ 759.425982] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x504fb0=0x2000d 0x504fb4=0x20 0x504fa8=0x4c1eb72 0x504fac=0x174
Nov 15 16:19:18 dell-PowerEdge-T640 kernel: [ 759.428897] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 0, TPC 2, SM 0): Out Of Range Register
Nov 15 16:19:18 dell-PowerEdge-T640 kernel: [ 759.431464] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Global Exception on (GPC 0, TPC 2, SM 0): Multiple Warp Errors
Nov 15 16:19:18 dell-PowerEdge-T640 kernel: [ 759.434045] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x505730=0x4000d 0x505734=0x24 0x505728=0x4c1eb72 0x50572c=0x174
Nov 15 16:19:18 dell-PowerEdge-T640 kernel: [ 759.436952] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 0, TPC 2, SM 1): Out Of Range Register
Nov 15 16:19:18 dell-PowerEdge-T640 kernel: [ 759.439518] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x5057b0=0x1000d 0x5057b4=0x20 0x5057a8=0x4c1eb72 0x5057ac=0x174
Nov 15 16:19:18 dell-PowerEdge-T640 kernel: [ 759.442430] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 0, TPC 3, SM 0): Out Of Range Register
Nov 15 16:19:18 dell-PowerEdge-T640 kernel: [ 759.487819] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x505f30=0x2000d 0x505f34=0x20 0x505f28=0x4c1eb72 0x505f2c=0x174
Nov 15 16:19:18 dell-PowerEdge-T640 kernel: [ 759.577346] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 0, TPC 3, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.666743] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x505fb0=0xd 0x505fb4=0x20 0x505fa8=0x4c1eb72 0x505fac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.666800] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 0, TPC 4, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.666803] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x506730=0x1000d 0x506734=0x20 0x506728=0x4c1eb72 0x50672c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.666845] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 0, TPC 4, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.666848] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x5067b0=0x2000d 0x5067b4=0x20 0x5067a8=0x4c1eb72 0x5067ac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.666900] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 1, TPC 0, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.666903] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x50c730=0x1000d 0x50c734=0x20 0x50c728=0x4c1eb72 0x50c72c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.666945] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 1, TPC 0, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.666949] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x50c7b0=0x1000d 0x50c7b4=0x20 0x50c7a8=0x4c1eb72 0x50c7ac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.666999] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 1, TPC 1, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667002] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x50cf30=0x2000d 0x50cf34=0x20 0x50cf28=0x4c1eb72 0x50cf2c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667048] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 1, TPC 1, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667052] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x50cfb0=0x1000d 0x50cfb4=0x20 0x50cfa8=0x4c1eb72 0x50cfac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667104] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 1, TPC 2, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667107] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x50d730=0x2000d 0x50d734=0x20 0x50d728=0x4c1eb72 0x50d72c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667148] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 1, TPC 2, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667152] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x50d7b0=0x1000d 0x50d7b4=0x20 0x50d7a8=0x4c1eb72 0x50d7ac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667202] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 1, TPC 3, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667205] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x50df30=0xd 0x50df34=0x20 0x50df28=0x4c1eb72 0x50df2c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667246] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 1, TPC 3, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667250] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x50dfb0=0x3000d 0x50dfb4=0x20 0x50dfa8=0x4c1eb72 0x50dfac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667300] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 1, TPC 4, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667303] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x50e730=0x2000d 0x50e734=0x20 0x50e728=0x4c1eb72 0x50e72c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667345] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 1, TPC 4, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667348] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x50e7b0=0x7000d 0x50e7b4=0x20 0x50e7a8=0x4c1eb72 0x50e7ac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667401] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 2, TPC 0, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667404] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x514730=0x1000d 0x514734=0x20 0x514728=0x4c1eb72 0x51472c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667446] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 2, TPC 0, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667449] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x5147b0=0x1000d 0x5147b4=0x20 0x5147a8=0x4c1eb72 0x5147ac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667499] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 2, TPC 1, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667503] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Global Exception on (GPC 2, TPC 1, SM 0): Multiple Warp Errors
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667505] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x514f30=0x3000d 0x514f34=0x24 0x514f28=0x4c1eb72 0x514f2c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667547] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 2, TPC 1, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667550] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x514fb0=0x1000d 0x514fb4=0x20 0x514fa8=0x4c1eb72 0x514fac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667601] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 2, TPC 2, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667604] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x515730=0x3000d 0x515734=0x20 0x515728=0x4c1eb72 0x51572c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667647] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 2, TPC 2, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667650] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x5157b0=0x1000d 0x5157b4=0x20 0x5157a8=0x4c1eb72 0x5157ac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667702] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 2, TPC 3, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667706] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x515f30=0x3000d 0x515f34=0x20 0x515f28=0x4c1eb72 0x515f2c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667748] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 2, TPC 3, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667751] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x515fb0=0x4000d 0x515fb4=0x20 0x515fa8=0x4c1eb72 0x515fac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667801] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 2, TPC 4, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667804] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x516730=0x2000d 0x516734=0x20 0x516728=0x4c1eb72 0x51672c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667846] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 2, TPC 4, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667849] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x5167b0=0xd 0x5167b4=0x20 0x5167a8=0x4c1eb72 0x5167ac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667899] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 2, TPC 5, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667902] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x516f30=0xd 0x516f34=0x20 0x516f28=0x4c1eb72 0x516f2c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667944] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 2, TPC 5, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.667947] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x516fb0=0x3000d 0x516fb4=0x20 0x516fa8=0x4c1eb72 0x516fac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668000] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 3, TPC 0, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668003] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x51c730=0x7000d 0x51c734=0x20 0x51c728=0x4c1eb72 0x51c72c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668044] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 3, TPC 0, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668048] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x51c7b0=0x2000d 0x51c7b4=0x20 0x51c7a8=0x4c1eb72 0x51c7ac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668098] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 3, TPC 1, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668101] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x51cf30=0x2000d 0x51cf34=0x20 0x51cf28=0x4c1eb72 0x51cf2c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668143] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 3, TPC 1, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668146] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x51cfb0=0x2000d 0x51cfb4=0x20 0x51cfa8=0x4c1eb72 0x51cfac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668197] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 3, TPC 2, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668200] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x51d730=0x1000d 0x51d734=0x20 0x51d728=0x4c1eb72 0x51d72c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668242] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 3, TPC 2, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668245] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x51d7b0=0x2000d 0x51d7b4=0x20 0x51d7a8=0x4c1eb72 0x51d7ac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668295] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 3, TPC 3, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668298] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Global Exception on (GPC 3, TPC 3, SM 0): Multiple Warp Errors
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668301] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x51df30=0x1000d 0x51df34=0x24 0x51df28=0x4c1eb72 0x51df2c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668343] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 3, TPC 3, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668346] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x51dfb0=0x1000d 0x51dfb4=0x20 0x51dfa8=0x4c1eb72 0x51dfac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668395] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 3, TPC 4, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668398] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x51e730=0x1000d 0x51e734=0x20 0x51e728=0x4c1eb72 0x51e72c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668440] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 3, TPC 4, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668443] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x51e7b0=0x2000d 0x51e7b4=0x20 0x51e7a8=0x4c1eb72 0x51e7ac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668493] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 3, TPC 5, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668496] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x51ef30=0x3000d 0x51ef34=0x20 0x51ef28=0x4c1eb72 0x51ef2c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668538] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 3, TPC 5, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668541] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Global Exception on (GPC 3, TPC 5, SM 1): Multiple Warp Errors
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668544] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x51efb0=0x2000d 0x51efb4=0x24 0x51efa8=0x4c1eb72 0x51efac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668595] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 4, TPC 0, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668599] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x524730=0x1000d 0x524734=0x20 0x524728=0x4c1eb72 0x52472c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668641] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 4, TPC 0, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668644] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x5247b0=0xd 0x5247b4=0x20 0x5247a8=0x4c1eb72 0x5247ac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668693] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 4, TPC 1, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668696] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x524f30=0x1000d 0x524f34=0x20 0x524f28=0x4c1eb72 0x524f2c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668738] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 4, TPC 1, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668741] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x524fb0=0x2000d 0x524fb4=0x20 0x524fa8=0x4c1eb72 0x524fac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668791] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 4, TPC 2, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668794] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x525730=0x1000d 0x525734=0x20 0x525728=0x4c1eb72 0x52572c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668836] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 4, TPC 2, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668839] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x5257b0=0x3000d 0x5257b4=0x20 0x5257a8=0x4c1eb72 0x5257ac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668889] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 4, TPC 3, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668892] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x525f30=0xd 0x525f34=0x20 0x525f28=0x4c1eb72 0x525f2c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668934] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 4, TPC 3, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668937] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x525fb0=0xd 0x525fb4=0x20 0x525fa8=0x4c1eb72 0x525fac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668987] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 4, TPC 4, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.668990] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x526730=0x1000d 0x526734=0x20 0x526728=0x4c1eb72 0x52672c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669032] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 4, TPC 4, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669035] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x5267b0=0xd 0x5267b4=0x20 0x5267a8=0x4c1eb72 0x5267ac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669085] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 4, TPC 5, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669088] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x526f30=0x2000d 0x526f34=0x20 0x526f28=0x4c1eb72 0x526f2c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669129] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 4, TPC 5, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669133] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x526fb0=0x1000d 0x526fb4=0x20 0x526fa8=0x4c1eb72 0x526fac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669184] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 5, TPC 0, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669187] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x52c730=0x2000d 0x52c734=0x20 0x52c728=0x4c1eb72 0x52c72c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669229] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 5, TPC 0, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669232] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x52c7b0=0x3000d 0x52c7b4=0x20 0x52c7a8=0x4c1eb72 0x52c7ac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669282] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 5, TPC 1, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669286] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x52cf30=0x1000d 0x52cf34=0x20 0x52cf28=0x4c1eb72 0x52cf2c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669327] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 5, TPC 1, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669330] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x52cfb0=0x1000d 0x52cfb4=0x20 0x52cfa8=0x4c1eb72 0x52cfac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669380] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 5, TPC 2, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669383] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x52d730=0x1000d 0x52d734=0x20 0x52d728=0x4c1eb72 0x52d72c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669425] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 5, TPC 2, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669428] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x52d7b0=0x3000d 0x52d7b4=0x20 0x52d7a8=0x4c1eb72 0x52d7ac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669478] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 5, TPC 3, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669481] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x52df30=0x3000d 0x52df34=0x20 0x52df28=0x4c1eb72 0x52df2c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669523] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 5, TPC 3, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669526] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x52dfb0=0x5000d 0x52dfb4=0x20 0x52dfa8=0x4c1eb72 0x52dfac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669576] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 5, TPC 4, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669579] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x52e730=0x1000d 0x52e734=0x20 0x52e728=0x4c1eb72 0x52e72c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669621] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 5, TPC 4, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669624] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x52e7b0=0x1000d 0x52e7b4=0x20 0x52e7a8=0x4c1eb72 0x52e7ac=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669674] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 5, TPC 5, SM 0): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669677] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x52ef30=0x2000d 0x52ef34=0x20 0x52ef28=0x4c1eb72 0x52ef2c=0x174
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669724] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 5, TPC 5, SM 1): Out Of Range Register
Nov 15 16:19:29 dell-PowerEdge-T640 kernel: [ 759.669728] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x52efb0=0xd 0x52efb4=0x20 0x52efa8=0x4c1eb72 0x52efac=0x174
Nov 15 16:19:48 dell-PowerEdge-T640 kernel: [ 789.532000] NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ChID 0010, Class 0000c5c0, Offset 00000000, Data 00000000
Nov 15 16:21:48 dell-PowerEdge-T640 kernel: [ 909.618360] NVRM: Xid (PCI:0000:b1:00): 31, pid=11160, Ch 00000010, intr 00000000. MMU Fault: ENGINE GRAPHICS GPCCLIENT_T1_1 faulted @ 0x7f9f_23250000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_WRITE
Nov 15 16:22:18 dell-PowerEdge-T640 kernel: [ 939.753042] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000000
Nov 15 16:22:40 dell-PowerEdge-T640 kernel: [ 961.909344] INFO: task nvidia-smi:11441 blocked for more than 120 seconds.
Nov 15 16:22:40 dell-PowerEdge-T640 kernel: [ 962.068017] nvidia-smi D ffff886e4c5bbaa8 0 11441 11406 0x00000000
Nov 15 16:22:48 dell-PowerEdge-T640 kernel: [ 969.799819] NVRM: Xid (PCI:0000:b1:00): 45, pid=4354, Ch 00000001
Nov 15 16:22:48 dell-PowerEdge-T640 kernel: [ 969.843144] NVRM: Xid (PCI:0000:b1:00): 45, pid=4354, Ch 00000002
Nov 15 16:22:48 dell-PowerEdge-T640 kernel: [ 969.886126] NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000003
Nov 15 16:22:48 dell-PowerEdge-T640 kernel: [ 969.926755] NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000004
Nov 15 16:22:48 dell-PowerEdge-T640 kernel: [ 969.966007] NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000005
Nov 15 16:23:18 dell-PowerEdge-T640 kernel: [ 1000.002561] NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000006
Nov 15 16:23:18 dell-PowerEdge-T640 kernel: [ 1000.039680] NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000007
Nov 15 16:23:18 dell-PowerEdge-T640 kernel: [ 1000.076891] NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000008
Nov 15 16:23:18 dell-PowerEdge-T640 kernel: [ 1000.111829] NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000010
Nov 15 16:23:18 dell-PowerEdge-T640 kernel: [ 1000.145446] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000011
Nov 15 16:23:18 dell-PowerEdge-T640 kernel: [ 1000.178193] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000012
Nov 15 16:23:18 dell-PowerEdge-T640 kernel: [ 1000.209963] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000013
Nov 15 16:23:19 dell-PowerEdge-T640 kernel: [ 1000.240533] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000014
Nov 15 16:23:19 dell-PowerEdge-T640 kernel: [ 1000.269775] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000015
Nov 15 16:23:19 dell-PowerEdge-T640 kernel: [ 1000.297912] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000016
Nov 15 16:23:19 dell-PowerEdge-T640 kernel: [ 1000.325034] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000017
Nov 15 16:23:19 dell-PowerEdge-T640 kernel: [ 1000.350998] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000018
Nov 15 16:23:19 dell-PowerEdge-T640 kernel: [ 1000.375683] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000019
Nov 15 16:23:19 dell-PowerEdge-T640 kernel: [ 1000.399225] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 0000001a
Nov 15 16:23:19 dell-PowerEdge-T640 kernel: [ 1000.421721] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 0000001b
Nov 15 16:23:19 dell-PowerEdge-T640 kernel: [ 1000.443054] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 0000001c
Nov 15 16:23:19 dell-PowerEdge-T640 kernel: [ 1000.463082] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 0000001d
Nov 15 16:23:19 dell-PowerEdge-T640 kernel: [ 1000.481935] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 0000001e
Nov 15 16:23:19 dell-PowerEdge-T640 kernel: [ 1000.499726] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 0000001f
Nov 15 16:23:19 dell-PowerEdge-T640 kernel: [ 1000.516379] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000020
Nov 15 16:23:49 dell-PowerEdge-T640 kernel: [ 1030.529359] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000021
Nov 15 16:23:49 dell-PowerEdge-T640 kernel: [ 1030.543570] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000022
Nov 15 16:23:49 dell-PowerEdge-T640 kernel: [ 1030.556583] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000023
Nov 15 16:25:19 dell-PowerEdge-T640 kernel: [ 1120.737798] NVRM: Xid (PCI:0000:b1:00): 32, pid=11160, Channel ID 00000003 intr1 00000010 HCE_DBG0 00000000 HCE_DBG1 2004000a
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1180.835384] NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000000
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1180.847992] NVRM: Xid (PCI:0000:b1:00): 45, pid=4354, Ch 00000001
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1180.860472] NVRM: Xid (PCI:0000:b1:00): 45, pid=4354, Ch 00000002
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1180.872900] NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000003
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1180.885333] NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000004
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1180.901578] NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000005
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1180.913947] NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000006
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1180.926298] NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000007
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1180.938622] NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000008
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1180.950707] NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000010
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1180.962861] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000011
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1180.975239] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000012
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1180.987633] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000013
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1181.000057] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000014
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1181.012493] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000015
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1181.024893] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000016
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1181.037335] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000017
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1181.049597] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000018
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1181.061887] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000019
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1181.074124] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 0000001a
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1181.086355] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 0000001b
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1181.098628] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 0000001c
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1181.110977] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 0000001d
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1181.123317] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 0000001e
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1181.135587] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 0000001f
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1181.147824] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000020
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1181.160102] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000021
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1181.172377] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000022
Nov 15 16:26:19 dell-PowerEdge-T640 kernel: [ 1181.184228] NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000023
Nov 15 16:28:14 dell-PowerEdge-T640 kernel: [ 1295.514789] NVRM: Xid (PCI:0000:b1:00): 45, pid=4435, Ch 00000000
Nov 15 16:28:14 dell-PowerEdge-T640 kernel: [ 1295.525037] NVRM: Xid (PCI:0000:b1:00): 45, pid=4354, Ch 00000001
Nov 15 16:28:14 dell-PowerEdge-T640 kernel: [ 1295.534767] NVRM: Xid (PCI:0000:b1:00): 45, pid=4354, Ch 00000002
Nov 15 16:28:14 dell-PowerEdge-T640 kernel: [ 1295.544488] NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000003
Nov 15 16:28:14 dell-PowerEdge-T640 kernel: [ 1295.554158] NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000004
Nov 15 16:28:14 dell-PowerEdge-T640 kernel: [ 1295.563792] NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000005
Nov 15 16:28:14 dell-PowerEdge-T640 kernel: [ 1295.573392] NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000006
Nov 15 16:28:14 dell-PowerEdge-T640 kernel: [ 1295.582945] NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000007
Nov 15 16:37:58 dell-PowerEdge-T640 kernel: [ 1879.434610] NVRM: GPU 0000:b1:00.0: RmInitAdapter failed! (0x24:0x65:1184)
Nov 15 16:37:58 dell-PowerEdge-T640 kernel: [ 1879.434635] NVRM: GPU 0000:b1:00.0: rm_init_adapter failed, device minor number 2
Nov 15 16:38:53 dell-PowerEdge-T640 kernel: [ 1934.472201] NVRM: GPU 0000:b1:00.0: RmInitAdapter failed! (0x24:0x65:1184)
Nov 15 16:38:53 dell-PowerEdge-T640 kernel: [ 1934.472223] NVRM: GPU 0000:b1:00.0: rm_init_adapter failed, device minor number 2
Nov 15 17:33:39 dell-PowerEdge-T640 kernel: [ 5220.711355] NVRM: GPU 0000:b1:00.0: RmInitAdapter failed! (0x24:0x65:1184)
Nov 15 17:33:39 dell-PowerEdge-T640 kernel: [ 5220.711378] NVRM: GPU 0000:b1:00.0: rm_init_adapter failed, device minor number 2

/var/log/dmesg:

journalctl -b -0:
11月 15 16:06:53 dell-PowerEdge-T640 kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 244
11月 15 16:06:54 dell-PowerEdge-T640 kernel: NVRM: loading NVIDIA UNIX x86_64 Kernel Module 435.21 Sun Aug 25 08:17:57 CDT 2019
11月 15 16:06:54 dell-PowerEdge-T640 kernel: nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 435.21 Sun Aug 25 08:07:52 CDT 2019
11月 15 16:06:54 dell-PowerEdge-T640 kernel: [drm] [nvidia-drm] [GPU ID 0x00003b00] Loading driver
11月 15 16:06:54 dell-PowerEdge-T640 kernel: [drm] [nvidia-drm] [GPU ID 0x00005e00] Loading driver
11月 15 16:06:54 dell-PowerEdge-T640 kernel: [drm] [nvidia-drm] [GPU ID 0x0000b100] Loading driver
11月 15 16:09:09 dell-PowerEdge-T640 kernel: nvidia-uvm: Loaded the UVM driver, major device number 242.
11月 15 16:19:18 dell-PowerEdge-T640 kernel: NVRM: GPU at PCI:0000:b1:00: GPU-2c48c2eb-cdc7-02bf-ac64-aa3ef8c88e96
11月 15 16:19:18 dell-PowerEdge-T640 kernel: NVRM: GPU Board Serial Number: 0324218061405
11月 15 16:19:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 61, pid=4354, 0cde(308c) 00000000 00000000
11月 15 16:19:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 0, TPC 0, SM 0): Out Of Range Register
11月 15 16:19:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x504730=0x4000d 0x504734=0x0 0x504728=0x4c1eb72 0x50472c=0x174
11月 15 16:19:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 0, TPC 0, SM 1): Out Of Range Register
11月 15 16:19:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x5047b0=0x3000d 0x5047b4=0x20 0x5047a8=0x4c1eb72 0x5047ac=0x174
11月 15 16:19:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 0, TPC 1, SM 0): Out Of Range Register
11月 15 16:19:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x504f30=0xd 0x504f34=0x20 0x504f28=0x4c1eb72 0x504f2c=0x174
11月 15 16:19:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 0, TPC 1, SM 1): Out Of Range Register
11月 15 16:19:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x504fb0=0x2000d 0x504fb4=0x20 0x504fa8=0x4c1eb72 0x504fac=0x174
11月 15 16:19:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 0, TPC 2, SM 0): Out Of Range Register
11月 15 16:19:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Global Exception on (GPC 0, TPC 2, SM 0): Multiple Warp Errors
11月 15 16:19:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x505730=0x4000d 0x505734=0x24 0x505728=0x4c1eb72 0x50572c=0x174
11月 15 16:19:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 0, TPC 2, SM 1): Out Of Range Register
11月 15 16:19:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x5057b0=0x1000d 0x5057b4=0x20 0x5057a8=0x4c1eb72 0x5057ac=0x174
11月 15 16:19:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 0, TPC 3, SM 0): Out Of Range Register
11月 15 16:19:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x505f30=0x2000d 0x505f34=0x20 0x505f28=0x4c1eb72 0x505f2c=0x174
11月 15 16:19:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 0, TPC 3, SM 1): Out Of Range Register
11月 15 16:19:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x505fb0=0xd 0x505fb4=0x20 0x505fa8=0x4c1eb72 0x505fac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 0, TPC 4, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x506730=0x1000d 0x506734=0x20 0x506728=0x4c1eb72 0x50672c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 0, TPC 4, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x5067b0=0x2000d 0x5067b4=0x20 0x5067a8=0x4c1eb72 0x5067ac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 1, TPC 0, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x50c730=0x1000d 0x50c734=0x20 0x50c728=0x4c1eb72 0x50c72c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 1, TPC 0, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x50c7b0=0x1000d 0x50c7b4=0x20 0x50c7a8=0x4c1eb72 0x50c7ac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 1, TPC 1, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x50cf30=0x2000d 0x50cf34=0x20 0x50cf28=0x4c1eb72 0x50cf2c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 1, TPC 1, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x50cfb0=0x1000d 0x50cfb4=0x20 0x50cfa8=0x4c1eb72 0x50cfac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 1, TPC 2, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x50d730=0x2000d 0x50d734=0x20 0x50d728=0x4c1eb72 0x50d72c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 1, TPC 2, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x50d7b0=0x1000d 0x50d7b4=0x20 0x50d7a8=0x4c1eb72 0x50d7ac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 1, TPC 3, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x50df30=0xd 0x50df34=0x20 0x50df28=0x4c1eb72 0x50df2c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 1, TPC 3, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x50dfb0=0x3000d 0x50dfb4=0x20 0x50dfa8=0x4c1eb72 0x50dfac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 1, TPC 4, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x50e730=0x2000d 0x50e734=0x20 0x50e728=0x4c1eb72 0x50e72c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 1, TPC 4, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x50e7b0=0x7000d 0x50e7b4=0x20 0x50e7a8=0x4c1eb72 0x50e7ac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 2, TPC 0, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x514730=0x1000d 0x514734=0x20 0x514728=0x4c1eb72 0x51472c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 2, TPC 0, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x5147b0=0x1000d 0x5147b4=0x20 0x5147a8=0x4c1eb72 0x5147ac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 2, TPC 1, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Global Exception on (GPC 2, TPC 1, SM 0): Multiple Warp Errors
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x514f30=0x3000d 0x514f34=0x24 0x514f28=0x4c1eb72 0x514f2c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 2, TPC 1, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x514fb0=0x1000d 0x514fb4=0x20 0x514fa8=0x4c1eb72 0x514fac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 2, TPC 2, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x515730=0x3000d 0x515734=0x20 0x515728=0x4c1eb72 0x51572c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 2, TPC 2, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x5157b0=0x1000d 0x5157b4=0x20 0x5157a8=0x4c1eb72 0x5157ac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 2, TPC 3, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x515f30=0x3000d 0x515f34=0x20 0x515f28=0x4c1eb72 0x515f2c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 2, TPC 3, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x515fb0=0x4000d 0x515fb4=0x20 0x515fa8=0x4c1eb72 0x515fac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 2, TPC 4, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x516730=0x2000d 0x516734=0x20 0x516728=0x4c1eb72 0x51672c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 2, TPC 4, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x5167b0=0xd 0x5167b4=0x20 0x5167a8=0x4c1eb72 0x5167ac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 2, TPC 5, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x516f30=0xd 0x516f34=0x20 0x516f28=0x4c1eb72 0x516f2c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 2, TPC 5, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x516fb0=0x3000d 0x516fb4=0x20 0x516fa8=0x4c1eb72 0x516fac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 3, TPC 0, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x51c730=0x7000d 0x51c734=0x20 0x51c728=0x4c1eb72 0x51c72c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 3, TPC 0, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x51c7b0=0x2000d 0x51c7b4=0x20 0x51c7a8=0x4c1eb72 0x51c7ac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 3, TPC 1, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x51cf30=0x2000d 0x51cf34=0x20 0x51cf28=0x4c1eb72 0x51cf2c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 3, TPC 1, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x51cfb0=0x2000d 0x51cfb4=0x20 0x51cfa8=0x4c1eb72 0x51cfac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 3, TPC 2, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x51d730=0x1000d 0x51d734=0x20 0x51d728=0x4c1eb72 0x51d72c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 3, TPC 2, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x51d7b0=0x2000d 0x51d7b4=0x20 0x51d7a8=0x4c1eb72 0x51d7ac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 3, TPC 3, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Global Exception on (GPC 3, TPC 3, SM 0): Multiple Warp Errors
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x51df30=0x1000d 0x51df34=0x24 0x51df28=0x4c1eb72 0x51df2c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 3, TPC 3, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x51dfb0=0x1000d 0x51dfb4=0x20 0x51dfa8=0x4c1eb72 0x51dfac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 3, TPC 4, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x51e730=0x1000d 0x51e734=0x20 0x51e728=0x4c1eb72 0x51e72c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 3, TPC 4, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x51e7b0=0x2000d 0x51e7b4=0x20 0x51e7a8=0x4c1eb72 0x51e7ac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 3, TPC 5, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x51ef30=0x3000d 0x51ef34=0x20 0x51ef28=0x4c1eb72 0x51ef2c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 3, TPC 5, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Global Exception on (GPC 3, TPC 5, SM 1): Multiple Warp Errors
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x51efb0=0x2000d 0x51efb4=0x24 0x51efa8=0x4c1eb72 0x51efac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 4, TPC 0, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x524730=0x1000d 0x524734=0x20 0x524728=0x4c1eb72 0x52472c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 4, TPC 0, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x5247b0=0xd 0x5247b4=0x20 0x5247a8=0x4c1eb72 0x5247ac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 4, TPC 1, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x524f30=0x1000d 0x524f34=0x20 0x524f28=0x4c1eb72 0x524f2c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 4, TPC 1, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x524fb0=0x2000d 0x524fb4=0x20 0x524fa8=0x4c1eb72 0x524fac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 4, TPC 2, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x525730=0x1000d 0x525734=0x20 0x525728=0x4c1eb72 0x52572c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 4, TPC 2, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x5257b0=0x3000d 0x5257b4=0x20 0x5257a8=0x4c1eb72 0x5257ac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 4, TPC 3, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x525f30=0xd 0x525f34=0x20 0x525f28=0x4c1eb72 0x525f2c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 4, TPC 3, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x525fb0=0xd 0x525fb4=0x20 0x525fa8=0x4c1eb72 0x525fac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 4, TPC 4, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x526730=0x1000d 0x526734=0x20 0x526728=0x4c1eb72 0x52672c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 4, TPC 4, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x5267b0=0xd 0x5267b4=0x20 0x5267a8=0x4c1eb72 0x5267ac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 4, TPC 5, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x526f30=0x2000d 0x526f34=0x20 0x526f28=0x4c1eb72 0x526f2c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 4, TPC 5, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x526fb0=0x1000d 0x526fb4=0x20 0x526fa8=0x4c1eb72 0x526fac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 5, TPC 0, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x52c730=0x2000d 0x52c734=0x20 0x52c728=0x4c1eb72 0x52c72c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 5, TPC 0, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x52c7b0=0x3000d 0x52c7b4=0x20 0x52c7a8=0x4c1eb72 0x52c7ac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 5, TPC 1, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x52cf30=0x1000d 0x52cf34=0x20 0x52cf28=0x4c1eb72 0x52cf2c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 5, TPC 1, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x52cfb0=0x1000d 0x52cfb4=0x20 0x52cfa8=0x4c1eb72 0x52cfac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 5, TPC 2, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x52d730=0x1000d 0x52d734=0x20 0x52d728=0x4c1eb72 0x52d72c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 5, TPC 2, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x52d7b0=0x3000d 0x52d7b4=0x20 0x52d7a8=0x4c1eb72 0x52d7ac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 5, TPC 3, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x52df30=0x3000d 0x52df34=0x20 0x52df28=0x4c1eb72 0x52df2c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 5, TPC 3, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x52dfb0=0x5000d 0x52dfb4=0x20 0x52dfa8=0x4c1eb72 0x52dfac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 5, TPC 4, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x52e730=0x1000d 0x52e734=0x20 0x52e728=0x4c1eb72 0x52e72c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 5, TPC 4, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x52e7b0=0x1000d 0x52e7b4=0x20 0x52e7a8=0x4c1eb72 0x52e7ac=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 5, TPC 5, SM 0): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x52ef30=0x2000d 0x52ef34=0x20 0x52ef28=0x4c1eb72 0x52ef2c=0x174
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics SM Warp Exception on (GPC 5, TPC 5, SM 1): Out Of Range Register
11月 15 16:19:29 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ESR 0x52efb0=0xd 0x52efb4=0x20 0x52efa8=0x4c1eb72 0x52efac=0x174
11月 15 16:19:48 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 13, pid=4354, Graphics Exception: ChID 0010, Class 0000c5c0, Offset 00000000, Data 00000000
11月 15 16:21:48 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 31, pid=11160, Ch 00000010, intr 00000000. MMU Fault: ENGINE GRAPHICS GPCCLIENT_T1_1 faulted @ 0x7f9f_23250000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_WRITE
11月 15 16:22:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000000
11月 15 16:22:40 dell-PowerEdge-T640 kernel: INFO: task nvidia-smi:11441 blocked for more than 120 seconds.
11月 15 16:22:40 dell-PowerEdge-T640 kernel: nvidia-smi D ffff886e4c5bbaa8 0 11441 11406 0x00000000
11月 15 16:22:48 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=4354, Ch 00000001
11月 15 16:22:48 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=4354, Ch 00000002
11月 15 16:22:48 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000003
11月 15 16:22:48 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000004
11月 15 16:22:48 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000005
11月 15 16:23:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000006
11月 15 16:23:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000007
11月 15 16:23:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000008
11月 15 16:23:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000010
11月 15 16:23:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000011
11月 15 16:23:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000012
11月 15 16:23:18 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000013
11月 15 16:23:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000014
11月 15 16:23:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000015
11月 15 16:23:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000016
11月 15 16:23:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000017
11月 15 16:23:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000018
11月 15 16:23:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000019
11月 15 16:23:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 0000001a
11月 15 16:23:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 0000001b
11月 15 16:23:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 0000001c
11月 15 16:23:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 0000001d
11月 15 16:23:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 0000001e
11月 15 16:23:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 0000001f
11月 15 16:23:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000020
11月 15 16:23:49 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000021
11月 15 16:23:49 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000022
11月 15 16:23:49 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000023
11月 15 16:25:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 32, pid=11160, Channel ID 00000003 intr1 00000010 HCE_DBG0 00000000 HCE_DBG1 2004000a
11月 15 16:26:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000000
11月 15 16:26:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=4354, Ch 00000001
11月 15 16:26:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=4354, Ch 00000002
11月 15 16:26:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000003
11月 15 16:26:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000004
11月 15 16:26:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000005
11月 15 16:26:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000006
11月 15 16:26:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000007
11月 15 16:26:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000008
11月 15 16:26:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=4304, Ch 00000010
11月 15 16:26:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000011
11月 15 16:26:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000012
11月 15 16:26:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000013
11月 15 16:26:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000014
11月 15 16:26:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160, Ch 00000015
11月 15 16:26:19 dell-PowerEdge-T640 kernel: NVRM: Xid (PCI:0000:b1:00): 45, pid=11160,