I have trained my model using pytorch,after few hour,my program has no response just like frozon.I thought maybe one of my graph card is faulty.So,I use nvidia-bug-report …There is something might unnormal in nvidia-bug-report.log:
Nov 15 12:43:10 sdu-dpai-turing-1 kernel: [ 7613.977841] NVRM: Xid (PCI:0000:03:00): 13, pid=10635, Graphics SM Warp Exception on (GPC 0, TPC 0, SM 0): Illegal Instruction Encoding
Nov 15 12:43:10 sdu-dpai-turing-1 kernel: [ 7613.977856] NVRM: Xid (PCI:0000:03:00): 13, pid=10635, Graphics Exception: ESR 0x504730=0x100009 0x504734=0x0 0x504728=0x4c1eb72 0x50472c=0x174
Nov 15 12:43:10 sdu-dpai-turing-1 kernel: [ 7613.977966] NVRM: Xid (PCI:0000:03:00): 13, pid=10635, Graphics SM Warp Exception on (GPC 0, TPC 0, SM 1): Illegal Instruction Encoding
Nov 15 12:43:10 sdu-dpai-turing-1 kernel: [ 7613.977977] NVRM: Xid (PCI:0000:03:00): 13, pid=10635, Graphics SM Global Exception on (GPC 0, TPC 0, SM 1): Multiple Warp Errors
Nov 15 12:43:10 sdu-dpai-turing-1 kernel: [ 7613.977986] NVRM: Xid (PCI:0000:03:00): 13, pid=10635, Graphics Exception: ESR 0x5047b0=0x1e0009 0x5047b4=0x4 0x5047a8=0x4c1eb72 0x5047ac=0x174
Nov 15 12:43:10 sdu-dpai-turing-1 kernel: [ 7613.978107] NVRM: Xid (PCI:0000:03:00): 13, pid=10635, Graphics SM Warp Exception on (GPC 0, TPC 1, SM 0): Illegal Instruction Encoding
Nov 15 12:43:10 sdu-dpai-turing-1 kernel: [ 7613.978117] NVRM: Xid (PCI:0000:03:00): 13, pid=10635, Graphics Exception: ESR 0x504f30=0x1d0009 0x504f34=0x0 0x504f28=0x4c1eb72 0x504f2c=0x174
Nov 15 12:43:10 sdu-dpai-turing-1 kernel: [ 7613.978222] NVRM: Xid (PCI:0000:03:00): 13, pid=10635, Graphics SM Warp Exception on (GPC 0, TPC 1, SM 1): Illegal Instruction Encoding
Nov 15 12:43:10 sdu-dpai-turing-1 kernel: [ 7613.978232] NVRM: Xid (PCI:0000:03:00): 13, pid=10635, Graphics SM Global Exception on (GPC 0, TPC 1, SM 1): Multiple Warp Errors
Nov 15 12:43:10 sdu-dpai-turing-1 kernel: [ 7613.978240] NVRM: Xid (PCI:0000:03:00): 13, pid=10635, Graphics Exception: ESR 0x504fb0=0x1e0009 0x504fb4=0x4 0x504fa8=0x4c1eb72 0x504fac=0x174
Nov 15 12:43:10 sdu-dpai-turing-1 kernel: [ 7613.978362] NVRM: Xid (PCI:0000:03:00): 13, pid=10635, Graphics SM Warp Exception on (GPC 0, TPC 2, SM 0): Illegal Instruction Encoding
Nov 15 12:43:10 sdu-dpai-turing-1 kernel: [ 7613.978372] NVRM: Xid (PCI:0000:03:00): 13, pid=10635, Graphics Exception: ESR 0x505730=0x130009 0x505734=0x0 0x505728=0x4c1eb72 0x50572c=0x174
Nov 15 12:43:10 sdu-dpai-turing-1 kernel: [ 7613.978478] NVRM: Xid (PCI:0000:03:00): 13, pid=10635, Graphics SM Warp Exception on (GPC 0, TPC 2, SM 1): Illegal Instruction Encoding
Nov 15 12:43:10 sdu-dpai-turing-1 kernel: [ 7613.978488] NVRM: Xid (PCI:0000:03:00): 13, pid=10635, Graphics Exception: ESR 0x5057b0=0x1f0009 0x5057b4=0x0 0x5057a8=0x4c1eb72 0x5057ac=0x174
Is （PCI:0000:03:00 gtx2080ti）gpu broken？
nvidia-bug-report.log.gz (2.4 MB)