# RTX 2080 cards crashed when training longer a PyTorch model **URL:** \ **Category:** Linux\ **Created:** [November 5, 2019, 10:05pm UTC](https://forums.developer.nvidia.com/t/rtx-2080-cards-crashed-when-training-longer-a-pytorch-model/106913 "2019-11-05T22:05:24Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![bqcaomail](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@bqcaomail](https://forums.developer.nvidia.com/u/bqcaomail)\ **Post date:** [November 5, 2019, 10:05pm UTC](https://forums.developer.nvidia.com/t/rtx-2080-cards-crashed-when-training-longer-a-pytorch-model/106913/1 "2019-11-05T22:05:24Z")
Hi There My OS is Ubuntu 18.04, I have 4 RTX 2080 on board. When I ran smaller training, it worked, but when I started running longer training the very first time, I got this: nvidia-smi Unable to determine the device handle for GPU 0000:19:00.0: GPU is lost. Reboot the system to recover this GPU sudo dmesg |tail -n 60 [76298.527993] wlx7cdd903371a2: send auth to 70:3a:0e:54:49:60 (try 1/3) [76298.529737] wlx7cdd903371a2: authenticated [76298.533225] wlx7cdd903371a2: associate with 70:3a:0e:54:49:60 (try 1/3) [76298.536445] wlx7cdd903371a2: RX AssocResp from 70:3a:0e:54:49:60 (capab=0x431 status=0 aid=1) [76298.539510] wlx7cdd903371a2: associated [76298.547546] IPv6: ADDRCONF(NETDEV\_CHANGE): wlx7cdd903371a2: link becomes ready [76617.186011] logitech-hidpp-device 0003:046D:200A.0006: unable to retrieve the name of the device [76771.242742] NVRM: GPU at PCI:0000:19:00: GPU-63b97003-35a2-c293-323b-8c2342e7bd46 [76771.242747] NVRM: GPU Board Serial Number: [76771.242751] NVRM: Xid (PCI:0000:19:00): 79, GPU has fallen off the bus. [76771.242756] NVRM: GPU at 00000000:19:00.0 has fallen off the bus. [76771.242757] NVRM: GPU is on Board . [76771.242769] NVRM: A GPU crash dump has been created. If possible, please run NVRM: nvidia-bug-report.sh as root to collect this data before NVRM: the NVIDIA kernel module is unloaded. [79799.361585] logitech-hidpp-device 0003:046D:200A.0006: unable to retrieve the name of the device [79952.669854] WARNING: CPU: 5 PID: 2407 at /tmp/selfgz3707/NVIDIA-Linux-x86\_64-418.39/kernel/nvidia/nv.c:5123 nvidia\_dev\_put\_uuid+0x49/0x50 [nvidia] [79952.669854] Modules linked in: nvidia\_uvm(OE) tcp\_diag inet\_diag xt\_conntrack ipt\_MASQUERADE nf\_nat\_masquerade\_ipv4 nf\_conntrack\_netlink nfnetlink xfrm\_user xfrm\_algo xt\_addrtype iptable\_filter iptable\_nat nf\_conntrack\_ipv4 nf\_defrag\_ipv4 nf\_nat\_ipv4 nf\_nat nf\_conntrack libcrc32c br\_netfilter bridge stp llc ccm aufs overlay snd\_hda\_codec\_hdmi nls\_iso8859\_1 nvidia\_drm(POE) nvidia\_modeset(POE) arc4 nvidia(POE) snd\_seq\_midi snd\_seq\_midi\_event intel\_rapl rt2800usb x86\_pkg\_temp\_thermal intel\_powerclamp coretemp rt2x00usb eeepc\_wmi snd\_hda\_codec\_realtek rt2800lib snd\_hda\_codec\_generic asus\_wmi snd\_rawmidi rt2x00lib sparse\_keymap snd\_hda\_intel mac80211 mxm\_wmi video kvm\_intel intel\_wmi\_thunderbolt wmi\_bmof cfg80211 snd\_hda\_codec kvm drm\_kms\_helper snd\_hda\_core irqbypass drm crct10dif\_pclmul joydev snd\_seq [79952.669879] crc32\_pclmul snd\_hwdep input\_leds ipmi\_devintf ghash\_clmulni\_intel pcbc snd\_seq\_device ipmi\_msghandler aesni\_intel fb\_sys\_fops snd\_pcm aes\_x86\_64 syscopyarea crypto\_simd snd\_timer glue\_helper sysfillrect sysimgblt cryptd intel\_cstate snd intel\_rapl\_perf soundcore mei\_me mei ioatdma mac\_hid shpchp wmi sch\_fq\_codel parport\_pc ppdev lp parport ip\_tables x\_tables autofs4 hid\_logitech\_hidpp hid\_generic hid\_logitech\_dj usbhid hid igb i2c\_algo\_bit e1000e dca ptp ahci pps\_core libahci [79952.669901] CPU: 5 PID: 2407 Comm: ZMQbg/1 Tainted: P OE 4.15.0-48-generic #51-Ubuntu [79952.669901] Hardware name: System manufacturer System Product Name/WS X299 SAGE, BIOS 0905 11/30/2018 [79952.669979] RIP: 0010:nvidia\_dev\_put\_uuid+0x49/0x50 [nvidia] [79952.669980] RSP: 0018:ffffafbda4b5fb68 EFLAGS: 00010202 [79952.669980] RAX: 0000000000000026 RBX: ffff928194e40800 RCX: ffffafbda4b5fb08 [79952.669981] RDX: 0000000000000087 RSI: 0000000000000246 RDI: 0000000000000246 [79952.669981] RBP: ffffafbda4b5fb78 R08: ffffcfbd7f960310 R09: 0000000000000000 [79952.669982] R10: ffffffffc3121bc0 R11: 0000000000000400 R12: ffff9271a7143000 [79952.669982] R13: ffffffffc09dbfe0 R14: ffffafbd8680e5c0 R15: 0000000000000000 [79952.669983] FS: 00007fba1d140700(0000) GS:ffff92819f540000(0000) knlGS:0000000000000000 [79952.669984] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [79952.669984] CR2: 00007f607f8bd100 CR3: 0000000aafa0a001 CR4: 00000000003606e0 [79952.669985] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [79952.669985] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [79952.669986] Call Trace: [79952.670042] nvUvmInterfaceUnregisterGpu+0x22/0x30 [nvidia] [79952.670053] remove\_gpu+0x28b/0x2e0 [nvidia\_uvm] [79952.670057] uvm\_gpu\_release\_locked+0x17/0x20 [nvidia\_uvm] [79952.670062] uvm\_va\_space\_destroy+0x3f7/0x490 [nvidia\_uvm] [79952.670066] uvm\_release+0x11/0x20 [nvidia\_uvm] [79952.670068] \_\_fput+0xea/0x220 [79952.670069] \_\_\_\_fput+0xe/0x10 [79952.670071] task\_work\_run+0x9d/0xc0 [79952.670073] do\_exit+0x2ec/0xb40 [79952.670074] do\_group\_exit+0x43/0xb0 [79952.670075] get\_signal+0x27b/0x590 [79952.670093] do\_signal+0x37/0x730 [79952.670095] ? compat\_poll\_select\_copy\_remaining+0x130/0x130 [79952.670096] ? compat\_poll\_select\_copy\_remaining+0x130/0x130 [79952.670098] exit\_to\_usermode\_loop+0x73/0xd0 [79952.670099] do\_syscall\_64+0x115/0x130 [79952.670101] entry\_SYSCALL\_64\_after\_hwframe+0x3d/0xa2 [79952.670103] RIP: 0033:0x7fbe71f15bf9 [79952.670103] RSP: 002b:00007fba1d13fda0 EFLAGS: 00000293 ORIG\_RAX: 0000000000000007 [79952.670104] RAX: fffffffffffffdfc RBX: 00007fba10000bd0 RCX: 00007fbe71f15bf9 [79952.670104] RDX: 0000000000000064 RSI: 000000000000000b RDI: 00007fba10000bd0 [79952.670105] RBP: 000000000000000b R08: 0000000000000000 R09: 00007fba1d140700 [79952.670105] R10: 00007fba1d13fdb0 R11: 0000000000000293 R12: 0000000000000064 [79952.670106] R13: 0000557a696a0400 R14: 00005579bed2ad78 R15: 000000000000000b [79952.670106] Code: c7 4c 89 e6 e8 69 b7 ff ff 31 d2 48 89 de 4c 89 e7 e8 3c c6 6d 00 85 c0 75 11 48 8d bb 58 04 00 00 e8 fc 3d 34 ca 5b 41 5c 5d c3 \<0f\> 0b eb eb 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 41 55 41 54 53 [79952.670123] —[end trace b855b128c44ee309]— The log is in attachment. Thanks! Bao [nvidia-bug-report.log.gz](https://forums.developer.nvidia.com/uploads/short-url/wtTd61iisMY0dMtOfJx4JqEYHgj.gz) (1.81 MB) ---
**Author:** ![generix](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/generix/32/16024_2.png) [@generix](https://forums.developer.nvidia.com/u/generix)\ **Post date:** [November 5, 2019, 10:23pm UTC](https://forums.developer.nvidia.com/t/rtx-2080-cards-crashed-when-training-longer-a-pytorch-model/106913/2 "2019-11-05T22:23:17Z")
xid 79 points to lack of power or overheating. ---
**Author:** ![bqcaomail](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@bqcaomail](https://forums.developer.nvidia.com/u/bqcaomail)\ **Post date:** [November 5, 2019, 11:08pm UTC](https://forums.developer.nvidia.com/t/rtx-2080-cards-crashed-when-training-longer-a-pytorch-model/106913/3 "2019-11-05T23:08:57Z")
Thanks. What’s the threshold temperature for RTX 2080? ---
**Author:** ![generix](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/generix/32/16024_2.png) [@generix](https://forums.developer.nvidia.com/u/generix)\ **Post date:** [November 6, 2019, 7:52am UTC](https://forums.developer.nvidia.com/t/rtx-2080-cards-crashed-when-training-longer-a-pytorch-model/106913/4 "2019-11-06T07:52:36Z")
Around 100°C ---
**Author:** ![bqcaomail](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@bqcaomail](https://forums.developer.nvidia.com/u/bqcaomail)\ **Post date:** [November 6, 2019, 1:25pm UTC](https://forums.developer.nvidia.com/t/rtx-2080-cards-crashed-when-training-longer-a-pytorch-model/106913/5 "2019-11-06T13:25:17Z")
Appreciate