Hello,
We have random crashes/reboots on our GPU-enabled RHEL7 servers. I believe it’s related to nvidia, but not 100% sure. Wondering if anyone else has experienced similar failures.
Thank you!
3.10.0-1160.21.1.el7.x86_64
nvidia-driver-latest-460.32.03-1.el7.x86_64
GeForce RTX 2080 Ti
CPUS: 48
DATE: Thu Apr 8 04:56:42 2021
UPTIME: 4 days, 16:06:26
LOAD AVERAGE: 4.26, 4.50, 4.64#1 SMP Tue Mar 16 13:23:19 EDT 2021
crash> bt#0 [ffff9376fd7839f0] machine_kexec at ffffffffae0662c4#1 [ffff9376fd783a50] kimage_load_segment at ffffffffae122732#2 [ffff9376fd783b20] __crash_kexec at ffffffffae122820#3 [ffff9376fd783b38] oops_end at ffffffffae78d798#4 [ffff9376fd783b60] no_context at ffffffffae075d14#5 [ffff9376fd783bb0] __bad_area_nosemaphore at ffffffffae075fe2#6 [ffff9376fd783c00] bad_area_nosemaphore at ffffffffae076104#7 [ffff9376fd783c10] __do_page_fault at ffffffffae790750#8 [ffff9376fd783c80] do_page_fault at ffffffffae790975#9 [ffff9376fd783cb0] page_fault at ffffffffae78c778#10 [ffff9376fd783d68] os_get_current_tick at ffffffffc6cf415c [nvidia]#11 [ffff9376fd783da0] _nv009219rm at ffffffffc6d25761 [nvidia]#12 [ffff9376fd783dd0] _nv036101rm at ffffffffc6d2657c [nvidia]#13 [ffff9376fd783df0] _nv032953rm at ffffffffc6d6f883 [nvidia]#14 [ffff9376fd783e20] rm_run_rc_callback at ffffffffc75af4e6 [nvidia]#15 [ffff9376fd783e40] nvidia_rc_timer_callback at ffffffffc6ce4fdc [nvidia]#16 [ffff9376fd783e58] nv_timer_callback_typed_data at ffffffffc6ce447d [nvidia]#17 [ffff9376fd783e68] call_timer_fn at ffffffffae0abcf8#18 [ffff9376fd783ea0] run_timer_softirq at ffffffffae0ae30d#19 [ffff9376fd783f18] __do_softirq at ffffffffae0a4b35#20 [ffff9376fd783f88] call_softirq at ffffffffae7994ec#21 [ffff9376fd783fa0] do_softirq at ffffffffae02f715#22 [ffff9376fd783fd8] smp_apic_timer_interrupt at ffffffffae79aa88#23 [ffff9376fd783ff0] apic_timer_interrupt at ffffffffae796fba#24 [ffff93492fb93e88] apic_timer_interrupt at ffffffffae796fba#25 [ffff93492fb93f50] auditsys at ffffffffae79612d
Thanks for reporting this. It’s being tracked in internal bug number 3279571. The bug tracker is not public but you can refer to this number in future correspondence.
damr
June 18, 2021, 7:36am
3
Hi @aplattner , I have the same issue as describe above on multiple Centos7 systems running the versions below
NVIDIA-SMI 465.19.01 Driver Version: 465.19.01 CUDA Version: 11.3
Would it be possible to have an update on the bug 3279571 ? any workaround or is that fix in another release ?
Thanks
I face the same problem on eulerosv2r7, Driver Version: 460.80, @aplattner does it fixed in another release?
nadas1
August 3, 2021, 12:01pm
5
Hi @aplattner We see this also and have for some time. At least 2080 RTI, RTX 6000, RTX 8000 and A6000 GPUs. Various driver versions, most recently 460.73.01. Centos= 3.10.0-1160.31.1.el7.x86_64 #1 . Seen on different Supermicro systems.
Thanks,
edw
August 13, 2021, 11:01am
6
Hi @aplattner , we are facing the same issue with driver ver. 460.73.01 or 460.32.03.
RHEL 7.6 with Kernel v. 3.10.0-1160.31.1.el7.x86_64
Driver ver. 460.73.01 or 460.32.03
HPE XL270d
Tesla V100 (8x GPU Cards)
We cant take the machines in production if we don’t have a fix. Delaying projects is no fun. Thanks.
edw
August 23, 2021, 5:41am
7
Solution for me: I upgraded the system to RHEL 7.9 with 7.9 Kernel - 3.10.0-1160.31.1.el7.x86_64 and NVIDIA driver version 460.32.03 runs stable. 5 days now without crash. It is interesting that I didn’t get any help from NVIDIA (enterprise) because I don’t run a grid version and from HEP because this constellation is not supported.
Update: Problem not resolved… crashed after 5 days…
Any update about the bug: 3279571?
That bug should have been fixed in 470.94. Are you experiencing a crash? If so, please generate and attach a new bug report log.
normspe
January 13, 2022, 7:50pm
10
We resolved our problem by enabling persistence mode. It may seem trivial, but we had absolutely no crashes with pm enabled (or using daemon), and as soon as pm was disabled, crashes showed up again. Re-enabled pm and no crashes for months now.
EDIT: Just to clarify, we saw crashes for months on several systems, but once we enabled pm, the crashes were gone on all systems. We disabled pm just to confirm our findings.
1 Like
sqy0727
September 28, 2022, 7:45am
11
driver 515.65.01 the similar problem on cnetos = 3.10.0-1160.el7.x86_64 RTX3080 GPU
[73501.074032] BUG: unable to handle kernel paging request at 0000000050769420 IP: [] cpuacct_charge+0x2d/0x50 Thread overran stack, or stack corrupted #1 ] SMPCPU: 32 PID: 110544 Comm: irq/312-nvidia Kdump: loaded Tainted: P OE ------------ 3.10.0-1160.el7.x86_64 #1
NVIDIA-SMI 510.47.03 Driver Version: 510.47.03 CUDA Version: 11.6 ### CentOS 7.9 kernel 3.10.0-1160.83.1.el7.x86_64 GPU A6000 total 8 cards [ 3029.597801] BUG: unable to handle kernel paging request at fffffffd97e12550 #1 ] SMP[ 3029.598232] CPU: 63 PID: 9423 Comm: irq/240-nvidia Kdump: loaded Tainted: P OE ------------ 3.10.0-1160.83.1.el7.x86_64 #1