We have Tesla K20Xm GPU on Linux server where we have NICE-DEV and Star-CCM application runs & till last week we we were able to run both the application together without any issue,now its getting crashed when we start Star-CCM application after starting NICE-DEV & Linux dmesg logs looks below
[root@GPUNODE1 ~]# grep -i nvidia-modeset /var/log/messages
Nov 26 18:41:18 GPUNODE1 kernel: [4409342.930437] nvidia-modeset: Allocated GPU:0 (GPU-26fab751-63cc-0e1a-a30a-00a593935319) @ PCI:0000:0d:00.0
Nov 26 18:41:18 GPUNODE1 kernel: [4409342.930820] nvidia-modeset: WARNING: GPU:0: Correcting number of heads for current head configuration (0x00)
Nov 26 18:41:18 GPUNODE1 kernel: [4409342.930994] nvidia-modeset: Freed GPU:0 (GPU-26fab751-63cc-0e1a-a30a-00a593935319) @ PCI:0000:0d:00.0
Nov 26 18:41:18 GPUNODE1 kernel: [4409342.931196] nvidia-modeset: Allocated GPU:0 (GPU-26fab751-63cc-0e1a-a30a-00a593935319) @ PCI:0000:0d:00.0
Nov 26 18:41:18 GPUNODE1 kernel: [4409342.931615] nvidia-modeset: WARNING: GPU:0: Correcting number of heads for current head configuration (0x00)
Nov 26 18:41:18 GPUNODE1 kernel: [4409342.931727] nvidia-modeset: Freed GPU:0 (GPU-26fab751-63cc-0e1a-a30a-00a593935319) @ PCI:0000:0d:00.0
Nov 26 19:16:46 GPUNODE1 kernel: [4411470.473115] nvidia-modeset: Allocated GPU:0 (GPU-26fab751-63cc-0e1a-a30a-00a593935319) @ PCI:0000:0d:00.0
Nov 26 19:16:46 GPUNODE1 kernel: [4411470.473508] nvidia-modeset: WARNING: GPU:0: Correcting number of heads for current head configuration (0x00)
Nov 26 19:16:46 GPUNODE1 kernel: [4411470.473721] nvidia-modeset: Freed GPU:0 (GPU-26fab751-63cc-0e1a-a30a-00a593935319) @ PCI:0000:0d:00.0
[root@GPUNODE1 ~]#
[root@GPUNODE1 ~]# grep -i dcvconfig /var/log/messages
Nov 28 17:00:59 GPUNODE1 kernel: [4576118.700128] dcvconfig[29826]: segfault at 48 ip 00007f69c332f3b9 sp 00007fff0349ea08 error 6 in libnvidia-glcore.so.384.66[7f69c2161000+1889000]
Nov 28 17:01:02 GPUNODE1 kernel: [4576121.685529] dcvconfig[29893]: segfault at 48 ip 00007efef312b3b9 sp 00007ffe09072ae8 error 6 in libnvidia-glcore.so.384.66[7efef1f5d000+1889000]
Nov 28 17:01:05 GPUNODE1 kernel: [4576124.687451] dcvconfig[30015]: segfault at 48 ip 00007f5ed732f3b9 sp 00007ffcf05631a8 error 6 in libnvidia-glcore.so.384.66[7f5ed6161000+1889000]
Nov 28 17:01:08 GPUNODE1 kernel: [4576127.686131] dcvconfig[30118]: segfault at 48 ip 00007efc2332f3b9 sp 00007ffc2bf9fb08 error 6 in libnvidia-glcore.so.384.66[7efc22161000+1889000]
Nov 28 17:01:11 GPUNODE1 kernel: [4576130.688272] dcvconfig[30179]: segfault at 48 ip 00007f22c99243b9 sp 00007fffd1b198f8 error 6 in libnvidia-glcore.so.384.66[7f22c8756000+1889000]
Nov 28 17:01:14 GPUNODE1 kernel: [4576133.688044] dcvconfig[30279]: segfault at 48 ip 00007f4c3efee3b9 sp 00007ffedb78ba28 error 6 in libnvidia-glcore.so.384.66[7f4c3de20000+1889000]
Nov 28 17:01:17 GPUNODE1 kernel: [4576136.692254] dcvconfig[30339]: segfault at 48 ip 00007f88c732f3b9 sp 00007ffe475f3bb8 error 6 in libnvidia-glcore.so.384.66[7f88c6161000+1889000]