Dell R730’s with 2 K1 cards each with Xendesktop 7.6
We have a pool of 3 servers for our XenDesktop environment and use vgpu with the latest drivers. Today, the users that have a vgpu connected all locked up. The users that did not have a vgpu card attached were fine. I could not access the console via XenCenter or ssh into the server. We tried to shut down a VM and it would hang. I am not sure if it was related but we have a couple XenAPP servers using vgpu that I was able to rdp into and shut them down. At the same time, my console and ssh started responding and all the users where able to log back on to their session. I check the logs on the XS server and noticed a few errors in the kernel.log related to nvidia the same time we had the issue. Perhaps we have a bad card?
BTW, are there any nvidia specific logs generated on XenServer?
Sep 9 08:25:40 xenserv1 kernel: [1531154.743329] NVRM: Xid (PCI:0000:86:00): 38, 003f 0000a097 00000000 00000000 00000000 00000000
Sep 9 08:25:40 xenserv1 kernel: [1531155.252162] NVRM: Xid (PCI:0000:86:00): 43, Ch 00000043, engmask 00000101
Sep 9 08:25:45 xenserv1 kernel: [1531160.223941] NVRM: Xid (PCI:0000:86:00): 43, Ch 0000003f, engmask 00000101
Sep 9 08:30:37 xenserv1 kernel: [1531451.953662] INFO: task vgpu:25831 blocked for more than 120 seconds.
Sep 9 08:30:37 xenserv1 kernel: [1531451.953673] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 9 08:30:37 xenserv1 kernel: [1531451.953677] vgpu D 0000000000000000 0 25831 1 0x00000000
Sep 9 08:30:37 xenserv1 kernel: [1531451.953682] ffff88002eb67b28 0000000000000282 ffff88002eb67a68 ffffffff81007f13
Sep 9 08:30:37 xenserv1 kernel: [1531451.953686] ffff88018880b110 0000000000013fc0 ffff88005c579710 ffffffff81a13420
Sep 9 08:30:37 xenserv1 kernel: [1531451.953689] 0000000000000001 0000000000000001 0000000000000000 0000000000000001
Sep 9 08:30:37 xenserv1 kernel: [1531451.953692] Call Trace:
Sep 9 08:30:37 xenserv1 kernel: [1531451.953702] [<ffffffff81007f13>] ? xen_flush_tlb_all+0x163/0x170
Sep 9 08:30:37 xenserv1 kernel: [1531451.953705] [<ffffffff81007fec>] ? __xen_remap_domain_mfn_range+0xcc/0xf0
Sep 9 08:30:37 xenserv1 kernel: [1531451.953710] [<ffffffff81505ab5>] schedule+0x55/0x60
Sep 9 08:30:37 xenserv1 kernel: [1531451.953712] [<ffffffff815038ba>] schedule_timeout+0x3a/0x200
Sep 9 08:30:37 xenserv1 kernel: [1531451.953715] [<ffffffff81504866>] __down+0x76/0xb0
Sep 9 08:30:37 xenserv1 kernel: [1531451.953719] [<ffffffff8107ce88>] down+0x38/0x50
Sep 9 08:30:37 xenserv1 kernel: [1531451.953781] [<ffffffffa081f127>] os_acquire_mutex+0x37/0x50 [nvidia]
Sep 9 08:30:37 xenserv1 kernel: [1531451.953817] [<ffffffffa07ff2a8>] _nv010795rm+0x18/0x30 [nvidia]
Sep 9 08:30:37 xenserv1 kernel: [1531451.953867] [<ffffffffa077378d>] ? _nv000227rm+0xd/0x30 [nvidia]
Sep 9 08:30:37 xenserv1 kernel: [1531451.953911] [<ffffffffa07b9d5d>] ? _nv012224rm+0x3d/0x120 [nvidia]
Sep 9 08:30:37 xenserv1 kernel: [1531451.953956] [<ffffffffa07b7601>] ? _nv012268rm+0x531/0x620 [nvidia]
Sep 9 08:30:37 xenserv1 kernel: [1531451.953999] [<ffffffffa07b7702>] ? _nv000645rm+0x12/0x20 [nvidia]
Sep 9 08:30:37 xenserv1 kernel: [1531451.954044] [<ffffffffa079b265>] ? _nv001518rm+0x2415/0x3930 [nvidia]
Sep 9 08:30:37 xenserv1 kernel: [1531451.954081] [<ffffffffa07faf80>] ? _nv000692rm+0x700/0x860 [nvidia]
Sep 9 08:30:37 xenserv1 kernel: [1531451.954115] [<ffffffffa0804ee3>] ? rm_ioctl+0x73/0x100 [nvidia]
Sep 9 08:30:37 xenserv1 kernel: [1531451.954120] [<ffffffff81157900>] ? __kmalloc+0x20/0x170
Sep 9 08:30:37 xenserv1 kernel: [1531451.954154] [<ffffffffa0816941>] ? nvidia_ioctl+0x431/0x4c0 [nvidia]
Sep 9 08:30:37 xenserv1 kernel: [1531451.954159] [<ffffffff810440a3>] ? bad_area_nosemaphore+0x13/0x20
Sep 9 08:30:37 xenserv1 kernel: [1531451.954181] [<ffffffffa03350d9>] ? nvidia_frontend_ioctl+0x39/0x80 [nvidia]
Sep 9 08:30:37 xenserv1 kernel: [1531451.954204] [<ffffffffa033516d>] ? nvidia_frontend_unlocked_ioctl+0x1d/0x30 [nvidia]
Sep 9 08:30:37 xenserv1 kernel: [1531451.954208] [<ffffffff81175b0d>] ? vfs_ioctl+0x1d/0x40
Sep 9 08:30:37 xenserv1 kernel: [1531451.954210] [<ffffffff8117651b>] ? do_vfs_ioctl+0x4bb/0x520
Sep 9 08:30:37 xenserv1 kernel: [1531451.954213] [<ffffffff811765ed>] ? SyS_ioctl+0x6d/0xa0
Sep 9 08:30:37 xenserv1 kernel: [1531451.954218] [<ffffffff8150ee99>] ? system_call_fastpath+0x16/0x1b
Sep 9 08:30:37 xenserv1 kernel: [1531451.954231] INFO: task kworker/3:2:27753 blocked for more than 120 seconds.
Sep 9 08:30:37 xenserv1 kernel: [1531451.954235] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.