GRID K2 NOT wokring on XenServer 6.5SP1

We have two XenServer hosts in a K2 pool, but one of them does not work. Only K2P machines can start on it. K200, K220, K240, K260 and K280 machines could not start on it. Error message from XenServer is Internal error: xenopsd internal error: Device.Ioemu_failed("vgpu exited unexpectedly"). I tried two version of friver as 367.43 and 367.64, neither of them worked. View log files and obtain below information:

Jan 17 11:24:18 xxxxxxxxxxen032 kernel: [417949.279318] block tda: sector-size: 512/512 capacity: 629145600
Jan 17 11:24:18 xxxxxxxxxxen032 kernel: [417949.478882] device vif428.0 entered promiscuous mode
Jan 17 11:24:18 xxxxxxxxxxen032 kernel: [417949.538774] IPv6: ADDRCONF(NETDEV_UP): vif428.0: link is not ready
Jan 17 11:24:49 xxxxxxxxxxen032 kernel: [417980.526518] xapi7: port 2(vif428.0) entered disabled state
Jan 17 11:24:49 xxxxxxxxxxen032 kernel: [417980.526628] device vif428.0 left promiscuous mode
Jan 17 11:24:49 xxxxxxxxxxen032 kernel: [417980.526630] xapi7: port 2(vif428.0) entered disabled state
Jan 17 11:24:56 xxxxxxxxxxen032 kernel: [417987.488401] NVRM: RmInitAdapter failed! (0x24:0x40:1035)

I want to ask who encountered this kind of issue before, please?
Thank you.

I think it might be best to post on the Citrix forums or raise a support case with them as the errors are all xapi…

GPU Technologies - Discussions might be a good place as the XS team monitor it.

Check the host license and host driver and update the host BIOS is the best I can think of.

worth checking your hypervisor version on 32-bit (6.2 XS and lower) >4GB MMIO needs disabling: http://discussions.citrix.com/topic/351335-every-4th-guest-fails-with-error-vgpu-exited-unexpectedly/

Error can arise if power cable issue: https://support.citrix.com/article/CTX210153

You can also get this error if host has not been rebooted: http://www.florisvanderploeg.com/upgrading-the-nvidia-grid-vgpu-driver-on-xenserver/