Hi,
We use GPUs on an HPC system which limits the GPU to the user via cgroups.
While using the 450.51.06 CentOS7 driver we had no issues with vulkan, but since updating to 460.27.04 and 460.32.03
we now get an error:
$ vulkaninfo
ERROR: [Loader Message] Code 0 : loader_scanned_icd_add: Could not get 'vkCreateInstance' via 'vk_icdGetInstanceProcAddr' for ICD libGLX_nvidia.so.0
Cannot create Vulkan instance.
This problem is often caused by a faulty installation of the Vulkan driver or attempting to use a GPU that does not support Vulkan.
ERROR at /root/sdk-build/1.2.170.0/source/Vulkan-Tools/vulkaninfo/vulkaninfo.h:666:vkCreateInstance failed with ERROR_INCOMPATIBLE_DRIVER
ssh’ing to the machine directly does not have this issue
$ vulkaninfo | head
'DISPLAY' environment variable not set... skipping surface info
==========
VULKANINFO
==========
Vulkan Instance Version: 1.2.170
Instance Extensions: count = 17
===============================
VK_EXT_acquire_xlib_display : extension revision 1
If all GPUs on the machine are requested, it works OK
The issue can be replicated without an HPC scheduler system by limiting access to one of the GPUs for example:
# echo "c 195:1 rw" > devices.deny
$ nvidia-smi -L
GPU 0: Tesla K40c (UUID: GPU-726c6795-9380-37b0-6719-91ebd50ea1af)
$ module load gcc use.own vulkan170
$ vulkaninfo
ERROR: [Loader Message] Code 0 : loader_scanned_icd_add: Could not get 'vkCreateInstance' via 'vk_icdGetInstanceProcAddr' for ICD libGLX_nvidia.so.0
Cannot create Vulkan instance.
This problem is often caused by a faulty installation of the Vulkan driver or attempting to use a GPU that does not support Vulkan.
ERROR at /root/sdk-build/1.2.170.0/source/Vulkan-Tools/vulkaninfo/vulkaninfo.h:666:vkCreateInstance failed with ERROR_INCOMPATIBLE_DRIVER
We are using V100 and K40c /K80 cards and has happened on all the cuda 11.2 driver installs.