Running Vulkan Inside Singularity Container

Hi,
I am trying to run a program that uses vulkan inside of a singularity container (for an HPC environment). I have been able to build a docker container that is able to run vulkan (vulkaninfo gives correct output), but when I pull that container to Singularity or re-build it from scratch using a singularity definition file based on the Dockerfile, I cannot seem to get vulkan to run. I get the following error:

===========
VULKAN INFO
===========

Vulkan Instance Version: 1.1.70

No protocol specified
No protocol specified
No protocol specified
No protocol specified
No protocol specified
No protocol specified
ERROR: [Loader Message] Code 0 : loader_scanned_icd_add: Could not get 'vkCreateInstance' via 'vk_icdGetInstanceProcAddr' for ICD libGLX_nvidia.so.450.66
Cannot create Vulkan instance.
/build/vulkan-UL09PJ/vulkan-1.1.70+dfsg1/demos/vulkaninfo.c:768: failed with VK_ERROR_INCOMPATIBLE_DRIVER 

Does anyone have a working example of running Vulkan inside a Singularity container.
For reference, my Singularity recipe is the following:

Bootstrap: docker
From: nvidia/cuda:10.2-cudnn7-devel-ubuntu18.04

%files
	nvidia_icd.json
%post
	apt-get update -q \
    && DEBIAN_FRONTEND=noninteractive apt-get install -y \
    curl \
    git \
    libgl1-mesa-dev \
    libgl1-mesa-glx \
    libglew-dev \
    libosmesa6-dev \
    software-properties-common \
    net-tools \
    unzip \
    vim \
    virtualenv \
    wget \
    xpra \
    xserver-xorg-dev \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*
        apt-get update -q \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y \
libxcursor-dev \
libxrandr-dev \
libxinerama-dev \
libxi-dev \
mesa-common-dev \
zip \
unzip \
make \
g++ \
python2.7 \
wget \
vulkan-utils \
mesa-vulkan-drivers \
libopenmpi-dev \
libspatialindex-dev \
python3-dev \
python3-tk \
libpython3.7 \
libcairo2-dev \
python3-cairo-dev
        apt-get update \
 && apt-get install -y --no-install-recommends \
 libxcursor-dev \
libxrandr-dev \
 libxinerama-dev \
 libxi-dev \
 mesa-common-dev \
 zip \
 unzip \
 make \
 g++ \
 python2.7 \
 wget \
 vulkan-utils \
 mesa-vulkan-drivers \
 libopenmpi-dev
        apt-get update \
 && apt-get install -y --no-install-recommends \
    build-essential \
    curl \
    software-properties-common \
 && rm -rf /var/lib/apt/lists/*
        apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y git


        DEBIAN_FRONTEND=noninteractive add-apt-repository --yes ppa:deadsnakes/ppa && apt-get update
        DEBIAN_FRONTEND=noninteractive apt-get install --yes python3.7-dev python3.7 python3-pip
        python3.7 -m pip install -q --no-cache-dir -U pip
        python3.7 -m pip install -q --no-cache-dir -U setuptools
        python3.7 -m pip install -q --no-cache-dir ipython requests numpy pandas quandl
      	
	cat nvidia_icd.json
	apt-get update
	apt-get install -y --no-install-recommends libvulkan1 vulkan-utils mesa-vulkan-drivers
	cp nvidia_icd.json ${SINGULARITY_ROOTFS}/usr/share/vulkan/icd.d/
	NVIDIA_DRIVER_VERSION=450.66
	sed -i "s/\${nvidia_driver}/${NVIDIA_DRIVER_VERSION}/" /usr/share/vulkan/icd.d/nvidia_icd.json
	cat /usr/share/vulkan/icd.d/nvidia_icd.json
	wget -q http://us.download.nvidia.com/XFree86/Linux-x86_64/${NVIDIA_DRIVER_VERSION}/NVIDIA-Linux-x86_64-${NVIDIA_DRIVER_VERSION}.run
	sh NVIDIA-Linux-x86_64-${NVIDIA_DRIVER_VERSION}.run --extract-only
	cp -r NVIDIA-Linux-x86_64-${NVIDIA_DRIVER_VERSION}/* /usr/lib/x86_64-linux-gnu/

Hi! I’ve had good results with using the nvidia/vulkan:1.1.121-cuda-10.1-alpha image for using Vulkan in a CI system. Have you tried that image by chance? As additional packages, I’ve put libvulkan1, libvulkan-dev, and vulkan-utils.

Hope this helps 👍

Hi!
Thanks so much for the response, it was very helpful. I was able to get the following singularity file to run vulkaninfo on my local machine:

Bootstrap: docker
From: nvidia/vulkan:1.1.121-cuda-10.1-alpha
%post
 apt-get update -q \
    && DEBIAN_FRONTEND=noninteractive apt-get install -y vulkan-utils libvulkan-dev libvulkan1

This runs on a machine with nvidia driver version 450.66 and has vulkan installed and is running ubuntu 20.04.
I am trying to run this singularity container on an HPC machine that has the same driver version, but does not have vulkan installed as the machine is running CentOS 6. On that machine the singularity image fails to run and vulkan gives the following error message:

===========
VULKAN INFO
===========

Vulkan API Version: 1.0.61

INFO: [Loader Message] Code 0 : ReadDataFilesInSearchPaths: Searching the following paths for manifest files: /etc/xdg/vulkan/implicit_layer.d:/etc/opt/vulkan/build/Vulkan-Loader/build/install/vulkan/implicit_layer.d:/etc/vulkan/implicit_layer.d:/usr/local/share/vulkan/implicit_layer.d:/usr/share/vulkan/implicit_layer.d:/home/mdalal/.local/share/vulkan/implicit_layer.d

INFO: [Loader Message] Code 0 : ReadDataFilesInSearchPaths: Searching the following paths for manifest files: /etc/xdg/vulkan/explicit_layer.d:/etc/opt/vulkan/build/Vulkan-Loader/build/install/vulkan/explicit_layer.d:/etc/vulkan/explicit_layer.d:/usr/local/share/vulkan/explicit_layer.d:/usr/share/vulkan/explicit_layer.d:/home/mdalal/.local/share/vulkan/explicit_layer.d

INFO: [Loader Message] Code 0 : Found manifest file /usr/local/share/vulkan/explicit_layer.d/VkLayer_core_validation.json, version "1.1.0"
INFO: [Loader Message] Code 0 : Found manifest file /usr/local/share/vulkan/explicit_layer.d/VkLayer_khronos_validation.json, version "1.1.0"
INFO: [Loader Message] Code 0 : Found manifest file /usr/local/share/vulkan/explicit_layer.d/VkLayer_object_lifetimes.json, version "1.1.0"
INFO: [Loader Message] Code 0 : Found manifest file /usr/local/share/vulkan/explicit_layer.d/VkLayer_standard_validation.json, version "1.1.1"
INFO: [Loader Message] Code 0 : Encountered meta-layer VK_LAYER_LUNARG_standard_validation
INFO: [Loader Message] Code 0 : Found manifest file /usr/local/share/vulkan/explicit_layer.d/VkLayer_stateless_validation.json, version "1.1.0"
INFO: [Loader Message] Code 0 : Found manifest file /usr/local/share/vulkan/explicit_layer.d/VkLayer_thread_safety.json, version "1.1.0"
INFO: [Loader Message] Code 0 : Found manifest file /usr/local/share/vulkan/explicit_layer.d/VkLayer_unique_objects.json, version "1.1.0"
INFO: [Loader Message] Code 0 : Found manifest file /usr/share/vulkan/explicit_layer.d/VkLayer_core_validation.json, version "1.1.0"
INFO: [Loader Message] Code 0 : Found manifest file /usr/share/vulkan/explicit_layer.d/VkLayer_object_tracker.json, version "1.1.0"
INFO: [Loader Message] Code 0 : Found manifest file /usr/share/vulkan/explicit_layer.d/VkLayer_parameter_validation.json, version "1.1.0"
INFO: [Loader Message] Code 0 : Found manifest file /usr/share/vulkan/explicit_layer.d/VkLayer_standard_validation.json, version "1.1.1"
INFO: [Loader Message] Code 0 : Encountered meta-layer VK_LAYER_LUNARG_standard_validation
INFO: [Loader Message] Code 0 : Found manifest file /usr/share/vulkan/explicit_layer.d/VkLayer_threading.json, version "1.1.0"
INFO: [Loader Message] Code 0 : Found manifest file /usr/share/vulkan/explicit_layer.d/VkLayer_unique_objects.json, version "1.1.0"
INFO: [Loader Message] Code 0 : Meta-layer VK_LAYER_LUNARG_standard_validation component layer VK_LAYER_KHRONOS_validation adding instance extension VK_EXT_debug_report
INFO: [Loader Message] Code 0 : Meta-layer VK_LAYER_LUNARG_standard_validation component layer VK_LAYER_KHRONOS_validation adding device extension VK_EXT_debug_marker
INFO: [Loader Message] Code 0 : Meta-layer VK_LAYER_LUNARG_standard_validation component layer VK_LAYER_KHRONOS_validation adding device extension VK_EXT_validation_cache
INFO: [Loader Message] Code 0 : Meta-layer VK_LAYER_LUNARG_standard_validation all 1 component layers appear to be valid.
WARNING: [Loader Message] Code 0 : verifyMetaLayerComponentLayers: Meta-layer uses API version 1.0, but component layer 0 uses API version 1.1.  Skipping this layer.
INFO: [Loader Message] Code 0 : Removing meta-layer VK_LAYER_LUNARG_standard_validation from instance layer list since it appears invalid.
INFO: [Loader Message] Code 0 : ReadDataFilesInSearchPaths: Searching the following paths for manifest files: /etc/xdg/vulkan/icd.d:/etc/opt/vulkan/build/Vulkan-Loader/build/install/vulkan/icd.d:/etc/vulkan/icd.d:/usr/local/share/vulkan/icd.d:/usr/share/vulkan/icd.d:/home/mdalal/.local/share/vulkan/icd.d

INFO: [Loader Message] Code 0 : Found ICD manifest file /etc/vulkan/icd.d/nvidia_icd.json, version "1.0.0"
INFO: [Loader Message] Code 0 : Searching for ICD drivers named libGLX_nvidia.so.0
ERROR: [Loader Message] Code 0 : loader_scanned_icd_add: Could not get 'vkCreateInstance' via 'vk_icdGetInstanceProcAddr' for ICD libGLX_nvidia.so.0
INFO: [Loader Message] Code 0 : Found ICD manifest file /usr/share/vulkan/icd.d/intel_icd.x86_64.json, version "1.0.0"
INFO: [Loader Message] Code 0 : Searching for ICD drivers named /usr/lib/x86_64-linux-gnu/libvulkan_intel.so
INFO: [Loader Message] Code 0 : Found ICD manifest file /usr/share/vulkan/icd.d/radeon_icd.x86_64.json, version "1.0.0"
INFO: [Loader Message] Code 0 : Searching for ICD drivers named /usr/lib/x86_64-linux-gnu/libvulkan_radeon.so
INFO: [Loader Message] Code 0 : Build ICD instance extension list
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_external_fence_capabilities (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_external_memory_capabilities (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_external_semaphore_capabilities (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_get_physical_device_properties2 (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_get_surface_capabilities2 (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_surface (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.25
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_wayland_surface (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.6
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_xcb_surface (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.6
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_xlib_surface (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.6
INFO: [Loader Message] Code 0 : Instance Extension: VK_EXT_debug_report (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.8
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_external_fence_capabilities (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_external_memory_capabilities (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_external_semaphore_capabilities (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_get_physical_device_properties2 (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_get_surface_capabilities2 (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_surface (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.25
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_wayland_surface (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.6
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_xcb_surface (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.6
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_xlib_surface (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.6
INFO: [Loader Message] Code 0 : Instance Extension: VK_EXT_debug_report (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.9
INFO: [Loader Message] Code 0 : Build ICD instance extension list
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_external_fence_capabilities (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_external_memory_capabilities (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_external_semaphore_capabilities (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_get_physical_device_properties2 (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_get_surface_capabilities2 (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_surface (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.25
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_wayland_surface (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.6
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_xcb_surface (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.6
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_xlib_surface (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.6
INFO: [Loader Message] Code 0 : Instance Extension: VK_EXT_debug_report (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.8
INFO: [Loader Message] Code 0 : Build ICD instance extension list
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_external_fence_capabilities (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_external_memory_capabilities (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_external_semaphore_capabilities (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_get_physical_device_properties2 (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_get_surface_capabilities2 (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_surface (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.25
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_wayland_surface (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.6
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_xcb_surface (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.6
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_xlib_surface (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.6
INFO: [Loader Message] Code 0 : Instance Extension: VK_EXT_debug_report (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.9

Instance Extensions:
====================
Instance Extensions count = 11
 VK_KHR_external_fence_capabilities  : extension revision  1
 VK_KHR_external_memory_capabilities : extension revision  1
 VK_KHR_external_semaphore_capabilities: extension revision  1
 VK_KHR_get_physical_device_properties2: extension revision  1
 VK_KHR_get_surface_capabilities2    : extension revision  1
 VK_KHR_surface                      : extension revision 25
 VK_KHR_wayland_surface              : extension revision  6
 VK_KHR_xcb_surface                  : extension revision  6
 VK_KHR_xlib_surface                 : extension revision  6
 VK_EXT_debug_report                 : extension revision  8
 VK_EXT_debug_utils                  : extension revision  1
/build/vulkan-YO5iw0/vulkan-1.0.61.1+dfsg1/demos/vulkaninfo.c:1722: failed with VK_ERROR_INITIALIZATION_FAILED

Is this error due to the fact that vulkan is not installed on the host system? If so is there anyway to get around this - it doesn’t seem like there is a way to install vulkan on centos 6 - the sys admin tried but couldn’t find a way. So basically, is there any way to run vulkan on a singularity image on a host that does not have vulkan.