Running Vulkan Inside Singularity Container

mdalal · October 21, 2020, 7:52pm

Hi,
I am trying to run a program that uses vulkan inside of a singularity container (for an HPC environment). I have been able to build a docker container that is able to run vulkan (vulkaninfo gives correct output), but when I pull that container to Singularity or re-build it from scratch using a singularity definition file based on the Dockerfile, I cannot seem to get vulkan to run. I get the following error:

===========
VULKAN INFO
===========

Vulkan Instance Version: 1.1.70

No protocol specified
No protocol specified
No protocol specified
No protocol specified
No protocol specified
No protocol specified
ERROR: [Loader Message] Code 0 : loader_scanned_icd_add: Could not get 'vkCreateInstance' via 'vk_icdGetInstanceProcAddr' for ICD libGLX_nvidia.so.450.66
Cannot create Vulkan instance.
/build/vulkan-UL09PJ/vulkan-1.1.70+dfsg1/demos/vulkaninfo.c:768: failed with VK_ERROR_INCOMPATIBLE_DRIVER

Does anyone have a working example of running Vulkan inside a Singularity container.
For reference, my Singularity recipe is the following:

Bootstrap: docker
From: nvidia/cuda:10.2-cudnn7-devel-ubuntu18.04

%files
	nvidia_icd.json
%post
	apt-get update -q \
    && DEBIAN_FRONTEND=noninteractive apt-get install -y \
    curl \
    git \
    libgl1-mesa-dev \
    libgl1-mesa-glx \
    libglew-dev \
    libosmesa6-dev \
    software-properties-common \
    net-tools \
    unzip \
    vim \
    virtualenv \
    wget \
    xpra \
    xserver-xorg-dev \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*
        apt-get update -q \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y \
libxcursor-dev \
libxrandr-dev \
libxinerama-dev \
libxi-dev \
mesa-common-dev \
zip \
unzip \
make \
g++ \
python2.7 \
wget \
vulkan-utils \
mesa-vulkan-drivers \
libopenmpi-dev \
libspatialindex-dev \
python3-dev \
python3-tk \
libpython3.7 \
libcairo2-dev \
python3-cairo-dev
        apt-get update \
 && apt-get install -y --no-install-recommends \
 libxcursor-dev \
libxrandr-dev \
 libxinerama-dev \
 libxi-dev \
 mesa-common-dev \
 zip \
 unzip \
 make \
 g++ \
 python2.7 \
 wget \
 vulkan-utils \
 mesa-vulkan-drivers \
 libopenmpi-dev
        apt-get update \
 && apt-get install -y --no-install-recommends \
    build-essential \
    curl \
    software-properties-common \
 && rm -rf /var/lib/apt/lists/*
        apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y git


        DEBIAN_FRONTEND=noninteractive add-apt-repository --yes ppa:deadsnakes/ppa && apt-get update
        DEBIAN_FRONTEND=noninteractive apt-get install --yes python3.7-dev python3.7 python3-pip
        python3.7 -m pip install -q --no-cache-dir -U pip
        python3.7 -m pip install -q --no-cache-dir -U setuptools
        python3.7 -m pip install -q --no-cache-dir ipython requests numpy pandas quandl
      	
	cat nvidia_icd.json
	apt-get update
	apt-get install -y --no-install-recommends libvulkan1 vulkan-utils mesa-vulkan-drivers
	cp nvidia_icd.json ${SINGULARITY_ROOTFS}/usr/share/vulkan/icd.d/
	NVIDIA_DRIVER_VERSION=450.66
	sed -i "s/\${nvidia_driver}/${NVIDIA_DRIVER_VERSION}/" /usr/share/vulkan/icd.d/nvidia_icd.json
	cat /usr/share/vulkan/icd.d/nvidia_icd.json
	wget -q http://us.download.nvidia.com/XFree86/Linux-x86_64/${NVIDIA_DRIVER_VERSION}/NVIDIA-Linux-x86_64-${NVIDIA_DRIVER_VERSION}.run
	sh NVIDIA-Linux-x86_64-${NVIDIA_DRIVER_VERSION}.run --extract-only
	cp -r NVIDIA-Linux-x86_64-${NVIDIA_DRIVER_VERSION}/* /usr/lib/x86_64-linux-gnu/

ulrik · October 23, 2020, 6:44pm

Hi! I’ve had good results with using the nvidia/vulkan:1.1.121-cuda-10.1-alpha image for using Vulkan in a CI system. Have you tried that image by chance? As additional packages, I’ve put libvulkan1, libvulkan-dev, and vulkan-utils.

Hope this helps 👍

mdalal · November 1, 2020, 6:05am

Hi!
Thanks so much for the response, it was very helpful. I was able to get the following singularity file to run vulkaninfo on my local machine:

Bootstrap: docker
From: nvidia/vulkan:1.1.121-cuda-10.1-alpha
%post
 apt-get update -q \
    && DEBIAN_FRONTEND=noninteractive apt-get install -y vulkan-utils libvulkan-dev libvulkan1

This runs on a machine with nvidia driver version 450.66 and has vulkan installed and is running ubuntu 20.04.
I am trying to run this singularity container on an HPC machine that has the same driver version, but does not have vulkan installed as the machine is running CentOS 6. On that machine the singularity image fails to run and vulkan gives the following error message:

===========
VULKAN INFO
===========

Vulkan API Version: 1.0.61

INFO: [Loader Message] Code 0 : ReadDataFilesInSearchPaths: Searching the following paths for manifest files: /etc/xdg/vulkan/implicit_layer.d:/etc/opt/vulkan/build/Vulkan-Loader/build/install/vulkan/implicit_layer.d:/etc/vulkan/implicit_layer.d:/usr/local/share/vulkan/implicit_layer.d:/usr/share/vulkan/implicit_layer.d:/home/mdalal/.local/share/vulkan/implicit_layer.d

INFO: [Loader Message] Code 0 : ReadDataFilesInSearchPaths: Searching the following paths for manifest files: /etc/xdg/vulkan/explicit_layer.d:/etc/opt/vulkan/build/Vulkan-Loader/build/install/vulkan/explicit_layer.d:/etc/vulkan/explicit_layer.d:/usr/local/share/vulkan/explicit_layer.d:/usr/share/vulkan/explicit_layer.d:/home/mdalal/.local/share/vulkan/explicit_layer.d

INFO: [Loader Message] Code 0 : Found manifest file /usr/local/share/vulkan/explicit_layer.d/VkLayer_core_validation.json, version "1.1.0"
INFO: [Loader Message] Code 0 : Found manifest file /usr/local/share/vulkan/explicit_layer.d/VkLayer_khronos_validation.json, version "1.1.0"
INFO: [Loader Message] Code 0 : Found manifest file /usr/local/share/vulkan/explicit_layer.d/VkLayer_object_lifetimes.json, version "1.1.0"
INFO: [Loader Message] Code 0 : Found manifest file /usr/local/share/vulkan/explicit_layer.d/VkLayer_standard_validation.json, version "1.1.1"
INFO: [Loader Message] Code 0 : Encountered meta-layer VK_LAYER_LUNARG_standard_validation
INFO: [Loader Message] Code 0 : Found manifest file /usr/local/share/vulkan/explicit_layer.d/VkLayer_stateless_validation.json, version "1.1.0"
INFO: [Loader Message] Code 0 : Found manifest file /usr/local/share/vulkan/explicit_layer.d/VkLayer_thread_safety.json, version "1.1.0"
INFO: [Loader Message] Code 0 : Found manifest file /usr/local/share/vulkan/explicit_layer.d/VkLayer_unique_objects.json, version "1.1.0"
INFO: [Loader Message] Code 0 : Found manifest file /usr/share/vulkan/explicit_layer.d/VkLayer_core_validation.json, version "1.1.0"
INFO: [Loader Message] Code 0 : Found manifest file /usr/share/vulkan/explicit_layer.d/VkLayer_object_tracker.json, version "1.1.0"
INFO: [Loader Message] Code 0 : Found manifest file /usr/share/vulkan/explicit_layer.d/VkLayer_parameter_validation.json, version "1.1.0"
INFO: [Loader Message] Code 0 : Found manifest file /usr/share/vulkan/explicit_layer.d/VkLayer_standard_validation.json, version "1.1.1"
INFO: [Loader Message] Code 0 : Encountered meta-layer VK_LAYER_LUNARG_standard_validation
INFO: [Loader Message] Code 0 : Found manifest file /usr/share/vulkan/explicit_layer.d/VkLayer_threading.json, version "1.1.0"
INFO: [Loader Message] Code 0 : Found manifest file /usr/share/vulkan/explicit_layer.d/VkLayer_unique_objects.json, version "1.1.0"
INFO: [Loader Message] Code 0 : Meta-layer VK_LAYER_LUNARG_standard_validation component layer VK_LAYER_KHRONOS_validation adding instance extension VK_EXT_debug_report
INFO: [Loader Message] Code 0 : Meta-layer VK_LAYER_LUNARG_standard_validation component layer VK_LAYER_KHRONOS_validation adding device extension VK_EXT_debug_marker
INFO: [Loader Message] Code 0 : Meta-layer VK_LAYER_LUNARG_standard_validation component layer VK_LAYER_KHRONOS_validation adding device extension VK_EXT_validation_cache
INFO: [Loader Message] Code 0 : Meta-layer VK_LAYER_LUNARG_standard_validation all 1 component layers appear to be valid.
WARNING: [Loader Message] Code 0 : verifyMetaLayerComponentLayers: Meta-layer uses API version 1.0, but component layer 0 uses API version 1.1.  Skipping this layer.
INFO: [Loader Message] Code 0 : Removing meta-layer VK_LAYER_LUNARG_standard_validation from instance layer list since it appears invalid.
INFO: [Loader Message] Code 0 : ReadDataFilesInSearchPaths: Searching the following paths for manifest files: /etc/xdg/vulkan/icd.d:/etc/opt/vulkan/build/Vulkan-Loader/build/install/vulkan/icd.d:/etc/vulkan/icd.d:/usr/local/share/vulkan/icd.d:/usr/share/vulkan/icd.d:/home/mdalal/.local/share/vulkan/icd.d

INFO: [Loader Message] Code 0 : Found ICD manifest file /etc/vulkan/icd.d/nvidia_icd.json, version "1.0.0"
INFO: [Loader Message] Code 0 : Searching for ICD drivers named libGLX_nvidia.so.0
ERROR: [Loader Message] Code 0 : loader_scanned_icd_add: Could not get 'vkCreateInstance' via 'vk_icdGetInstanceProcAddr' for ICD libGLX_nvidia.so.0
INFO: [Loader Message] Code 0 : Found ICD manifest file /usr/share/vulkan/icd.d/intel_icd.x86_64.json, version "1.0.0"
INFO: [Loader Message] Code 0 : Searching for ICD drivers named /usr/lib/x86_64-linux-gnu/libvulkan_intel.so
INFO: [Loader Message] Code 0 : Found ICD manifest file /usr/share/vulkan/icd.d/radeon_icd.x86_64.json, version "1.0.0"
INFO: [Loader Message] Code 0 : Searching for ICD drivers named /usr/lib/x86_64-linux-gnu/libvulkan_radeon.so
INFO: [Loader Message] Code 0 : Build ICD instance extension list
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_external_fence_capabilities (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_external_memory_capabilities (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_external_semaphore_capabilities (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_get_physical_device_properties2 (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_get_surface_capabilities2 (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_surface (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.25
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_wayland_surface (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.6
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_xcb_surface (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.6
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_xlib_surface (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.6
INFO: [Loader Message] Code 0 : Instance Extension: VK_EXT_debug_report (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.8
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_external_fence_capabilities (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_external_memory_capabilities (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_external_semaphore_capabilities (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_get_physical_device_properties2 (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_get_surface_capabilities2 (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_surface (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.25
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_wayland_surface (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.6
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_xcb_surface (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.6
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_xlib_surface (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.6
INFO: [Loader Message] Code 0 : Instance Extension: VK_EXT_debug_report (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.9
INFO: [Loader Message] Code 0 : Build ICD instance extension list
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_external_fence_capabilities (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_external_memory_capabilities (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_external_semaphore_capabilities (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_get_physical_device_properties2 (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_get_surface_capabilities2 (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_surface (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.25
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_wayland_surface (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.6
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_xcb_surface (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.6
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_xlib_surface (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.6
INFO: [Loader Message] Code 0 : Instance Extension: VK_EXT_debug_report (/usr/lib/x86_64-linux-gnu/libvulkan_intel.so) version 0.0.8
INFO: [Loader Message] Code 0 : Build ICD instance extension list
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_external_fence_capabilities (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_external_memory_capabilities (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_external_semaphore_capabilities (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_get_physical_device_properties2 (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_get_surface_capabilities2 (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.1
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_surface (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.25
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_wayland_surface (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.6
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_xcb_surface (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.6
INFO: [Loader Message] Code 0 : Instance Extension: VK_KHR_xlib_surface (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.6
INFO: [Loader Message] Code 0 : Instance Extension: VK_EXT_debug_report (/usr/lib/x86_64-linux-gnu/libvulkan_radeon.so) version 0.0.9

Instance Extensions:
====================
Instance Extensions count = 11
 VK_KHR_external_fence_capabilities  : extension revision  1
 VK_KHR_external_memory_capabilities : extension revision  1
 VK_KHR_external_semaphore_capabilities: extension revision  1
 VK_KHR_get_physical_device_properties2: extension revision  1
 VK_KHR_get_surface_capabilities2    : extension revision  1
 VK_KHR_surface                      : extension revision 25
 VK_KHR_wayland_surface              : extension revision  6
 VK_KHR_xcb_surface                  : extension revision  6
 VK_KHR_xlib_surface                 : extension revision  6
 VK_EXT_debug_report                 : extension revision  8
 VK_EXT_debug_utils                  : extension revision  1
/build/vulkan-YO5iw0/vulkan-1.0.61.1+dfsg1/demos/vulkaninfo.c:1722: failed with VK_ERROR_INITIALIZATION_FAILED

Is this error due to the fact that vulkan is not installed on the host system? If so is there anyway to get around this - it doesn’t seem like there is a way to install vulkan on centos 6 - the sys admin tried but couldn’t find a way. So basically, is there any way to run vulkan on a singularity image on a host that does not have vulkan.

baylyd · March 4, 2021, 10:37pm

I’m also interested in this exact situation. I’m trying to run singularity containers with Vulkan on my HPC. Did you figure anything out? Is there a way to upvote this thread so it becomes more visible?

mdalal · March 5, 2021, 12:57am

Hi, I forgot to update the post but the following singularity container worked for me:

Bootstrap: docker
From: nvidia/vulkan:1.1.121
%post
    apt-get update -q \
    && DEBIAN_FRONTEND=noninteractive apt-get install -y wget vulkan-utils libvulkan-dev libvulkan1
    NVIDIA_DRIVER_VERSION=418.43
    wget -q http://us.download.nvidia.com/XFree86/Linux-x86_64/${NVIDIA_DRIVER_VERSION}/NVIDIA-Linux-x86_64-${NVIDIA_DRIVER_VERSION}.run
    sh NVIDIA-Linux-x86_64-${NVIDIA_DRIVER_VERSION}.run --extract-only
    cp -r NVIDIA-Linux-x86_64-${NVIDIA_DRIVER_VERSION}/* ${SINGULARITY_ROOTFS}/usr/lib/x86_64-linux-gnu/

baylyd · August 23, 2021, 6:06pm

In investigating this further I had a couple more questions. @mdalal does your HPC have an /etc/vulkan/implicit_layer.d/nvidia_layers.json file? My HPC is missing this and we can’t quite figure out why, or what might have gone wrong in the nvidia drivers installation. I notice that also using the singularity build file from above we don’t end up creating a singularity environment in which this nvidia_layers.json file is present, but that could be perhaps because I’m building the container not on a GPU machine?

Anyways, I’d love to get some assistance if possible and then I’d be happy to take on the burden of writing an article about this setup to share the info more broadly.

Thanks for all your time and work!

Topic		Replies	Views
vkCreateInstance error (-9) CUDA Setup and Installation audio2face	6	262	January 20, 2025
Flatpak drivers don't place lvp_icd.x86_64.json file in known location for vulkan_loader Linux	0	570	November 17, 2023
Nvoglv64.dll crashes w/ "Access violation reading location" at vkCreateComputePipelines Vulkan	10	4515	September 15, 2022
Unable to engage Vulkan support for the latest NVIDIA Linux drivers 375.26 in Ubuntu 16.04 LTS Vulkan	4	2747	January 23, 2017
Vulkan Beta driver 418.52.* don't build from Linux kernel 5.0.7 Vulkan	4	1262	July 3, 2019
vulkaninfo doesn't work on ubuntu16.04 with nvidia 396.54.09 driver(GPU Tesla K10) Linux	2	1762	December 19, 2018
vulkan segfaults when running with nvidia 387.x (12/22) and ubuntu 17.10 Vulkan	3	1518	November 15, 2017
Get Vulkan working on TX2? Jetson TX2	4	2085	October 18, 2021
Running Replicator 1.6.3 from inside of kubernetes (microk8s nor minikube) Synthetic Data Generation (SDG) docker , vulkan , synthetic-data , kubernetes , omniverse	7	714	April 2, 2024
Headless Vulkan with multiple GPUs Linux	20	4781	February 11, 2025

Running Vulkan Inside Singularity Container

Related topics