File descriptor leak when Vulkan loader unloads libGLX_nvidia after vkDestroyInstance

I’m running a test suite on a Vulkan application, which involves repeatedly creating and destroying a single VkInstance. Each time the instance is destroyed, two file descriptors (/dev/nvidiactl and /dev/dri/renderD128) are leaked. This leads to vkCreateDevice failing halfway through with VK_ERROR_INITIALIZATION_FAILED when the fd limit is reached:

GENERAL [Loader Message (0)] : terminator_CreateDevice: Failed in ICD /usr/lib/libGLX_nvidia.so.610.43.03 vkCreateDevice call
GENERAL [Loader Message (0)] : vkCreateDevice:  Failed to create device chain.

It seems libGLX_nvidia.so is dlclosed each time the instance is destroyed and these fds aren’t cleaned up somehow. VK_LOADER_DISABLE_DYNAMIC_LIBRARY_UNLOADING=1 reliably eliminates the leak.

Environment

  • GPU: NVIDIA GeForce RTX 5070 Ti
  • Driver: 610.43.03 (nvidia-open kernel modules)
  • Kernel: 7.1.3-arch1-2 (Arch Linux)
  • Vulkan loader: 1.4.350.1

Reproducer

// gcc -o vkleak vkleak.c -lvulkan -ldl
// ./vkleak       -> leaks 2 fds per iteration
// ./vkleak pin   -> extra dlopen reference prevents ICD unload; no leak
#include <vulkan/vulkan.h>
#include <stdio.h>
#include <string.h>
#include <dlfcn.h>
#include <dirent.h>

static int count_fds(void) {
    DIR *d = opendir("/proc/self/fd");
    int n = 0; struct dirent *e;
    while ((e = readdir(d))) n++;
    closedir(d);
    return n - 2;
}

int main(int argc, char **argv) {
    if (argc > 1 && strcmp(argv[1], "pin") == 0)
        dlopen("libGLX_nvidia.so.0", RTLD_NOW | RTLD_LOCAL);
    for (int i = 1; i <= 30; i++) {
        VkApplicationInfo app = { .sType = VK_STRUCTURE_TYPE_APPLICATION_INFO,
                                  .apiVersion = VK_API_VERSION_1_3 };
        VkInstanceCreateInfo ici = { .sType = VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO,
                                     .pApplicationInfo = &app };
        VkInstance inst;
        vkCreateInstance(&ici, NULL, &inst);
        // do nothing with the instance, just destroy it
        vkDestroyInstance(inst, NULL);
        if (i % 10 == 0) printf("iter %d: fds=%d\n", i, count_fds());
    }
    return 0;
}

Output:

$ ./vkleak
iter 10: fds=25
iter 20: fds=45
iter 30: fds=65

$ ./vkleak pin
iter 10: fds=21
iter 20: fds=21
iter 30: fds=21

Doing something with the instance or creating/destroying multiple instances changes nothing.

Source

Layers are not involved (reproduces with VK_LOADER_LAYERS_DISABLE='*').

Looking at strace -k, the leaked opens seem to be these:

openat(AT_FDCWD, "/dev/nvidiactl", O_RDWR|O_CLOEXEC) = 14
 > /usr/lib/libc.so.6() [0x9405e]
 > /usr/lib/libc.so.6() [0x94084]
 > /usr/lib/libc.so.6(__open64+0x57) [0x10e467]
 > /usr/lib/libnvidia-glcore.so.610.43.03() [0x1023f31]
 > /usr/lib/libnvidia-glcore.so.610.43.03() [0xf6ad1e]
 > /usr/lib/libnvidia-glcore.so.610.43.03() [0xf688b4]
 > /usr/lib/libnvidia-glcore.so.610.43.03() [0xa06c28]
 > /usr/lib/libnvidia-glcore.so.610.43.03() [0xfb90f8]
 > /usr/lib/libnvidia-glcore.so.610.43.03() [0xfd0a40]  (*)
 > /usr/lib/libnvidia-glcore.so.610.43.03() [0x9fe36d]
 > /usr/lib/libGLX_nvidia.so.610.43.03() [0x553b7]
 > /usr/lib/libGLX_nvidia.so.610.43.03() [0x55538]
 > /usr/lib/libGLX_nvidia.so.610.43.03() [0x7f034]
 > /usr/lib/libGLX_nvidia.so.610.43.03() [0x9e603]
 > /usr/lib/libGLX_nvidia.so.610.43.03() [0x9f9d2]
 > /usr/lib/libvulkan.so.1.4.350() [0x20c43]
 > /usr/lib/libvulkan.so.1.4.350() [0x266e2]
 > /usr/lib/libvulkan.so.1.4.350(vkCreateInstance+0x1260) [0x51b60]
 > /tmp/vkleak(main+0xc9) [0x12d5]
   ...
openat(AT_FDCWD, "/dev/dri/renderD128", O_RDWR|O_CLOEXEC) = 15
 > /usr/lib/libc.so.6() [0x9405e]
 > /usr/lib/libc.so.6() [0x94084]
 > /usr/lib/libc.so.6(__open64+0x57) [0x10e467]
 > /usr/lib/libnvidia-glcore.so.610.43.03() [0xa17ed1]
 > /usr/lib/libnvidia-glcore.so.610.43.03() [0xa0693d]
 > /usr/lib/libnvidia-glcore.so.610.43.03() [0xfb90f8]
   ... rest identical to above (*)

Disclaimer: I used Claude Fable for finding the cause, but manually checked and reproduced all results described above.

Hi @felix.solcher

Thanks you for writing to us, I have filed a bug 6446591 for tracking purpose.
I will try to reproduce issue internally and will get back to you in required any additional information.

Hi @felix.solcher

I attempted to reproduce the reported file descriptor leak using shared reproducer code but no luck.
Test Configuration

  • GPU: NVIDIA GeForce RTX 5070 Ti

  • OS: Arch Linux

  • Kernel: 7.1.3-arch1-2

  • NVIDIA Driver: 610.43.03

  • Kernel Module: NVIDIA Open Kernel Module

  • Vulkan Loader: Initially 1.4.341.0, later upgraded to 1.4.350.1 to match your config.

  • Could you please help with the following?

    1. Was driver 610.43.03 installed using Arch packages, the NVIDIA .run installer, or another method?
    2. Please share the output of:
      pacman -Q glibc libglvnd nvidia-utils nvidia-open-dkms vulkan-icd-loader

    • 3. Does the issue still reproduce from a clean shell?
    env -i HOME=$HOME PATH=/usr/bin:/bin VK_LOADER_LAYERS_DISABLE=‘*’ ./vkleak

Hi @amrits

Thanks for the quick response! Here’s the requested data:

  1. It’s installed using regular Arch packages

  2. Output of pacman -Q glibc libglvnd nvidia-utils nvidia-open-dkms vulkan-icd-loader:

    glibc 2.43+r37+gfdf10644d6ee-1
    libglvnd 1.7.0-3
    nvidia-utils 610.43.03-2
    error: package 'nvidia-open-dkms' was not found
    vulkan-icd-loader 1.4.350.1-1
    
  3. It does not reproduce. Plain env -i ./vkleak is also fine.

I’ve went through the environment variables that are set and these two cause the issue to reappear:
env -i DISPLAY=:0 XAUTHORITY=/run/user/1000/xauth_xxxxxx ./vkleak.

I’m running wayland through xwayland, not sure if that’s relevant, versions:

wayland 1.25.0-1
xorg-xwayland 24.1.13-1

Let me know if you need more info.

Thanks @felix.solcher for sharing the information.

It helped to duplicate issue and we have also root caused the issue and verified the fix internally.

Fix will be incorporated in future release drivers.