I’m running a test suite on a Vulkan application, which involves repeatedly creating and destroying a single VkInstance. Each time the instance is destroyed, two file descriptors (/dev/nvidiactl and /dev/dri/renderD128) are leaked. This leads to vkCreateDevice failing halfway through with VK_ERROR_INITIALIZATION_FAILED when the fd limit is reached:
GENERAL [Loader Message (0)] : terminator_CreateDevice: Failed in ICD /usr/lib/libGLX_nvidia.so.610.43.03 vkCreateDevice call
GENERAL [Loader Message (0)] : vkCreateDevice: Failed to create device chain.
It seems libGLX_nvidia.so is dlclosed each time the instance is destroyed and these fds aren’t cleaned up somehow. VK_LOADER_DISABLE_DYNAMIC_LIBRARY_UNLOADING=1 reliably eliminates the leak.
Environment
- GPU: NVIDIA GeForce RTX 5070 Ti
- Driver: 610.43.03 (nvidia-open kernel modules)
- Kernel: 7.1.3-arch1-2 (Arch Linux)
- Vulkan loader: 1.4.350.1
Reproducer
// gcc -o vkleak vkleak.c -lvulkan -ldl
// ./vkleak -> leaks 2 fds per iteration
// ./vkleak pin -> extra dlopen reference prevents ICD unload; no leak
#include <vulkan/vulkan.h>
#include <stdio.h>
#include <string.h>
#include <dlfcn.h>
#include <dirent.h>
static int count_fds(void) {
DIR *d = opendir("/proc/self/fd");
int n = 0; struct dirent *e;
while ((e = readdir(d))) n++;
closedir(d);
return n - 2;
}
int main(int argc, char **argv) {
if (argc > 1 && strcmp(argv[1], "pin") == 0)
dlopen("libGLX_nvidia.so.0", RTLD_NOW | RTLD_LOCAL);
for (int i = 1; i <= 30; i++) {
VkApplicationInfo app = { .sType = VK_STRUCTURE_TYPE_APPLICATION_INFO,
.apiVersion = VK_API_VERSION_1_3 };
VkInstanceCreateInfo ici = { .sType = VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO,
.pApplicationInfo = &app };
VkInstance inst;
vkCreateInstance(&ici, NULL, &inst);
// do nothing with the instance, just destroy it
vkDestroyInstance(inst, NULL);
if (i % 10 == 0) printf("iter %d: fds=%d\n", i, count_fds());
}
return 0;
}
Output:
$ ./vkleak
iter 10: fds=25
iter 20: fds=45
iter 30: fds=65
$ ./vkleak pin
iter 10: fds=21
iter 20: fds=21
iter 30: fds=21
Doing something with the instance or creating/destroying multiple instances changes nothing.
Source
Layers are not involved (reproduces with VK_LOADER_LAYERS_DISABLE='*').
Looking at strace -k, the leaked opens seem to be these:
openat(AT_FDCWD, "/dev/nvidiactl", O_RDWR|O_CLOEXEC) = 14
> /usr/lib/libc.so.6() [0x9405e]
> /usr/lib/libc.so.6() [0x94084]
> /usr/lib/libc.so.6(__open64+0x57) [0x10e467]
> /usr/lib/libnvidia-glcore.so.610.43.03() [0x1023f31]
> /usr/lib/libnvidia-glcore.so.610.43.03() [0xf6ad1e]
> /usr/lib/libnvidia-glcore.so.610.43.03() [0xf688b4]
> /usr/lib/libnvidia-glcore.so.610.43.03() [0xa06c28]
> /usr/lib/libnvidia-glcore.so.610.43.03() [0xfb90f8]
> /usr/lib/libnvidia-glcore.so.610.43.03() [0xfd0a40] (*)
> /usr/lib/libnvidia-glcore.so.610.43.03() [0x9fe36d]
> /usr/lib/libGLX_nvidia.so.610.43.03() [0x553b7]
> /usr/lib/libGLX_nvidia.so.610.43.03() [0x55538]
> /usr/lib/libGLX_nvidia.so.610.43.03() [0x7f034]
> /usr/lib/libGLX_nvidia.so.610.43.03() [0x9e603]
> /usr/lib/libGLX_nvidia.so.610.43.03() [0x9f9d2]
> /usr/lib/libvulkan.so.1.4.350() [0x20c43]
> /usr/lib/libvulkan.so.1.4.350() [0x266e2]
> /usr/lib/libvulkan.so.1.4.350(vkCreateInstance+0x1260) [0x51b60]
> /tmp/vkleak(main+0xc9) [0x12d5]
...
openat(AT_FDCWD, "/dev/dri/renderD128", O_RDWR|O_CLOEXEC) = 15
> /usr/lib/libc.so.6() [0x9405e]
> /usr/lib/libc.so.6() [0x94084]
> /usr/lib/libc.so.6(__open64+0x57) [0x10e467]
> /usr/lib/libnvidia-glcore.so.610.43.03() [0xa17ed1]
> /usr/lib/libnvidia-glcore.so.610.43.03() [0xa0693d]
> /usr/lib/libnvidia-glcore.so.610.43.03() [0xfb90f8]
... rest identical to above (*)
Disclaimer: I used Claude Fable for finding the cause, but manually checked and reproduced all results described above.