glImportMemoryFdEXT erroring with GL_OUT_OF_MEMORY (in IPC scenario)

Platform: Linux-x86_64
Kernel: 5.8.0-49-generic (ubuntu 20.04)
API/Driver: GL 4.6.0 NVIDIA 460.56
Card: RTX 3090

I am attempting inter-process image (texture) sharing between Vulkan in one process and GL in another (under the same user). This is using GL’s and VK’s external memory extensions.

The file descriptor is generated in the vulkan process using vkGetMemoryFdKHR, passed into the GL application via a tcp socket.
The file descriptor is duped in the GL process via

auto newfd_path = formatpath("/proc/%d/fd/%d",original_pid,original_fd);
translated_fd = open(newfd_path.c_str(),O_RDWR);

pidfd_getfd was not available in headers so I hacked one together via:

int _pidfd_getfd(int pidfd, int fd, int flags) { return syscall(__NR_pidfd_getfd, pidfd, fd, flags); }

Unfortunately, this just returned -1 for some reason - hence why I switched to the /proc method.

Using the below snippet of GL code I get the later following output. The error is occuring on the glImportMemoryFdEXT call.

The extension documentation does not mention the meaning of error codes for this gl call…
Should I be using NVidia extensions instead of Khronos extensions for this ?


        printf("MEMOBJECT<%d> fd<%d> w<%d> h<%d> size<%zu>\n", 
               ipcdata->_image_size );


        glImportMemoryFdEXT(mem_object, // mem object
                            ipcdata->_image_size, // image size
                            GL_HANDLE_TYPE_OPAQUE_FD_EXT, // handle type
                            ipcdata->_image_fd); // file descriptor


MEMOBJECT<1> fd<45> w<800> h<600> size<2048000>

Interprocess sharing is fundamentally possible with external objects.
I believe what went wrong in your case is the transfer of the FD into the target process.
If _pidfd_getfd() is not supported, please read up on “unix domain sockets”.
These can be used to pass FDs between processes.