I am importing NVBufSurfaces (created for a v4l2 dec pipeline) into Vulkan by passing the bufferDesc from the surface to fd of VkImportMemoryFdInfoKHR.
This works great for a single stream.
Once I spin up a second stream in parallel, things can start to get wonky. Sometimes NVBufSurface will return fds for the second stream that are the same fd index as the first stream. It seems that vulkan takes ownership of these dma buffers and NVBufSurface thinks they are up for grabs? This results in issues between the two streams, and data from one gets smeared onto the other. This wonkiness doesnt happen every time, sometimes things work out fine and we get two streams in parallel working as intended.
I used to use nvbuf_utils.h on Jetpack 4.6 and things worked perfectly fine. Since moving to nvbufsurface.h and Jetpack 5.1 this issue started to appear.
Note: Since this issue is inconsistently appearing, I do two things to make it pop up more:
- I add a 750ms sleep to the first stream to have it wait a bit till the second stream starts to have conditions that are a bit more parallel, this causes the issue to appear more often, so I currently believe there is some sort of race condition happening somewhere.
- I increase the buffer count. I normally run with 4 buffers.
Am I using the wrong memory type? Or is this functionality just not supported? Or should I be forgoing NvBufSurface entirely and create memory with vulkan, export those as DMA buffers to v4l2 directly?
NVBufSurface is created with params as such
bool create_nvbufsurface(int* fd)
{
NvBufSurface* nvbuf_surf = NULL;
NvBufSurfaceAllocateParams dma_buf_create_params = {0};
dma_buf_create_params.params.width = vd->width;
dma_buf_create_params.params.height = vd->height;
dma_buf_create_params.params.layout = NVBUF_LAYOUT_BLOCK_LINEAR;
dma_buf_create_params.params.memType = NVBUF_MEM_SURFACE_ARRAY;
dma_buf_create_params.memtag = NvBufSurfaceTag_VIDEO_DEC;
dma_buf_create_params.params.colorFormat = NVBUF_COLOR_FORMAT_NV24;
ret = NvBufSurfaceAllocate(&nvbuf_surf, 1, &dma_buf_create_params);
if (ret < 0)
{
printf("Failed to create NvBufSurface %i", j);
return false;
}
*fd = nvbuf_surf->surfaceList[0].bufferDesc;
nvbuf_surf->numFilled = 1;
return true;
}
Vulkan import
bool create_vkbufsurface(nvDmaBuffer* buffer, VkDeviceMemory* mem)
{
NvBufSurface* nvbuf_surf = NULL;
NvBufSurfaceFromFd(buffers->fd, (void**) &nvbuf_surf);
if (!nvbuf_surf)
{
printf("Failed to get NvBufSurface from Fd");
return false;
}
VkImportMemoryFdInfoKHR import_fd_info = {};
import_fd_info.sType = VK_STRUCTURE_TYPE_IMPORT_MEMORY_FD_INFO_KHR;
import_fd_info.fd = buffer->fd;
import_fd_info.handleType = VK_EXTERNAL_MEMORY_HANDLE_TYPE_DMA_BUF_BIT_EXT;
VkMemoryAllocateInfo allocate_info = {};
allocate_info.sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO;
allocate_info.allocationSize = nvbuf_surf->surfaceList[0].planeParams.psize[0] + nvbuf_surf->surfaceList[0].planeParams.psize[1];;
allocate_info.memoryTypeIndex = VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT;
allocate_info.pNext = &import_fd_info;
if (VK_SUCCESS != vkAllocateMemory(device, &allocate_info, NULL, mem))
{
printf("Failed to allocate memory");
return false;
}
return true;
}
timeline of bad good creation of dma buffers
Stream 0; NvBufSurfaceAllocate created buffer with fd: 146
Stream 0; NvBufSurfaceAllocate created buffer with fd: 147
Stream 0; NvBufSurfaceAllocate created buffer with fd: 148
Stream 0; NvBufSurfaceAllocate created buffer with fd: 149
Stream 0; NvBufSurfaceAllocate created buffer with fd: 150
Stream 0; NvBufSurfaceAllocate created buffer with fd: 152
Stream 0; NvBufSurfaceAllocate created buffer with fd: 151
Stream 0; NvBufSurfaceAllocate created buffer with fd: 153
Stream 0; NvBufSurfaceAllocate created buffer with fd: 154
Stream 0; NvBufSurfaceAllocate created buffer with fd: 155
Stream 0; NvBufSurfaceAllocate created buffer with fd: 156
Stream 0; NvBufSurfaceAllocate created buffer with fd: 157
Stream 0; NvBufSurfaceAllocate created buffer with fd: 158
Stream 0; NvBufSurfaceAllocate created buffer with fd: 159
Stream 0; Importing DMA Bufs: 146
Stream 0; Importing DMA Bufs: 147
Stream 0; Importing DMA Bufs: 148
Stream 0; Importing DMA Bufs: 149
Stream 0; Importing DMA Bufs: 150
Stream 0; Importing DMA Bufs: 152
Stream 0; Importing DMA Bufs: 151
Stream 0; Importing DMA Bufs: 153
Stream 0; Importing DMA Bufs: 154
Stream 0; Importing DMA Bufs: 155
Stream 0; Importing DMA Bufs: 156
Stream 0; Importing DMA Bufs: 157
Stream 0; Importing DMA Bufs: 158
Stream 0; Importing DMA Bufs: 159
Stream 1; NvBufSurfaceAllocate created buffer with fd: 227
Stream 1; NvBufSurfaceAllocate created buffer with fd: 228
Stream 1; NvBufSurfaceAllocate created buffer with fd: 230
Stream 1; NvBufSurfaceAllocate created buffer with fd: 229
Stream 1; NvBufSurfaceAllocate created buffer with fd: 231
Stream 1; NvBufSurfaceAllocate created buffer with fd: 232
Stream 1; NvBufSurfaceAllocate created buffer with fd: 233
Stream 1; NvBufSurfaceAllocate created buffer with fd: 234
Stream 1; NvBufSurfaceAllocate created buffer with fd: 235
Stream 1; NvBufSurfaceAllocate created buffer with fd: 236
Stream 1; NvBufSurfaceAllocate created buffer with fd: 237
Stream 1; NvBufSurfaceAllocate created buffer with fd: 238
Stream 1; NvBufSurfaceAllocate created buffer with fd: 240
Stream 1; NvBufSurfaceAllocate created buffer with fd: 239
Stream 1; Importing DMA Bufs: 227
Stream 1; Importing DMA Bufs: 228
Stream 1; Importing DMA Bufs: 230
Stream 1; Importing DMA Bufs: 229
Stream 1; Importing DMA Bufs: 231
Stream 1; Importing DMA Bufs: 232
Stream 1; Importing DMA Bufs: 233
Stream 1; Importing DMA Bufs: 234
Stream 1; Importing DMA Bufs: 235
Stream 1; Importing DMA Bufs: 236
Stream 1; Importing DMA Bufs: 237
Stream 1; Importing DMA Bufs: 238
Stream 1; Importing DMA Bufs: 240
Stream 1; Importing DMA Bufs: 239
timeline of bad creation of dma buffers (notice how fd indexes are reused for Stream1)
Stream 0; NvBufSurfaceAllocate created buffer with fd: 148
Stream 0; NvBufSurfaceAllocate created buffer with fd: 147
Stream 0; NvBufSurfaceAllocate created buffer with fd: 149
Stream 0; NvBufSurfaceAllocate created buffer with fd: 150
Stream 0; NvBufSurfaceAllocate created buffer with fd: 151
Stream 0; NvBufSurfaceAllocate created buffer with fd: 152
Stream 0; NvBufSurfaceAllocate created buffer with fd: 153
Stream 0; NvBufSurfaceAllocate created buffer with fd: 154
Stream 0; NvBufSurfaceAllocate created buffer with fd: 155
Stream 0; NvBufSurfaceAllocate created buffer with fd: 156
Stream 0; NvBufSurfaceAllocate created buffer with fd: 157
Stream 0; NvBufSurfaceAllocate created buffer with fd: 158
Stream 0; NvBufSurfaceAllocate created buffer with fd: 159
Stream 0; NvBufSurfaceAllocate created buffer with fd: 160
Stream 0; Importing DMA Bufs: 148
Stream 0; Importing DMA Bufs: 147
Stream 0; Importing DMA Bufs: 149
Stream 0; Importing DMA Bufs: 150
Stream 0; Importing DMA Bufs: 151
Stream 0; Importing DMA Bufs: 152
Stream 0; Importing DMA Bufs: 153
Stream 0; Importing DMA Bufs: 154
Stream 0; Importing DMA Bufs: 155
Stream 0; Importing DMA Bufs: 156
Stream 0; Importing DMA Bufs: 157
Stream 0; Importing DMA Bufs: 158
Stream 0; Importing DMA Bufs: 159
Stream 0; Importing DMA Bufs: 160
Stream 1; NvBufSurfaceAllocate created buffer with fd: 151
Stream 1; NvBufSurfaceAllocate created buffer with fd: 152
Stream 1; NvBufSurfaceAllocate created buffer with fd: 153
Stream 1; NvBufSurfaceAllocate created buffer with fd: 154
Stream 1; NvBufSurfaceAllocate created buffer with fd: 155
Stream 1; NvBufSurfaceAllocate created buffer with fd: 156
Stream 1; NvBufSurfaceAllocate created buffer with fd: 157
Stream 1; NvBufSurfaceAllocate created buffer with fd: 158
Stream 1; NvBufSurfaceAllocate created buffer with fd: 159
Stream 1; NvBufSurfaceAllocate created buffer with fd: 245
Stream 1; NvBufSurfaceAllocate created buffer with fd: 246
Stream 1; NvBufSurfaceAllocate created buffer with fd: 160
Stream 1; NvBufSurfaceAllocate created buffer with fd: 247
Stream 1; Importing DMA Bufs: 151
Stream 1; Importing DMA Bufs: 152
Stream 1; Importing DMA Bufs: 153
Stream 1; Importing DMA Bufs: 154
Stream 1; Importing DMA Bufs: 155
Stream 1; Importing DMA Bufs: 156
Stream 1; Importing DMA Bufs: 157
Stream 1; Importing DMA Bufs: 158
Stream 1; Importing DMA Bufs: 159
Stream 1; Importing DMA Bufs: 245
Stream 1; Importing DMA Bufs: 246
Stream 1; Importing DMA Bufs: 160
Stream 1; Importing DMA Bufs: 247