Nvbuf_utils cant free dmabuf when imported by cudaImportExternalMemory

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

hi,
when I create a NvBufferFd use NvBufferCreate(),then map it to a cuda buffer named ‘cuda_ptr’ using cudaImportExternalMemory() and cudaExternalMemoryGetMappedBuffer(). ‘cuda_ptr’ can pass to a cuda kernel to do some operations correctly. It seems that everything goes well. but when I want to release the memory allocated to the NvBuffer, No matter how i try it doesn’t work. i try to use cudaFree(cuda_ptr) or NvBufferDestroy(fd) or NvReleaseFd(fd) or some combinations of them. the memory still con’t be free unless i kill the process.
so anyone can give me some suggestion how to free memory in this case？

2

i will paste the test code i use follow. please ignore the log function and error-check function.

void test_func(){
    int ret = 0;
    int fd = -1;
    int cols = 1920 * 5 * 10, rows = 1080 * 5;  // alloc size is big so the memory change is significant
    ret = NvBufferCreate(&fd, cols, rows, NvBufferLayout_Pitch, NvBufferColorFormat_ABGR32);
    if (ret < 0) {
        BVA_ERROR("Failed to create NvBuffer:%d, for:%dx%d\n", ret, cols, rows);
    }
    BVA_INFO("Create nv buffer fd:%d, for:%dx%d\n", fd, cols, rows);

    // 2. import external memory
    NvBufferParams params;
    memset(&params, 0, sizeof(params));
    NvBufferGetParams(fd, &params);
    BVA_INFO("NvBuffer pitch:%d\twidth:%d\theight:%d\n", params.pitch[0], params.width[0], params.height[0]);

    cudaExternalMemoryHandleDesc memHandleDesc;
    memset(&memHandleDesc, 0, sizeof(memHandleDesc));
    memHandleDesc.type = cudaExternalMemoryHandleTypeOpaqueFd;
    memHandleDesc.size = params.pitch[0] * params.height[0];
    memHandleDesc.handle.fd = fd;
    cudaExternalMemory_t extMem = nullptr;

    checkCudaErrors(cudaImportExternalMemory(&extMem, &memHandleDesc));
    BVA_INFO("cudaImportExternalMemory success\n");

    // 3. get dev_buf
    cudaExternalMemoryBufferDesc bufferDesc;
    memset(&bufferDesc, 0, sizeof(bufferDesc));
    bufferDesc.offset = 0;
    bufferDesc.size = memHandleDesc.size;
    void *dev_ptr = nullptr;
    checkCudaErrors(cudaExternalMemoryGetMappedBuffer(&dev_ptr, extMem, &bufferDesc));
    BVA_INFO("cudaExternalMemoryGetMappedBuffer success\n");

    //=============
    // cuda kernel function using dev_ptr as input 
    //=============

    // 4. free memory(any combinations of following oprations is not work)
    // 4.1
    checkCudaErrors(cudaDestroyExternalMemory(extMem));
    BVA_INFO("cudaDestroyExternalMemory success\n");
    // 4.2
    if (dev_ptr) {
        checkCudaErrors(cudaFree(dev_ptr));
        dev_ptr = nullptr;
        BVA_INFO("cudaFree success\n");
    }
    // 4.3
    // ret = NvReleaseFd(fd);
    // if (ret < 0) {
    //     BVA_ERROR("NvReleaseFd failed\n");
    // } else {
    //     BVA_INFO("NvReleaseFd success\n");
    // }
    // 4.4
    // ret = NvBufferDestroy(fd);
    // if (ret < 0) {
    //     BVA_ERROR("NvBufferDestroy failed\n");
    // } else {
    //     BVA_INFO("NvBufferDestroy success\n");
    // }
}
4

Hi,

NvBufferCreate just maps the buffer into GPU accessible without memcpy.
The destroy will only release the “mapping” rather than buffer itself.

To free the memory completely, please release the original buffer after destroying the mapping.

Thanks.

5

hi，thanks for reply.
but i still confused about “release the original buffer” in your answer.
according to my understanding, in the code i paste before, i got 2 ways to operate the buffer, one is file descriptor fd, another is device pointer dev_ptr.
which API should I call to free the original memory? I already try the cudaFree, but no use.
plus，there is a error report “NVMAP_IOC_FREE failed: Bad file descriptor” when I call both NvBufferDestroy and cudaFree, although the return value is success. when i change the order of the calls，there always a same error report in the later functions. can you explain this？
thanks ~

6

Hi,

The workflow looks like this:

Create DMA buffer → Map as GPU accessible → Get GPU pointer

There is no GPU buffer created but only a pointer when mapping to GPU.
As a result, you cannot use cudaFree to release it.
(since you cannot release a DMA buffer with CUDA API)

Instead, please check the creation of the DMA buffer.
And destroy it correctly to release the memory.

Thanks.

7

Hi, I’m glad you can reply to me.
the workflow is correct. But about your reply this time, I still have two questions
firstly, about your description “you cannot use cudaFree to release it. ( since you cannot release a DMA buffer with CUDA API )”, but i got another description in cuda official documention, which says “Ownership of the file descriptor is transferred to the CUDA driver when the handle is imported successfully. Performing any operations on the file descriptor after it is imported results in undefined behavior.” how should I understand it?
secondly, following your suggestions, i also tried to using NvBufferDestroy or NvReleaseFd instead of cudaFree to release the DMA buffer’s memory, sadly, no use again.
Any other suggestions for me? I can’t find any API(no matter Nvbuffer’s or CUDA’s) which can make sense.
I am looking forward to your reply.

8

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

Hi,

Could you share a complete source with us so we can check it deeper?
Thanks

10

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.