Registering a numa-allocated host buffer with the GPU DMA engine for peak transfer performance in OpenCL

Accelerated Computing CUDA CUDA Programming and Performance
1

OS: OpenSuse Leap 15.5

GPU: RTX Pro 5000 Blackwell

driver version: 580.76.05

I am trying to register a large numa-allocated and pinned host buffer into an OpenCL context for peak transfer performance in smaller chunks to VRAM buffers. But as soon as clEnqueueMapBuffer is called with such a buffer, the driver wants to mirror it in the VRAM, which is not possible since its size exceeds the VRAM size.

The extension function clCreateBufferNV will indeed allocate only on the host but only when calling it with CL_MEM_ALLOC_HOST_PTR. This is not an option in my program. The buffer is allocated by another component.

Is there a way to register the large host buffer in the OpenCL context without the driver trying to mirror it?

P.S. I could not find documentation about Nvidia’s OpenCL extensions. I used the web archive to find a presentation for the clCreateBufferNV extension and I used that to test it. Other available documentation seems to be very old. I would be grateful if I got something newer.

2

Did you try CL_MEM_USE_HOST_PTR ?

// The OpenCL implementation will try to use or pin this memory
buffer_use_host_ptr = clCreateBuffer(context, 
                                     CL_MEM_READ_WRITE | CL_MEM_USE_HOST_PTR,
                                     data_size, 
                                     host_data_2, // Pass the existing host pointer
                                     &ret);
3

Yes, I have tried both clCreateBuffer and Nvidia’s extension clCreateBufferNV with CL_MEM_USE_HOST_PTR and they are equivalent. The driver attempts to mirror the buffer in VRAM when clEnqueueMapBuffer is called.

The only way to get a buffer without it being mirrored by clEnqueueMapBuffer is by creating the buffer with:

clCreateBufferNV(context, CL_MEM_ALLOC_HOST_PTR, CL_MEM_LOCATION_HOST_NV, size, NULL, &err)

The following call succeeds but the buffer is mirrored by clEnqueueMapBuffer:

clCreateBufferNV(context, CL_MEM_USE_HOST_PTR, CL_MEM_PINNED_NV, size, ptr_host_data, &err)

It has the same effect also when the buffer is created with CL_MEM_USE_HOST_PTR and without the CL_MEM_PINNED_NVflag.