Using RDMA ibverbs registered memory as pinned memory for CUDA

Hello,

My question is simple, I have a memory region that I am registering to ibverbs using ibv_reg_mr to be able to perform RDMA operations. Registering with ibverbs means that the memory is now pinned and not pageable.

Now I also happen to want to use the same region for memory copies between the GPU and CPU.

At first I thought that CUDA will automatically detect that this region is pinned however it was treating it as pageable (Saying the memcpy is from device to pageable memory in the nvidia profiler). So I thought about registering the same memory again with CUDA using cudaHostRegister. But I’m not sure if this is safe and what flag I should use for this.

I’m thinking either cudaHostRegisterDefaultor cudaHostRegisterIoMemory might make sense.

Any thoughts on the safety of what I’m doing and on which flags to use would be appreciated.

Thanks

1 Like