Thanks for reading. I find the feature in Physical Address Memory Region https://community.mellanox.com/s/article/physical-address-memory-region very attractive for my use-case. I am trying to allocate a physical MR that allows access to all memory (ibv_exp_reg_mr_in in.addr = NULL and in.length=0). This is my setup:
Compile MLNX_OFED with --with-pa-mr
Allocate (or not allocate) some huge pages during boot
Try to register above ibv_exp_reg_mr_in
and eventually got a registration failure.
I am now trying to figure out the reason. If someone can help answer following questions, that would be super helpful. I apology if they look silly as I am new to RDMA and Mellanox devices:
- What does “allowing access to all memory” (mentioned in the link above) mean here? Is it allowing access to all memory on the host? But since physical memory region needs to be pinned down and continuous, does it mean I need to pin all memory with huge pages?
or “allowing access to all memory” means it can only access memory pinned by the huge pages, while unable to touch the remaining
Does physical memory region require hardware support? Right now I only have ConnectX-3 Pro (I can try getting a X-4/5). Does it support physical memory region? I am wondering whether the registration failure is caused by my hardware, not the code.
If someone can provide a hello-world example/manual on this. That would be great.
Again thanks for your time.