Ibv_reg limit

[anandhuasokkumarp]

I need to register 10 GB of memory using RDMA, but the program is limiting it to 4 GB. Could this be due to a uint32_t limitation in ibv_reg_mr, or is there another reason for this restriction?

[ssimcoejr]

Hi anandhuasokkumarp,

Welcome, and thanks for posting your inquiry to the NVIDIA Developer Forums.

There is no uint32 limitation in ibv_reg_mr - ibv_reg_mr size limit is equal to device’s max_mr_size.
ibv_devinfo -v will show you max_mr_size for all adapters in your system. Example from a pair of ConnectX-7 adapters, showing maximum total number of MRs supported (max_mr) as well:

root@localhost:~# ibv_devinfo -v | grep max_mr
        max_mr_size:                    0xffffffffffffffff
        max_mr:                         16777216
        max_mr_size:                    0xffffffffffffffff
        max_mr:                         16777216

Could be that your system’s ulimit is too low - check ulimit -lto see how much memory (in KB) your process can lock. Configuring this to ‘unlimited’ may help here.

Outside of the above recommendation, this type of query is out-of-scope of NVEX support. We cannot provide programming assistance for custom applications. If you require programming assistance, we recommend contacting your account team. They can assist you with developing a solution for your business needs.

You can also reach out to our Sales and Solutions team via the following link: https://www.nvidia.com/en-us/contact/sales/

Thanks, and best regards,
NVIDIA Enterprise Experience