Is there any technical specification on how much GPU memory can be registered for RDMA with ibv_reg_mr?
Also, is there any trick to register more than the BAR1 size memory or to increase it? Ideally I’d like to expose the whole GPU memory for RDMA operations, but so far it seems limited to the 256MB of BAR1.
For reference, I am using a Nvidia T4 on a Dell Poweredge R740xd with 2 Cascade Lake CPUs and a ConnectX-5 NIC.
There is no way to use more memory than what is exposed through the BAR. PCIE has a resizable BAR feature, but it requires specific support both from the endpoint vendor as well as the system OEM. AFAIK this feature would not be available in the configuration you describe.
Thanks, so basically no more than 256MB can be exposed to RDMA?
that might be true for T4. I don’t happen to have the T4 BAR size info at my fingertips.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.