CUDA Quadro P2000 BAR1 memory allocation issue, can only allocate 128MB, 160MB, 192MB


Working on a project using RDMA with Quadro P2000 and using nvidia_p2p_get_pages to allocated memory to BAR1. In Quadro specification, it said it can support up to 256MB but 32MB of them are reserved.

The issue now is that i can allocate 0~128MB successfully. However, above 128MB, only allocating 160MB(128+32), 192MB(128+64) can be done. In PCI specification, it mentions that only some upper bits of BAR register can be used to address the BAR memory. Is this issue is not actually a issue but a limitation of Quadro P2000? Any document or user manual can explain more details of this? Any comment will be highly appreciated.