My Dell R740 server has two NVIDIA Tesla T4 GPUs. I was experimenting GPU Direct (RDMA directly to the GPU memory). Although the GPUDirect works well between CPU and GPU memory as well as GPU to GPU memory, the PCI BAR1 window aka VRAM aperture is only 256 MB. That limited the maximum amount of GPU memory can be registered for RDMA operation at a time. According to the NVIDIA hardware document, the PCI BAR1 window is configured during GPU initialization through PSTRAPS. And the straps values can be overriden by the driver to override at runtime. However, the PSTRAPS document does not cover how to use the driver to do that (because that is the hardware document).
I’m wondering how I can change the PCI BAR1 window using nvidia’s linux driver.
OS kernel: Linux compute31 5.0.0-58-generic #62~18.04.1-Ubuntu SMP Tue Jul 14 03:37:30 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
nvidia driver: 450.51.06
nvidia-bug-report.log.gz (2.3 MB)