Has anyone had any success using GPUDirect RDMA with a third party PCIe device with one of the newer (Pascal or later) Tesla/Quadro GPUs using 64bit addressing?
Additional details:
I have been using a Tesla K20 with a PCIe NIC and GPUDirect RDMA was working fine. Additionally, the Tesla K20 maps 256MB into the BAR1 region. I upgraded to a Telsa P40 and had to enable “above 4G decoding” (https://www.supermicro.com/support/faqs/faq.cfm?faq=17088) on my motherboard which switches all PCIe addresses to 64bit. The Tesla P40 now maps 32768 MB into the BAR1 region and I can no longer do RDMA with the NIC and have been informed by the manufacturer that the device is only 32bit capable.
Also, does anyone know if it is possible get the GPU to map less memory into the BAR1 region, so that 32bit addressing would work again?
According to the Tesla P40 product brief, page 5: http://images.nvidia.com/content/pdf/tesla/Tesla-P40-Product-Brief.pdf, the GPU supports Compute and Graphics mode. In Graphics mode it only maps 256MBs into BAR1. Does anyone know how to enable Graphics mode? “nvidia-smi” doesn’t mention anything about it.
In the thread linked by Robert Crovella it says “Graphics support for P40 needs GRID5.0” but also “There is no modeswitch any more for Pascal boards”. I read that to mean that there is a way to use graphics mode with a P40, just that it’s not accessible via the legacy mode-switching utility.
While one can write unit tests to ensure code works as intended, there is no technique yet that I am aware of that can ensure the accuracy of documentation. Mistakes in documentation do happen, though in this case it has not even been established there is one.
In practical terms, deploying a 64-bit capable NIC is probably the best way forward. At least that’s the direction I would pursue.
At the time the product brief was originally written, it was anticipated that there may be a need to support graphics mode in certain configurations (if it were deemed to be needed for that config). There is no indication in the product brief that this would be end-user configurable. Graphics mode was intended for situations where it might be needed to support particular features of the GRID product offering, and/or support particular OEM system configurations. As already pointed out, where there is discussion of actual modification of the mode in-situ, the P40 is explicitly excluded from that discussion.
During subsequent product development and the lifecycle to date, this feature has never been needed. Therefore all P40 are shipped in Compute mode.
Tesla products are only intended to be sold by certified partners, and then only in systems that are certified to support those products. We don’t support Tesla products for use in a platform that was not certified by the OEM for that specific Tesla product. For OEM supported configurations, the specific product offerings and supported configurations are defined by the OEM and are not necessarily end-user modifiable.
No OEM has ever offered this product with support for Graphics mode, therefore placing the product into graphics mode would be an unsupported configuration across the board. Due to that, no tools have ever been developed or provided to manage the configuration, since there is effectively only one supported configuration (with respect to the mode).
If you purchased the Tesla product from a certified OEM in a certified configuration, you also have the option to address your support concerns with that OEM or the channel through which you acquired the product.