Dell R730
Requires specific riser card to be installed with 1 x8 PCIE slow + 2, x16 PCIe slots.
Order a specific GPU Enablement Kit. (I didn’t know about this initially, its on order but a jury rig solution exists)
If you have access to a modern modular PSU (Corsair/EVGA), the PCIE x2 cables will work in a pinch, but the cable routing is tight and PCIE slot 1 will end up with partially blocked cooling. The GPU enablement kit is supped to provide some lower profile heatsinks to allow for some better routing.
Order the 1100W PSU; the other PSU options may not work with other GPU cards, thankfully the power demands of the P100 are lower compared to previous generations.
OS Install: VM Ware Specific notes
Passthrough must be turned on for the PCIE Device. This is done in the VMWare Gui before creating any VM. A restart of VMWare is required to enable passthrough.
When creating the VM, you can add a PCIE Device to your VM build.
VM Ware does not work at the moment with a Tesla P100. I was able to get a 1080 and 1060 to detect via VM Ware passthrough on Version 6.5, but not the Tesla P100. There are specific settings to add to your vmx file for your vm.
pciPassthru.use64bitMMIO=“TRUE”
hypervisor.cpuid.v0 = FALSE
pciHole.start = “2048”
With the Tesla P100, LSPCI will detect an nvidia card but the drivers will refuse to install.
For the Tesla P100, I was forced to go to a standalone Linux install.
Installing a driver for linux.
There is a bug in Ubuntu 14 and 16 that corrupts the nouveau driver. The trick is to install the driver without the opengl portion and NOT to reconfigure the x server. There are many work arounds posted around the net, but they lead to frustrating secondary problems with CUDA down the line.
The simplest way to install I have found is to go with the CUDA run file installer, say no to the opengl install portion, say no to the x config portion.
I have repeated this on the Dell R730, a X99 custom PC with Linux, and an X99 VMWare 6.5 ESXi server build with a 1060.
Tensorflow, Digits and many other machine learning libraries have their own quirks, but at least this provides some info on this particular hardware setup.