Hi! I appreciate any help I can get with building the following workstation:
I’m looking to make a DIGITS DevBox clone using the Asus Z10PED16, a dual socket motherboard based on the Intel C612 PCH chipset. The Asus website says the motherboard supports 4x16 PCIe 3.0 and suggests it works well for 4-GPU setups.
However… I’m still unsure if that particular motherboard is a good choice since the DIGITS DevBox documentation talks about how multi-GPU setups can only achieve fast P2P DMA if they’re all running on the same I/O hub. How can I tell if the (dual socket) board is sub-optimal in this respect?
It means that all 4 GPU sockets should be electrically or logically connected to a single CPU socket. Since 4 x16 PCIE lanes constitute 64 PCIE lanes, and there are no Intel CPUs that have 64 PCIE lanes coming out of the socket, this is generally not possible unless you either use a PCIE switch on the motherboard, or else limit some PCIE sockets to e.g. x8 electrical.
To verify a correct topology, it’s necessary to inspect the system topology diagram as well as motherboard specifications to verify this particular aspect of the design.
Putting e.g. 4 GTX Titan X in a single motherboard generally also has ramifications for the System BIOS; not all System BIOSes will respond correctly to such a config.
Having said all that, I don’t know about the particular motherboard you cite, and I can’t make any recommendations about it. If you don’t know what a PCIE switch is, and the concept of a PCIE fabric connected to a PCIE root complex, this might be difficult for you to verify. You might want to study those concepts first. In a nutshell, for all GPUs to be able to do GPUDirect P2P with each other, we want them all on the same root complex (which is kind of like saying they are all “connected” to the same CPU socket), or stated another way, all on the same logical PCIE fabric. If you want assurance, buy a DIGITS devbox, or a properly qualified system from a system vendor.
SLI is pretty much orthogonal to GPU computing. The general recommendation is to not install any SLI bridges, even if the cards you are using will accept them, if you are only interested in compute functionality.
SLI brings no benefit to CUDA (it is not used for compute purposes) and in fact could arguably be a complicating factor: