Hey all,
due to the quite confusing technical guide of the PowerEdge T620 and the missing picture of the PCIe slots available we wonder, whether our platform is capable of GPUDirect version 2 (P2P Transfers) between our two K20 GPUs. Do we have to buy a second CPU for both GPUs to operate on the same PCIe root complex? (One PCIe Slot of the first processor is occupied by the RAID controller)
The GPU enablement kit has been purchased.
Buying a second CPU may be necessary due to the RAID card. Referring to the manual you linked, page 20, if you have only a single processor, then only slots 1,2,3,4 are functional. This is because those slots are serviced by PCIE lanes emanating from the populated processor socket. Modern Intel CPUs have the PCIE root complex in the CPU itself.
A requirement for P2P (GPU to GPU transfers in the same server node) is that both GPUs in question must be on the same PCIE root complex. This effectively means they must both be plugged into slots that are serviced by the same CPU socket. (Note that there are other requirements for P2P support as well, but you should probably be OK with two K20c GPUs. As an additional warning, be sure you are using the K20c – fan-cooled version of the K20. Do not try to put K20m in that T620. The T620 cooling system is not designed to ensure that a K20m will be properly cooled. The K20c is immediately identifiable because it has a fan on it. The K20m (or K20Xm) does not.)
In your poweredge T620 case, this means slots 2 and 4 are serviced by the “first” CPU socket, and GPUs plugged into those two slots should be able to communicate with each other using P2P. However, since you have a RAID card in one of those slots, you won’t be able to use two K20c GPUs (at all) with only a single CPU populated. I believe this is pretty clearly spelled out at the bottom of Page 20. Please re-read that section. You cannot use RAID cards with two double-wide GPU cardss in a one processor configuration.
If you did configure the “second” CPU, then slots 5 and 7 would also become “usable”. In this case, if you had GPUs plugged into slots 5 and 7, they would be able to communicate P2P with each other but NOT to the GPUs (if there were any) in slots 2 and/or 4. There are other configuration restrictions as well; I have not addressed them all here. You should carefully read pages 20-21 of that document, and if still in doubt, contact Dell support.