If a machine has two GPUs, how is the data transfer between two GPUs’ device memory like? Are they connected by PCI-E two?
Speed?
two GPUs cannot communicate each other directly. You must bind two host threads to two GPUs respectively,
then you can switch data between two GPUs though host threads
[codebox]host thread 0 <------> GPU 0
^
|
|
v
host thread 1 <------> GPU 1
[/codebox]
In fact two 1.3 generation devices may communicate between each other, asynchronously, using Mapped Pinned Memory (Host memory accessed by all the GPU), this is the most interesting way to have 2 GPU exchanging data asynchronously without stopping the kernels execution on each side, and without host CPU being directly involved.
Notice: it also work on MCP79 IGP (Geforce 9300M/9400M), an 1.3 are G200 GPU : GTX260 and over (GTX 260M mobile GPU is NOT a 1.3 device but a rebranded 9800M!).
cheers :-)
A GTS260m is a 1.2 compute device and will indeed handle zero-copy memory. These are so new that they’re hard (impossible??) to find, their launch was more of an OEM availability announcement.
I’d rather have a GTS260m (with 96 SPs) than the GTX260m (with 112 SPs) just because of the 1.2 device support!
How about GTX 295? The two GPUs on GTX 295 cannot communicate with each other? Is the memory of size 1792 MB shared by the two GPUs?
In CUDA, the two gpus on the GTX295 can’t communicate with each other. And the memory size is 2 x 896Mb, not 1792Mb, ie. each GPU has its own, discrete memory space that knows nothing about the other GPU.
You have to think of the GTX 295 not as one card with 2 GPU, but it is logically 2 cards with PCI-express bridge, each GPU having 896MB of local memory.
And the interesting point is that these 2 logicial cards may communicate using Host Memory (Pinned Mapped Memory), asynchronously without interfering with CPU, nor any need to stop kernel execution, exactly as if you have 2 GTX 275 in one computer :-)
Pinned Mapped Memory is a great tool to have symetrical or asymetrical GPU<->GPU<->CPU communication!!!