Multi Device Bandwidth

Fellow Developers:

I am attempting to connect 3 GTX 295 cards into my system.

Running devicequery, I get fast results with the default device (device 0) but my connection to device 1 is more than twice as slow.

As a result, I do not get any benefit by splitting the load between the devices.

Is this a known issue?

Depends on the motherboard? You’re probably getting one 16x and two 8x slots.

The first question to ask is what motherboard you’re using. Does it support full x16 PCIe 2.0 speeds on all three of your PCIe slots?
It’s not too hard to find boards that supply x16 x8 x8 or x16 x16 x4 but it does take effort to find one that can supply x16 x16 x16.

well, that’s not REALLY three 16x slots, that’s 32 lanes split into one real 16x slot and two slots connected to a switch connected to 16 more PCIe lanes, so you can get a peak of 16x on either slot but only 8x if you’re transferring to both devices simultaneously.

the only thing that can do full 16x with more than two slots is a dual-Tylersburg motherboard for dual-socket Nehalem platforms, but they have their own problems with bandwidth (even once you get past NUMA affinity).

Thanks for the quick responses.

I believe that I have 2 16x and 1 8x.

The strange thing is that I even tried using only the 2 GPUs on one gtx295x device and yet, device0 is much faster than device1.

I then tried using one GPU on 2 different gtx 295x devices and the same issue occurs. I see how this can potentially be where the pci mobo problem comes into play but it doesnt really explain the first example.

Do you only have one GTX 295 in the machine or do you have three? If three, why do you believe that device 0 and device 1 are on the same GPU? There is no guaranteed order to device enumeration whatsoever and it is extremely likely that they are not on the same physical board.

I have 3 GTX295 devices. The reason that I think that I am using two GPUs is that when plugging two monitors into my single card, the deviceQuery locates only two devices (even though technically 3 GTX295 cards = 6 devices should be seen). I suspect that these two devices are the ones from my displays.

When I connect each monitor to 2 different boards, devicequery still only lists 2 devices. As a result, I suspect these two devices are the ones connected to each monitor.