second gpu not working correctly


I installed a second gpu to PCI-E slot (x-4 mode). But the bandwidth is very slow:
Using device 1: GeForce 8800 GTS
Quick Mode
Host to Device Bandwidth for Pageable memory
Transfer Size (Bytes) Bandwidth(MB/s)
33554432 58.9

Quick Mode
Device to Host Bandwidth for Pageable memory
Transfer Size (Bytes) Bandwidth(MB/s)
33554432 59.5

Quick Mode
Device to Device Bandwidth
Transfer Size (Bytes) Bandwidth(MB/s)
33554432 107.1

&&&& Test PASSED

The bandwidth of device 0 is ok. Do you know what’s the problem? Thank you very much!


try running
./bandwidthTest --memory=pinned

and then measure the bandwidth…
for more bandwidth test type in ./bandwidthTest --help



It will be crash to run ./bandwidthTest --memory=pinned on second gpu.

I have tested it under Vista and the second gpu also has a normal bandwidth (X4), although I have connect a monitor to the second GPU also.

I include the lspci -t information here and hope some one can help. Thanks!










       |                 \-00.1











What is crashing (the app, the computer, something else)?

Please generate and attach an nvidia-bug-report.log from the system that is exhibiting this problem.

The crashing is just the bandWithTest.

I solved this problem by following way:

My computer has one primary PCI-E X16 slot and one PCI-E X16 slot (but only working at X4 mode) .

The PCI-E x16 bus id is 01:00.0 and I have a X server on it. The second gpu is on PCI-X4 with bus id 04:00.0.

deviceQuery will report device id 0 for the primary slot GPU and device id 1 for the second gpu, the second gpu will not work on CUDA program.

But when I set the X server to use the second gpu only(i.e. set the device to PCI:04:00:00), then the device id for second gpu will be 0 and the first GPU will be 1. CUDA can work on both GPUs.

Other configurations will have the device id 0 for first gpu and device id 1 for the second gpu and the CUDA will not work on the second GPU. These configurations are

  1. each gpu has a screen with monitor connected

  2. in Xorg.conf, switch the pci bus id order between the two devices.


I also had some weird bug with the latest beta 2.0 driver. When I installed 8800GTX (G80), 8600GT, and 8800GTS (G92) altogether (in the specific order), the device query cannot report all 3 devices correctly, and when I tried to run any CUDA program on the third card, which is 8800GTS (G92), it will simply fail. However, if I remove the first 8800GTX, everything goes fine. I guess there’s a bug in the driver that it cannot identify all cards when more then 2 cards are installed in some special order.

By the way, I was using ASUS Striker Extreme (680SLi) with the latest BIOS and I am using Ubuntu 8.04. Since I’ve removed this configuration, I can’t generate some report now, but just let you know there’s a potential problem of mixing cards together.

Mu-Chi Sung,
Ubuntu-8.04 is not currently supported. However, in my experience Ubuntu’s kernels (from 7.04 and 7.10) have issues parsing some motherboards’ PCI topology which prevents hardware from appearing on the PCI bus. Unless you can provide an nvidia-bug-report.log (or at least lspci output) that shows that all of your hardware is visible on the PCI bus, this is likely an Ubuntu kernel bug.

I see, thanks for the info!

But except for that issue, everything should be fine, right? I am currently using it on 8.04 and didn’t find any other problem related to 8.04 so far.

I can’t say whether anything would be fine, as Ubuntu-8.04 is currently unsupported and untested with CUDA. A future release of CUDA (after 2.0) will support Ubuntu-8.04.