Can detect 1 GPU, but not 3 -- Ubuntu 64-bit

Accelerated Computing CUDA CUDA Programming and Performance
1

Dear CUDA fans,

I have a machine with 3 PCI-e x16 slots and 3 GTX 260s occupying them. I installed the 64-bit version of Ubuntu 9.04 and succeeded in getting the CUDA driver (version 2.3), toolkit, and SDK installed, but only when one GPU is plugged in. If I plug the second or third card in, X windows will not start (using the Nvidia driver), and if I try running deviceQuery at the command line, it doesn’t detect any CUDA-enabled cards – not even the first one. The devices /dev/nvidia0 and /dev/nvidiactl (that were there when one card worked) don’t exist anymore.

I saw elsewhere on the web that people using 32-bit versions of Linux had problems adding multiple cards, and theirs could be solved by passing some parameters to the kernel at boot time:

title Ubuntu
uppermem 524288
kernel /boot/vmlinuz-2.6.(blah blah blah) vmalloc=256MB pci=nommconf
initrd /boot/initrd-2.6.(blah blah)

I tried this (adding uppermem 524288 as a separate line, and vmalloc=256MB pci=nommconf at the end of the kernel line), but it had no effect. One card works alone, but no cards work if multiple ones are installed. The solution above may be irrelevant for 64-bit versions, since one poster commented it is intended to overcome the 32-bit limit of 4 GB addressable RAM.

I’m not sure how to debug this any further. Has anyone else encountered this and fixed it?

Johno

2

This is weird and unexpected, but the problem is solved. I had all three cards plugged into their PCI-e slots and was just connecting or disconnecting the extra power cords that the cards need. Just now I tried connecting the power cords for all three cards to check the PCI buses they were each connected to, and bam, X windows started properly and deviceQuery saw all three cards!

I don’t really understand it, so I hope it keeps working, but at least I can use all the cards now.

Johno