Cannot Recognize 4th Tesla

I just received 4 C1060’s through the developer program and have been working to install them in a nForce 780a motherboard (the Foxconn destroyer that used to be recommended). The machine is currently running ubuntu 8.04. I have downloaded and installed the 185.18.14 version of the driver and everything seemed to go smoothly. However whenever I query the devices with deviceQuery, I find the first one listed is a C1060, the second is the onboard nForce 780a SLI, the third is a C1060 and the fourth is a C1060. Running the typical tests confirms that devices 0, 2 and 3 perform like teslas and device 1 is quite slow.

How do I go about getting the 4th tesla recognized? I thought to disable the onboard graphics, (I don’t mind if the computer is headless). However the bios will not let me do this completely (it will only disable if an external graphics card is detected) and additionially I am not getting cuda to work without starting the x server. I will be happy to post any additional information that is helpful (such as output of lspci) but I didn’t want to prematurely clutter the message board with unnecessary information if this has a trivial solution.

Thanks,
Luke

It turns out my problem was a loose connection. One of the cards was not fully seated in the pci-e slot. Now that all of the devices are recognized I have one more question. I would like to reorder the devices so that the first four devices are the C1060s and the integrated graphics comes last. At the moment the integrated graphics is device 1. Can this easily be changed?

Luke

You can’t reorder, but with nvidia-smi you can exclude the on-board graphic from running CUDA code.

This sounds perfect, but I cannot figure out how to get it to work. I assume you mean to use nvidia-smi -t? If this is the case I cannot figure out the ID of the onboard card.

I get only

==============NVSMI LOG==============

Timestamp : Thu Jul 30 00:55:14 2009

when I run nvidia-smi without any arguments and

GPU #0: (084C10DE:0D0D105B) nForce 780a SLI

GPU #1: (05E710DE:066A10DE) Tesla C1060

GPU #2: (05E710DE:066A10DE) Tesla C1060

GPU #3: (05E710DE:066A10DE) Tesla C1060

GPU #4: (05E710DE:066A10DE) Tesla C1060

when I run nvidia-smi -L. I have tried nvidia-smi -t 0 to which I get the message: Unit number out of range! Likewise using either of the numbers in parenthesis pops up the help message.

This will exclude GPU 0 ( 2: Compute-prohibited mode, no compute programs may run on this GPU):

nvidia-smi -g 0 -c 2

If X is not running, you may need to run nvidia-smi in loop mode.
After you have issued the command, check with:

nvidia-smi -g 0 -s

I ran into that with the bottom Tesla of 4 on my Destroyer Mobo – the cables coming from the USB and 1394a sockets on the bottom of the board press against the mobo side of the bottom Tesla and act as springs trying to force it out of the PCIe slot. Tightening the cable clamps helped with the USB wires, but I ended up disconnecting the 1394a plug completely to solve this issue.