Dual Nvidia GPUs

I have a supermicro workstation with a Quadro M4000 as my GPU. I recently bought a Titan V. I would like to have both installed, using the M4000 for my primary graphics card (display out) and the Titan V available strictly for requested computations (python/CUDA).

(1) First, powering the Titan V. I have two 6+2 PCIe cables (the 2 comes off of the 6 connecter), and one 6+6, and 2 (where the 6 has another 6 coming off its connector, and then there is a separate 2 with dedicated wires). Not sure which to use, or if it matters as long as I have both the 6 and 8 pin ports occupied on the Titan V.

(2) Whenever I have both connected, I get no output to my monitor. Both are plugged into PCIe 3 x 16 slots (although the one I plugged the Titan V is labeled PCIe 3 x 8 (16 in)

Any thoughts?

Motherboard: Supermicro X10SRA, http://www.supermicro.com/products/motherboard/xeon/c600/x10sra.cfm
Power supply: SuperMicro PWS-1K25P-PQ (1200W Platinum PS)
OS: Ubuntu 16.04
nvidia-bug-report.log.gz (157 KB)


The pins on the power connectors are shaped differently, so as long as they fit and both 6 and 8 pin ports are fully utilized it shouldn’t matter.
Did you try to connect the monitor to each card, maybe the bios is just changing the primary card? Otherwise, try to ssh in and please run nvidia-bug-report.sh as root and attach the resulting .gz file to your post. Hovering the mouse over an existing post will reveal a paperclip icon.

Yup, I see the different shape of the pins, but wasnt sure if there was a different available power draw depending on which way I connected it (see attached photos for the two options to the OG post). I tried to connect to the monitor to the new card, still no output. I’ve attached the bug report as well. Thanks!

They both show up in lspci;
03:00.0 VGA compatible controller: NVIDIA Corporation GV100 [TITAN V] (rev a1)
03:00.1 Audio device: NVIDIA Corporation Device 10f2 (rev a1)
04:00.0 VGA compatible controller: NVIDIA Corporation GM204GL [Quadro M4000] (rev a1)
04:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1)

I now get display in Ubuntu by running;

$sudo nvidia-xconfig

then editing;

Section “Device”
Identifier “Device0”
Driver “nvidia”
VendorName “NVIDIA Corporation”
EndSection

to…

Section “Device”
Identifier “Device0”
Driver “nvidia”
VendorName “NVIDIA Corporation”
BUSID “PCI:4:0:0”
EndSection

but still no display in BIOS/pre-system boot. I’m wondering, does it set the default GPU in BUS order? Should I just swap the Titan to Bus 3 and the M4000 to Bus 4?

Yes, the bios is setting the Titan in on bus 3 as primary. So if there’s no option in bios to change that, you’ll be left to swap slots and change the xorg.conf accordingly.

BTW, according to the logs everything else is fine.

On my ASRock X99X, I use two discrete Nvidia cards as well, and have 5 monitors (3 to one card, 2 to the other). I have an option to enable UEFI compatibility for graphics/storage in my BIOS - if I enable UEFI for my graphics cards, the bios choose the HDMI connection on my secondary card(I used one monitor only on each different connection while booting to find out which one was enabling). If I choose the legacy BIOS mode for graphics, I receive the POST messages on my primary monitor (DP on primary card). In my case, it is UEFI that is the problem. See if your BIOS has some legacy settings.