8x GPU GTX issue under Windows

Has anybody been able to get 8x GTX (780) to work with CUDA 5.5 or 6.0 for Windows 7?

I know it works in Linux, and I’ve seen it work with Win 8, but this needs to be Win 7.

Scenario 1:
Using one of the GTX for video:
With 8 cards installed I get an alert next to 2 devices (code 43 I believe) in device manager.

Scenario 2:
If I switch to onboard, CUDA won’t communicate with the GPUs anymore. And there is no video output from any of the GPUs
I can see all 8 off the cards and they are correctly functioning in the device manager.

note:
I only need one output for display, the onboard is a BMC display so no direct3D function using XDDM drivers.
Also, this works perfectly under Linux, but this is not an option.

The problem in scenario 2 with the BMC graphics enabled is a win7 incompatibility between the default XPDM driver for the baseboard graphics and the WDDM driver for the GTX cards. With an XPDM driver “in the box” you cannot use/access WDDM devices. Yes, they will show up in device manager, but they are unusable for WDDM, which basically means they are unusable. (no issue here under linux, of course)

Win8 should work around this, because the “default” VGA driver is compatible with other WDDM devices.

My first guess at the problem in Scenario 1 is a resource allocation issue. The first step in resource allocation is the SBIOS. You might check to see if you have the latest SBIOS for your server (what server is it, by the way?) You might also check to see if there are SBIOS options that enable BAR decoding/resource placement above certain ranges (and enable these, if you find them). The fact that you can get this working in Linux would sort of argue against this, but I think windows and linux may differ in their expectations for resource placement and what they will decode/accept.

You can get 6 GTX devices to work in Win7?

Scenario 2:
This makes sense. We’re having issues using the WDDM drivers on the onboard. When we install the WDDM driver we end up with a BSOD. (I don’t remember the code)

Scenario 1:
As far as the Resource Allocation, the system works fine with other setups, for instance in Linux with these cards or using Tesla cards where there is no output.

Tyan B7059
I’m positive I have the newest drivers for the Tyan system and the GTX.

Ok, to clear something up I realized today may need to, I actually really only care about getting the 8x GTX cards to work. I’m not displaying anything on them, just need them active in Windows for CUDA.

Also:
With the Onboard Video turned off we have been able to get 7 cards to work, just not 8.

BAZINGA! I got it, the internet is a wonderful thing. I finally narrowed it down to WINDOWS Resources being set by Directx/Direct3D. But I’m not a programmer so I haven’t a clue how to set that. After some research I tracked the following down and it works!

  • Start Menu
  • Under Search type: Regedit (Press Enter)
  • Navigate to the following directory:
    o HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class{4D36E968-E325-11CE-BFC1-08002BE10318}
  • You will see directories labeled “0000”, “0001” and so on that are relative to the total amount of cards.
  • Under each of those Keys (eg. 0000) create the following two Dword values, once created set their value to 1
    o DisplayLessPolicy
    o LimitVideoPresentSources
  • You can create the Dword by right clicking on the key and NEW  DWORD

Note though that I didn’t create this, so it may limit something, or be detrimental in some other means. So far it’s ok.

Gotta love M$ for burying a setting like that, thanks for reporting the solution! Here’s an old thread here mentioning those same settings:

https://devtalk.nvidia.com/default/topic/454214/cuda-programming-and-performance/multi-gpu-not-recognized-by-vista-33-/

Ya actually now that I search for those keys on google I see the solution everywhere including a couple of times on here. We actually had Nvidia look at it as well and they Ok’d the use of the reg changes and for far using the system for a couple of weeks everything looks ok.

Also this fix worked on a series of K20/40s as well.

Sorry for hijacking this thread, but OP, could you post your detailed hardware setup: motherboard, case, CPU, power supply, etc.?

I, as well as some other forum members, are trying to build 4 to 8-GPU systems with GTX 780 and the like. The instructions for doing that are hard to come by.