CUDA Update to increase GPU limit in Windows 10

Hi,

from an associate to an AsRock employee I got the info, that CUDA driver will receive an update to increase the GPU limit of 8 in Windows 10 to 50.

Can you confirm this? I fail to understand, why via Linux more GPUs can be used (does Linux even have any GPU limit??), but only up to 8 in Windows 8. That’s a shame for any Gamer also engaging in crypto mining or professional miners.

I know that the topic has been discussed in https://devtalk.nvidia.com/default/topic/1004967/max-number-of-cuda-devices/#reply, but this is a slightly different topic. Also I do not agree with the assessment that this is a BIOS issue. Can’t be as more than 8 GPUs work fine under Linux. Or maybe GPU-Bios was meant, I don’t know.

Thanks!

I think you will find that NVIDIA (like most companies) does not provide public responses to rumors. You may also want to ask yourself how an “associate to an AsRock employee” would know about (supposed) features in future NVIDIA drivers.

BTW, how did you establish that there is a limit of 8 GPUs in Windows 10?

That person creates his own firmware for nvidia GPUs, thus claims to have received the info from Nvidia itself.

About your questions: This is commong knowledge among Miners. Latest info mentioned in this blog for instance: Asus B250 Mining Expert Motherboard with 19 PCI-E Slots - Crypto Mining Blog

The same limit of 8 GPUs under Windows exists obviously for both Nvidia as well as AMD. So IMO two possible root causes exist: Either the 8 GPU-limit is caused on driver level or on OS level. But since the OS usually relies upon drivers to recognize GPUs, I see this as a driver issue.

About the claim in https://devtalk.nvidia.com/default/topic/1004967/max-number-of-cuda-devices/#reply, that the issue is caused on Bios level: Can’t be true, since the limit exists even on Miner-specific mainboards like the AsRock H110 BTC+ http://www.asrock.com/MB/Intel/H110%20Pro%20BTC+/index.asp#Specification whose Bios supports up to 13 GPUs.

I see anecdotal evidence, not proof. Since the observed limit applies to both NVIDIA and AMD hardware, it is at least conceivable that this limit is imposed by WDDM 2.0 (the Windows 10 driver model). And contrary to your assertion, I see no proof that this is not an issue with BIOS resource limitations, such as memory aperture configuration, in some cases. One counter example does not constitute a proof.

Be that as it may, if the “associate to an AsRock employee” is correct (and you seem to think s/he is), all your worries will be a thing of the past soon. So keep trying new Windows driver versions or try switching to Linux if tired of waiting. An OS switch cannot be a serious obstacle to a dedicated miner, I would think.

My take: Given the popularity of mining Ether, if the issue were purely a driver issue, I think both AMD and NVIDIA would have been falling all over themselves to get a fix out quickly so they can sell more GPUs to miners before their competitor does. The fact that this has not happened would seem to indicate it is more than an (arbitrary) driver limit.

I don’t really understand why you’re so sceptical. Try it for yourself. It won’t work. On any machine. With any one GPU model. No one has gotten it to work. Ever. 8 GPUs run fine, you add the 9th, it crashes within seconds after booting.

But if you can provide me with a setup which you tried and verified to work with 9 GPUs, I’m all ears to hear about it.

I also don’t understand why you’re so stubborn to believe that motherboard vendors are in touch with GPU vendors. You know, whose products you put in motherboards. I’d be shocked if they WEREN’T.

OTOH you can have like 6 NV and 6 AMD cards, then it will work. So apparently it’s a limitation of the Windows drivers on both sides.

Additionally, the same 9 (or more) GPU setup runs perfectly fine on Linux. So apparently the Windows drivers are scaled up to 8 GPUs, which is not a problem if you mix up different vendors, because each driver needs to handle less than 8.

Nowhere did I state that there may not be an 8-GPU limit under Windows, I merely asked how OP established that limit. What I am skeptical about is that

(1) the limit is due to an (arbitrary) limitation in NVIDIA’s drivers
(2) this will be addressed “real soon now” by NVIDIA changing the limit from 8 to 50
(3) the OP has an inside track providing him reliable information regarding items (1) and (2)

If the OP was correct, and his source of information reliable, we should see NVIDIA drivers for Windows 10 by now that support 50 GPUs. To my knowledge, that hasn’t happened yet.

  1. Here’s the deal: in the world of computing you always have “arbitrary” limitations. Mostly because there’s no such thing as “infinite”, you have to set limits and boundaries which seem reasonable at the time. Like a file on NTFS can be 16TB at max, and a filename can be 255 characters tops. That’s the way it is.

  2. About this I’m also sceptical, 50 seems indeed too random, because it’s not a power of 2. I was to say it’s also way too high, but then I realized now they sell mining boards with up to 19(!) GPUs supported.

  3. That I don’t know either.

Anyways, you’re right that NV and AMD should definitely fix this, and/or push MS to allow a fix to happen. But miners are really good at working things around. That’s why you have GPUs connected to the boards via x1 connectors. And ATX boards installed in rack mount chassis, sometimes with 2 ATX PSUs installed. Etc.

But this problem affects really few people, because

  1. You can have up to 2x8 GPUs if you mix vendors, which is sometimes even desired when you do multi-algo/multi-currency mining.

  2. Most cases don’t support more than 8 GPUs anyway.

  3. The savings are minuscule when you increase the number of GPUs/system. Like within the 5% margin, and it’s a one time cost.

  4. You can use Linux where you don’t have this limitation, and the mainstream mining utils do have Linux ports.

So I guess that’s why neither NV nor AMD are hard pressed to solve this. But the problem’s definitely there.

I’d love to hear an official statement about the bump, but the fact of the matter is, there isn’t even any doc available that mention the current practical limit of 8 on Windows / driver.

I am not calling for any changes to be made regarding the maximum number of GPUs per Windows 10 driver. I don’t mine crypto currencies. Personally I am perfectly fine with the status quo. Feel free to file bug reports or enhancement requests with NVIDIA as you see fit. The relevant form is accessible from the registered developer website. Log in at [url]https://developer.nvidia.com/[/url]

Note: These forums are intended as a platform for a user community (“users helping users”). They are not designed as an official support or bug-reporting channel.

Update: I read somewhere that the Fall Creators Update actually raised this limit. Finally got to test it yesterday, and confirmed: works fine with 9 GPUs as well. Don’t know what the current exact practical limit is though.

@bviktor: Interesting news. Thanks! In the bitcointalk forum they were speculating if it works or not, but didn’t see a definite answer or what the current nvidia GPU limit is. Well, given that AMD just recently increase their GPU limit to 12, i sincerely hope it will be - at least in the future - 12.

@njuffa: You have so far not contributed a single helpful comment so far, just scepticism. That AsRock hotline guy’s contact might have been right or wrong (potentially takes its time to increase the limit, even if 50 seems way too much). I don’t care. Could you just refrain from commenting here anymore if you don’t have any useful info to share? Thanks!

Hey, it’s a free country, so I am free to comment here just like the next person is :-) Skepticism plays an important role in engineering: it helps people not to jump to conclusions.

I told people where to file bug reports if they suspect NVIDIA’s software is to blame for the limitations discussed here. That is actionable information that may lead to a resolution of the issue. Lamenting that a limitation exists does nothing to resolve it.

The information in #9 and #11 strongly suggests that the GPU limit was in fact one imposed by the operating system, not NVIDIA’s driver per se, as I suspected based on the information in #1 (no limit in Linux) and #3 (same GPU limit for NVIDIA and AMD).

PS: Mr The_Witcher, you wrote “refrain from commenting…” ?

Be careful what you wish for; if you spend some time here, you’ll soon learn that njuffa is one of this board’s most valuable features!

Couldn’t agree more. On top of that, in my experience, he has pretty thick skin.

@njuffa, skepticism does play an important role, but upending someones interests based on the ‘anecdotal evidence’ is silly. I mean what kind of evidence would you expect here? This isnt an engineering debate or one on corporate relations practices or lessons on burden of proof… The topic was brought up to pulse the airwaves for anyone else thats heard something similar. No one cares if you have any interest in the increase in GPUs able to be run or whether you are into crypto.

@nnunn, whether njuffa is valuable or not gives no permission slip for a big head or sharp remarks. This is supposed to be a supportive community. Not one that takes stabs at anyone excited or speculating.

@txbob, I joined this forum because it is nothing more than counterproductive to see someone put down or stirred up by a big head with thick skin. This topic was proven useful by @bviktor and @the_witcher only. Please reconsider the special perception given to your so called experienced members…for all of us searching for answers.

I’m running 9 GTX 1080 Ti GPUs since a week as well. The current GPU limit in Windows 10 after the Creator’s Update using mixed AMD + Nvidia setups should “theoretically” be 21, since AMD allows for 12 GPUs by now + 9 Nvidia GPUs.

But since I run a couple of mixed 13 GPU rigs, I know that 21, even if a mainboard would support it, would increase the load on the system and the Virtual and physical memory requirement. With 13 I use 16 GB physical RAM and 40+ GB Virtual Memory in addition. With up to 21, I suppose it would be more like 24-32 GB physical RAM and most likely 60+ GB Virtual Memory.

@supasnipa360: Thanks for your interest in the topic and your comment.

It’s definitely not 9 because we already tested with 10 in the meanwhile.

Thanks, sounds good. This would mean a theoretical maximum of 10 Nvidia + 12 AMD GPUs = 22 total, if a motherboard supports it and the rest of the hardware can take it (espec. RAM).

Do you run mixed GPU rigs (AMD + Nvidia)? How many GPU do you use total then?

Thanks and best regards!

It’s not rigs, it’s 4U 10 GPU servers, so we didn’t add any AMD GPUs, only the 10 NV cards :)