Hello, I am trying to utilize an offline ML setup for home/education purposes. I have two K80’s that I was hoping to use for this if it is still feasible.
If it is, what compatible board should I be looking at?
Any feedback including suggested builds for LLM’s would be greatly appreciated. Other than the two K80’s and a personal laptop with a NVIDIA 1650. I am starting from scratch on this build.
I have a system built around two Tesla P40 GPU cards, which is a couple years more recent than the K80s. I have since gotten a K80 since they are even cheaper than the P40s to see whether a similar system could be made with them. An Ubuntu 20.04 install on a ASRock B450M Pro4-CB with Ryzen 8-core CPU, 16GB system RAM, and a Nvidia Quadro K600 graphics card does recognize the K80, showing Nvidia driver 460 and CUDA 11.4. I am trying to get Llamafile to recognize the GPU, but so far am having no luck. However, it is still early on this attempt for me, and it took quite a while to get the configuration right for the P40s, so there is some hope yet. My last go with the P40s reached success by installing the Nvidia Container Toolkit and doing one-off tests in containers, rather than re-doing the system drivers with every test. I probably need to repeat that here.
I wonder if it might be possible to use them in a Proxmox Server setup.
The K80 is Cuda compute capability 3.7.
Kepler was supported until Cuda 11.8. Afterwards Cuda 12 (released in December 2022) dropped the support dropped its support.
So you should also use LLM frameworks compatible with those Cuda versions and they should not depend on Tensor Cores (introduced with Volta).
As long as you use them not continuously, you should be totally fine, otherwise your electricity bill could be worse than buying a newer GPU.
An advantage of the K80 is that they have fast double precision support (less needed with LLMs).
For the uninitiated, would you please share a reference for the claim “Kepler was supported until Cuda 11.8, dropped afterwards”?
For example:
chapter 1.3 / General CUDA:
- Kepler architecture support is removed from CUDA 12.0.
For a quick, but unofficial overview:
Confirmed! Thank you for the quick response <3
But a question on that wikipedia table. The K80 is sm=3.7, and on that wikipedia table I only see 3.7 going up to CUDA 10.0 - 10.2. Right? So while “Kepler” might go up to CUDA 11.8, the K80 specifically goes up to 10.2. Or am I interpreting wrong?
Can you say, where you see it in the table?
10.0-10.2: 3.0-7.5
11.0: 3.5-8.0
Or do you read it (wrongly) as 3.07.5?
Or another table on that Wikipedia page?
See also
-
For 12.0 (scroll horizontally):
CUDA C++ Programming Guide
You got the right table. But I’m paying attention by column instead, whereas you’re looking at rows.
The K80 is sm 3.7, therefore I look up 3.7 on that table. https://imgur.com/a/LWbiIZe
See the link. I appears to me that 3.7 overlaps with CUDA 6.5, 7.0-7.5, 8.0, 9.0-9.2, 10.0-10.2. No?
EDIT:
But indeed, your 11.8 link seems to settle the point quite definitely - I should be able to use the K80 with CUDA 11.8. Thank you!
No, if there is a number mentioned, it is the minimum CC number for that toolkit version. So the 3.5 of 11.0-11.8 also include the 3.7.
Otherwise e.g.
- 8.0 (A100) would have been supported just with toolkit 11.0 and not with later toolkits, which is not true. It is supported will all toolkits ever since 11.0.
- Or 2.0 would have been supported for 3.0 and 3.1. Then not any longer. And with 7.0 and 8.0 support would have been reactivated.
Read horizontally, then it makes sense :-)
The left number (at the left end of the horizontal green bars) is the minimum cc supported, the right number the maximum cc version.
1 Like
Understood, many thanks!