I recently purchased a used K80 and found that only one non-idle clock is available.
I’m using CUDA 8.0 on a new Linux computer, with PCI-e Gen 3 slots.
nvidia-smi -q -i 0 -d SUPPORTED_CLOCKS
Supported Clocks
Memory : 2505 MHz
Graphics : 562 MHz
Memory : 324 MHz
Graphics : 324 MHz
Is there any way to make the other clock speeds available? With only 562 MHz, the GPU is of limited usefulness. Or is this device damaged? Thank you for any reply.
I agree this looks weird. I would expect several frequencies to be listed, as in the example nvidia-smi output in the blog entry pointed to by txbob. Are you sure this GPU is a K80? Did you buy this used GPU from a reputable dealer? Does your GPU look like the results you get when you do a Google image search for “Tesla K80”?
Can you post the complete, unmodified output of nvidia-smi -q?
txbob: Thank you. I’ve seen that link, which adds to the mystery.
njufa: It looks like a K80, and I think the dealer is reputable, but he may have been duped when he bought it. Here is the complete output:
==============NVSMI LOG==============
Timestamp : Thu Apr 27 17:07:24 2017
Driver Version : 367.48
Attached GPUs : 2
GPU 0000:04:00.0
Product Name : Tesla K80
Product Brand : Tesla
Display Mode : Disabled
Display Active : Disabled
Persistence Mode : Enabled
Accounting Mode : Disabled
Accounting Mode Buffer Size : 1920
Driver Model
Current : N/A
Pending : N/A
Serial Number : 0321215003976
GPU UUID : GPU-e21c5415-4436-42b9-52db-d735b9c8f365
Minor Number : 0
VBIOS Version : 80.21.1F.00.07
MultiGPU Board : Yes
Board ID : 0x200
GPU Part Number : 900-22080-6300-000
Inforom Version
Image Version : 2080.0200.00.04
OEM Object : 1.1
ECC Object : 3.0
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GPU Virtualization Mode
Virtualization mode : None
PCI
Bus : 0x04
Device : 0x00
Domain : 0x0000
Device Id : 0x102D10DE
Bus Id : 0000:04:00.0
Sub System Id : 0x106C10DE
GPU Link Info
PCIe Generation
Max : 3
Current : 1
Link Width
Max : 16x
Current : 16x
Bridge Chip
Type : N/A
Firmware : N/A
Replays since reset : 0
Tx Throughput : N/A
Rx Throughput : N/A
Fan Speed : N/A
Performance State : P8
Clocks Throttle Reasons
Idle : Active
Applications Clocks Setting : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
Sync Boost : Not Active
Unknown : Not Active
FB Memory Usage
Total : 11439 MiB
Used : 0 MiB
Free : 11439 MiB
BAR1 Memory Usage
Total : 16384 MiB
Used : 2 MiB
Free : 16382 MiB
Compute Mode : Exclusive_Process
Utilization
Gpu : 0 %
Memory : 0 %
Encoder : 0 %
Decoder : 0 %
Ecc Mode
Current : Enabled
Pending : Enabled
ECC Errors
Volatile
Single Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Texture Shared : N/A
Total : 0
Double Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Texture Shared : N/A
Total : 0
Aggregate
Single Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Texture Shared : N/A
Total : 0
Double Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Texture Shared : N/A
Total : 0
Retired Pages
Single Bit ECC : 0
Double Bit ECC : 0
Pending : No
Temperature
GPU Current Temp : 33 C
GPU Shutdown Temp : 93 C
GPU Slowdown Temp : 88 C
Power Readings
Power Management : Supported
Power Draw : 26.35 W
Power Limit : 149.00 W
Default Power Limit : 149.00 W
Enforced Power Limit : 149.00 W
Min Power Limit : 100.00 W
Max Power Limit : 175.00 W
Clocks
Graphics : 324 MHz
SM : 324 MHz
Memory : 324 MHz
Video : 405 MHz
Applications Clocks
Graphics : 562 MHz
Memory : 2505 MHz
Default Applications Clocks
Graphics : 562 MHz
Memory : 2505 MHz
Max Clocks
Graphics : 562 MHz
SM : 562 MHz
Memory : 2505 MHz
Video : 540 MHz
Clock Policy
Auto Boost : On
Auto Boost Default : On
Processes : None
GPU 0000:05:00.0
Product Name : Tesla K80
Product Brand : Tesla
Display Mode : Disabled
Display Active : Disabled
Persistence Mode : Enabled
Accounting Mode : Disabled
Accounting Mode Buffer Size : 1920
Driver Model
Current : N/A
Pending : N/A
Serial Number : 0321215003976
GPU UUID : GPU-5696e7b9-8e99-ad48-5bf9-b796d05e9568
Minor Number : 1
VBIOS Version : 80.21.1F.00.08
MultiGPU Board : Yes
Board ID : 0x200
GPU Part Number : 900-22080-6300-000
Inforom Version
Image Version : 2080.0200.00.04
OEM Object : 1.1
ECC Object : 3.0
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GPU Virtualization Mode
Virtualization mode : None
PCI
Bus : 0x05
Device : 0x00
Domain : 0x0000
Device Id : 0x102D10DE
Bus Id : 0000:05:00.0
Sub System Id : 0x106C10DE
GPU Link Info
PCIe Generation
Max : 3
Current : 1
Link Width
Max : 16x
Current : 16x
Bridge Chip
Type : N/A
Firmware : N/A
Replays since reset : 0
Tx Throughput : N/A
Rx Throughput : N/A
Fan Speed : N/A
Performance State : P8
Clocks Throttle Reasons
Idle : Active
Applications Clocks Setting : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
Sync Boost : Not Active
Unknown : Not Active
FB Memory Usage
Total : 11439 MiB
Used : 0 MiB
Free : 11439 MiB
BAR1 Memory Usage
Total : 16384 MiB
Used : 2 MiB
Free : 16382 MiB
Compute Mode : Exclusive_Process
Utilization
Gpu : 0 %
Memory : 0 %
Encoder : 0 %
Decoder : 0 %
Ecc Mode
Current : Enabled
Pending : Enabled
ECC Errors
Volatile
Single Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Texture Shared : N/A
Total : 0
Double Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Texture Shared : N/A
Total : 0
Aggregate
Single Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Texture Shared : N/A
Total : 0
Double Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Texture Shared : N/A
Total : 0
Retired Pages
Single Bit ECC : 0
Double Bit ECC : 0
Pending : No
Temperature
GPU Current Temp : 26 C
GPU Shutdown Temp : 93 C
GPU Slowdown Temp : 88 C
Power Readings
Power Management : Supported
Power Draw : 28.42 W
Power Limit : 149.00 W
Default Power Limit : 149.00 W
Enforced Power Limit : 149.00 W
Min Power Limit : 100.00 W
Max Power Limit : 175.00 W
Clocks
Graphics : 324 MHz
SM : 324 MHz
Memory : 324 MHz
Video : 405 MHz
Applications Clocks
Graphics : 562 MHz
Memory : 2505 MHz
Default Applications Clocks
Graphics : 562 MHz
Memory : 2505 MHz
Max Clocks
Graphics : 562 MHz
SM : 562 MHz
Memory : 2505 MHz
Video : 540 MHz
Clock Policy
Auto Boost : On
Auto Boost Default : On
Processes : None
Also, here is:
nvidia-smi -q -d supported_clocks
==============NVSMI LOG==============
Timestamp : Thu Apr 27 17:31:46 2017
Driver Version : 367.48
Attached GPUs : 2
GPU 0000:04:00.0
Supported Clocks
Memory : 2505 MHz
Graphics : 562 MHz
Memory : 324 MHz
Graphics : 324 MHz
GPU 0000:05:00.0
Supported Clocks
Memory : 2505 MHz
Graphics : 562 MHz
Memory : 324 MHz
Graphics : 324 MHz
I don’t have an explanation as to what is going on, and I don’t have K80 to compare your output with.
Other than the maximum graphics/SM clocks reported, I do not see anything that would suggest that this might not be a K80. 562 MHz is the K80’s regular non-boosted base clock. But the K80 supports boosting up to 875 MHZ, so I would expect a higher clock rate to be listed under “Max Clocks”. The driver seems recent so I don’t a bug in the nvidia-smi reporting is likely.
The memory is reported as 11 GB instead of the 12 GB installed on the K80, but I think this is because ECC is enabled and nvidia-smi reports the useable memory, which is reduced with ECC turned on. Additional information to check whether this is a genuine K80 would be to the output of the deviceQuery sample app that ships with CUDA.
I see no obvious indications that the GPU is insufficiently cooled or operating with an insufficient power supply. Wouldn’t hurt to double check. This is a passively cooled card so requires plenty of airflow provided by a server enclosure. For the power supply, no splitters, adapters, etc should be in the power cables from the PSU.
I assume you have already checked the system log (dmesg) for any messages regarding this GPU? When you run a GPU accelerated application, what are the clocks reported by nvidia-smi while it is running?
Here’s what I see for a K80:
$ nvidia-smi -q | more
==============NVSMI LOG==============
Timestamp : Thu Apr 27 21:46:55 2017
Driver Version : 375.39
Attached GPUs : 8
GPU 0000:06:00.0
Product Name : Tesla K80
Product Brand : Tesla
Display Mode : Disabled
Display Active : Disabled
Persistence Mode : Enabled
Accounting Mode : Enabled
Accounting Mode Buffer Size : 1920
Driver Model
Current : N/A
Pending : N/A
Serial Number : 0323814024923
GPU UUID : GPU-03accf58-c2b2-c91c-3ca0-41e4f25fc123
Minor Number : 0
VBIOS Version : 80.21.1B.00.01
MultiGPU Board : Yes
Board ID : 0x400
GPU Part Number : 900-22080-0000-000
Inforom Version
Image Version : 2080.0200.00.04
OEM Object : 1.1
ECC Object : 3.0
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GPU Virtualization Mode
Virtualization mode : None
PCI
Bus : 0x06
Device : 0x00
Domain : 0x0000
Device Id : 0x102D10DE
Bus Id : 0000:06:00.0
Sub System Id : 0x106C10DE
GPU Link Info
PCIe Generation
Max : 3
Current : 1
Link Width
Max : 16x
Current : 16x
Bridge Chip
Type : N/A
Firmware : N/A
Replays since reset : 0
Tx Throughput : N/A
Rx Throughput : N/A
Fan Speed : N/A
Performance State : P8
Clocks Throttle Reasons
Idle : Active
Applications Clocks Setting : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
Sync Boost : Not Active
Unknown : Not Active
FB Memory Usage
Total : 11439 MiB
Used : 0 MiB
Free : 11439 MiB
BAR1 Memory Usage
Total : 16384 MiB
Used : 2 MiB
Free : 16382 MiB
Compute Mode : Default
Utilization
Gpu : 0 %
Memory : 0 %
Encoder : 0 %
Decoder : 0 %
Ecc Mode
Current : Enabled
Pending : Enabled
ECC Errors
Volatile
Single Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Texture Shared : N/A
Total : 0
Double Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Texture Shared : N/A
Total : 0
Aggregate
Single Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Texture Shared : N/A
Total : 0
Double Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Texture Shared : N/A
Total : 0
Retired Pages
Single Bit ECC : 0
Double Bit ECC : 0
Pending : No
Temperature
GPU Current Temp : 23 C
GPU Shutdown Temp : 93 C
GPU Slowdown Temp : 88 C
Power Readings
Power Management : Supported
Power Draw : 26.55 W
Power Limit : 149.00 W
Default Power Limit : 149.00 W
Enforced Power Limit : 149.00 W
Min Power Limit : 100.00 W
Max Power Limit : 175.00 W
Clocks
Graphics : 324 MHz
SM : 324 MHz
Memory : 324 MHz
Video : 405 MHz
Applications Clocks
Graphics : 562 MHz
Memory : 2505 MHz
Default Applications Clocks
Graphics : 562 MHz
Memory : 2505 MHz
Max Clocks
Graphics : 875 MHz
SM : 875 MHz
Memory : 2505 MHz
Video : 540 MHz
Clock Policy
Auto Boost : On
Auto Boost Default : On
Processes : None
GPU 0000:07:00.0
Product Name : Tesla K80
Product Brand : Tesla
Display Mode : Disabled
Display Active : Disabled
Persistence Mode : Enabled
Accounting Mode : Enabled
Accounting Mode Buffer Size : 1920
Driver Model
Current : N/A
Pending : N/A
Serial Number : 0323814024923
GPU UUID : GPU-216ec34f-ad3d-6069-9ce6-b2b54981ca2d
Minor Number : 1
VBIOS Version : 80.21.1B.00.02
MultiGPU Board : Yes
Board ID : 0x400
GPU Part Number : 900-22080-0000-000
Inforom Version
Image Version : 2080.0200.00.04
OEM Object : 1.1
ECC Object : 3.0
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GPU Virtualization Mode
Virtualization mode : None
PCI
Bus : 0x07
Device : 0x00
Domain : 0x0000
Device Id : 0x102D10DE
Bus Id : 0000:07:00.0
Sub System Id : 0x106C10DE
GPU Link Info
PCIe Generation
Max : 3
Current : 1
Link Width
Max : 16x
Current : 16x
Bridge Chip
Type : N/A
Firmware : N/A
Replays since reset : 0
Tx Throughput : N/A
Rx Throughput : N/A
Fan Speed : N/A
Performance State : P8
Clocks Throttle Reasons
Idle : Active
Applications Clocks Setting : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
Sync Boost : Not Active
Unknown : Not Active
FB Memory Usage
Total : 11439 MiB
Used : 0 MiB
Free : 11439 MiB
BAR1 Memory Usage
Total : 16384 MiB
Used : 2 MiB
Free : 16382 MiB
Compute Mode : Default
Utilization
Gpu : 0 %
Memory : 0 %
Encoder : 0 %
Decoder : 0 %
Ecc Mode
Current : Enabled
Pending : Enabled
ECC Errors
Volatile
Single Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Texture Shared : N/A
Total : 0
Double Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Texture Shared : N/A
Total : 0
Aggregate
Single Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Texture Shared : N/A
Total : 0
Double Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Texture Shared : N/A
Total : 0
Retired Pages
Single Bit ECC : 0
Double Bit ECC : 0
Pending : No
Temperature
GPU Current Temp : 24 C
GPU Shutdown Temp : 93 C
GPU Slowdown Temp : 88 C
Power Readings
Power Management : Supported
Power Draw : 28.45 W
Power Limit : 149.00 W
Default Power Limit : 149.00 W
Enforced Power Limit : 149.00 W
Min Power Limit : 100.00 W
Max Power Limit : 175.00 W
Clocks
Graphics : 324 MHz
SM : 324 MHz
Memory : 324 MHz
Video : 405 MHz
Applications Clocks
Graphics : 562 MHz
Memory : 2505 MHz
Default Applications Clocks
Graphics : 562 MHz
Memory : 2505 MHz
Max Clocks
Graphics : 875 MHz
SM : 875 MHz
Memory : 2505 MHz
Video : 540 MHz
Clock Policy
Auto Boost : On
Auto Boost Default : On
Processes : None
There are differences, but I haven’t researched it yet to see which if any are signficant.
Can you try latest R375 driver for this?
It is good to confirm from txbob’s log that the max clocks for the K80 are reported as 875 MHz, as I expected.
I compared the OP’s and txbob’s log and can’t spot any interesting differences other the max clock issue. It looks like the OP’s K80 is a newer card than txbob’s (looking at serial number and VBIOS versions). My assessment is that, in all likelihood, the OP’s GPU is in fact a real K80, not a fake with a modified VBIOS. It would still be a good idea to examine the output of deviceQuery to make sure.
So why is max clock == K80 base clock for the OP’s GPU? Driver issue? Refurbished card that was flashed with a modified VBIOS to prevent it from running at higher clocks where it might fail? A special OEM model from a dismantled supercomputer?
Here is the deviceQuery output. Meanwhile, I’ll start downloading the latest driver (8.0.61_375.26). (I don’t see a 375.39 for my OS, which is Ubuntu 16.04.). Thank you both for the help. Njuffa, those are interesting possibilities that you mentioned.
deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 2 CUDA Capable device(s)
Device 0: "Tesla K80"
CUDA Driver Version / Runtime Version 8.0 / 8.0
CUDA Capability Major/Minor version number: 3.7
Total amount of global memory: 11440 MBytes (11995578368 bytes)
(13) Multiprocessors, (192) CUDA Cores/MP: 2496 CUDA Cores
GPU Max Clock rate: 562 MHz (0.56 GHz)
Memory Clock rate: 2505 Mhz
Memory Bus Width: 384-bit
L2 Cache Size: 1572864 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 2 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Enabled
Device supports Unified Addressing (UVA): Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 4 / 0
Compute Mode:
< Exclusive Process (many threads in one process is able to use ::cudaSetDevice() with this device) >
Device 1: "Tesla K80"
CUDA Driver Version / Runtime Version 8.0 / 8.0
CUDA Capability Major/Minor version number: 3.7
Total amount of global memory: 11440 MBytes (11995578368 bytes)
(13) Multiprocessors, (192) CUDA Cores/MP: 2496 CUDA Cores
GPU Max Clock rate: 562 MHz (0.56 GHz)
Memory Clock rate: 2505 Mhz
Memory Bus Width: 384-bit
L2 Cache Size: 1572864 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 2 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Enabled
Device supports Unified Addressing (UVA): Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 5 / 0
Compute Mode:
< Exclusive Process (many threads in one process is able to use ::cudaSetDevice() with this device) >
> Peer access from Tesla K80 (GPU0) -> Tesla K80 (GPU1) : Yes
> Peer access from Tesla K80 (GPU1) -> Tesla K80 (GPU0) : Yes
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 8.0, CUDA Runtime Version = 8.0, NumDevs = 2, Device0 = Tesla K80, Device1 = Tesla K80
Result = PASS
The deviceQuery output confirms that this is a K80, all data is as expected.
I installed the newer driver, but the Max Clocks is still 562 MHz.
Here is the nvidia-smi -q using the newer driver. There are differences, but nothing that would seem to suggest a solution, at least to me. I had not enabled Persistence when I ran this, and it appears that the GPU was still doing initialization because it was in state P0. (It’s still in P0 several minutes later, even after compiling and running deviceQuery.) I have two of these, both with this problem; I’m returning one for a refund.
==============NVSMI LOG==============
Timestamp : Fri Apr 28 11:52:15 2017
Driver Version : 375.26
Attached GPUs : 2
GPU 0000:04:00.0
Product Name : Tesla K80
Product Brand : Tesla
Display Mode : Disabled
Display Active : Disabled
Persistence Mode : Disabled
Accounting Mode : Disabled
Accounting Mode Buffer Size : 1920
Driver Model
Current : N/A
Pending : N/A
Serial Number : 0321215003976
GPU UUID : GPU-e21c5415-4436-42b9-52db-d735b9c8f365
Minor Number : 0
VBIOS Version : 80.21.1F.00.07
MultiGPU Board : Yes
Board ID : 0x200
GPU Part Number : 900-22080-6300-000
Inforom Version
Image Version : 2080.0200.00.04
OEM Object : 1.1
ECC Object : 3.0
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GPU Virtualization Mode
Virtualization mode : None
PCI
Bus : 0x04
Device : 0x00
Domain : 0x0000
Device Id : 0x102D10DE
Bus Id : 0000:04:00.0
Sub System Id : 0x106C10DE
GPU Link Info
PCIe Generation
Max : 3
Current : 3
Link Width
Max : 16x
Current : 16x
Bridge Chip
Type : N/A
Firmware : N/A
Replays since reset : 0
Tx Throughput : N/A
Rx Throughput : N/A
Fan Speed : N/A
Performance State : P0
Clocks Throttle Reasons
Idle : Not Active
Applications Clocks Setting : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
Sync Boost : Not Active
Unknown : Not Active
FB Memory Usage
Total : 11439 MiB
Used : 0 MiB
Free : 11439 MiB
BAR1 Memory Usage
Total : 16384 MiB
Used : 2 MiB
Free : 16382 MiB
Compute Mode : Default
Utilization
Gpu : 0 %
Memory : 0 %
Encoder : 0 %
Decoder : 0 %
Ecc Mode
Current : Enabled
Pending : Enabled
ECC Errors
Volatile
Single Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Texture Shared : N/A
Total : 0
Double Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Texture Shared : N/A
Total : 0
Aggregate
Single Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Texture Shared : N/A
Total : 0
Double Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Texture Shared : N/A
Total : 0
Retired Pages
Single Bit ECC : 0
Double Bit ECC : 0
Pending : No
Temperature
GPU Current Temp : 40 C
GPU Shutdown Temp : 93 C
GPU Slowdown Temp : 88 C
Power Readings
Power Management : Supported
Power Draw : 62.19 W
Power Limit : 149.00 W
Default Power Limit : 149.00 W
Enforced Power Limit : 149.00 W
Min Power Limit : 100.00 W
Max Power Limit : 175.00 W
Clocks
Graphics : 562 MHz
SM : 562 MHz
Memory : 2505 MHz
Video : 540 MHz
Applications Clocks
Graphics : 562 MHz
Memory : 2505 MHz
Default Applications Clocks
Graphics : 562 MHz
Memory : 2505 MHz
Max Clocks
Graphics : 562 MHz
SM : 562 MHz
Memory : 2505 MHz
Video : 540 MHz
Clock Policy
Auto Boost : On
Auto Boost Default : On
Processes : None
GPU 0000:05:00.0
Product Name : Tesla K80
Product Brand : Tesla
Display Mode : Disabled
Display Active : Disabled
Persistence Mode : Disabled
Accounting Mode : Disabled
Accounting Mode Buffer Size : 1920
Driver Model
Current : N/A
Pending : N/A
Serial Number : 0321215003976
GPU UUID : GPU-5696e7b9-8e99-ad48-5bf9-b796d05e9568
Minor Number : 1
VBIOS Version : 80.21.1F.00.08
MultiGPU Board : Yes
Board ID : 0x200
GPU Part Number : 900-22080-6300-000
Inforom Version
Image Version : 2080.0200.00.04
OEM Object : 1.1
ECC Object : 3.0
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GPU Virtualization Mode
Virtualization mode : None
PCI
Bus : 0x05
Device : 0x00
Domain : 0x0000
Device Id : 0x102D10DE
Bus Id : 0000:05:00.0
Sub System Id : 0x106C10DE
GPU Link Info
PCIe Generation
Max : 3
Current : 3
Link Width
Max : 16x
Current : 16x
Bridge Chip
Type : N/A
Firmware : N/A
Replays since reset : 0
Tx Throughput : N/A
Rx Throughput : N/A
Fan Speed : N/A
Performance State : P0
Clocks Throttle Reasons
Idle : Not Active
Applications Clocks Setting : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
Sync Boost : Not Active
Unknown : Not Active
FB Memory Usage
Total : 11439 MiB
Used : 0 MiB
Free : 11439 MiB
BAR1 Memory Usage
Total : 16384 MiB
Used : 2 MiB
Free : 16382 MiB
Compute Mode : Default
Utilization
Gpu : 100 %
Memory : 4 %
Encoder : 0 %
Decoder : 0 %
Ecc Mode
Current : Enabled
Pending : Enabled
ECC Errors
Volatile
Single Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Texture Shared : N/A
Total : 0
Double Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Texture Shared : N/A
Total : 0
Aggregate
Single Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Texture Shared : N/A
Total : 0
Double Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : 0
Texture Shared : N/A
Total : 0
Retired Pages
Single Bit ECC : 0
Double Bit ECC : 0
Pending : No
Temperature
GPU Current Temp : 31 C
GPU Shutdown Temp : 93 C
GPU Slowdown Temp : 88 C
Power Readings
Power Management : Supported
Power Draw : 72.00 W
Power Limit : 149.00 W
Default Power Limit : 149.00 W
Enforced Power Limit : 149.00 W
Min Power Limit : 100.00 W
Max Power Limit : 175.00 W
Clocks
Graphics : 562 MHz
SM : 562 MHz
Memory : 2505 MHz
Video : 540 MHz
Applications Clocks
Graphics : 562 MHz
Memory : 2505 MHz
Default Applications Clocks
Graphics : 562 MHz
Memory : 2505 MHz
Max Clocks
Graphics : 562 MHz
SM : 562 MHz
Memory : 2505 MHz
Video : 540 MHz
Clock Policy
Auto Boost : On
Auto Boost Default : On
Processes : None
You asked about the clock rate while a task is running. The clock is 562 MHz then. Here is some output of “nvidia-smi dmon”. I’m using just one of the 2 GPU’s on the K80:
# gpu pwr temp sm mem enc dec mclk pclk
# Idx W C % % % % MHz MHz
0 65 54 92 5 0 0 2505 562
1 29 28 0 0 0 0 324 324
0 91 54 100 54 0 0 2505 562
1 29 28 0 0 0 0 324 324
0 79 54 100 21 0 0 2505 562
1 29 28 0 0 0 0 324 324
0 77 54 100 14 0 0 2505 562
1 29 28 0 0 0 0 324 324
0 64 54 92 5 0 0 2505 562
1 29 28 0 0 0 0 324 324
0 65 54 92 5 0 0 2505 562
1 29 28 0 0 0 0 324 324
0 64 54 91 5 0 0 2505 562
1 29 28 0 0 0 0 324 324
0 65 54 92 5 0 0 2505 562
1 29 28 0 0 0 0 324 324
0 65 54 92 5 0 0 2505 562
1 29 28 0 0 0 0 324 324
0 64 54 92 5 0 0 2505 562
1 29 28 0 0 0 0 324 324
0 84 54 100 18 0 0 2505 562
1 29 28 0 0 0 0 324 324
0 86 54 100 29 0 0 2505 562
1 29 28 0 0 0 0 324 324
0 87 55 100 51 0 0 2505 562
1 29 28 0 0 0 0 324 324
0 64 54 92 5 0 0 2505 562
1 29 28 0 0 0 0 324 324
0 65 54 92 5 0 0 2505 562
1 29 28 0 0 0 0 324 324
0 65 54 81 4 0 0 2505 562
1 29 28 0 0 0 0 324 324
0 65 54 92 5 0 0 2505 562
1 29 28 0 0 0 0 324 324
0 65 54 92 5 0 0 2505 562
1 29 28 0 0 0 0 324 324
0 61 54 0 0 0 0 2505 562
1 29 28 0 0 0 0 324 324
0 93 54 100 53 0 0 2505 562
1 29 28 0 0 0 0 324 324
Thanks for looking into it. I might try flashing the firmware if I get time, but not now.
OK, this confirms that there is no clock boosting and the clock is stuck at the “max clocks” limit printed by nvidia-smi. That’s a good indication that nvidia-smi is printing correct information.
At this point it seems clear that the max clock for this GPU has been clamped at the K80’s basic clock rate of 562 MHz, but it is not clear why. I have never encountered such an issue before, so I am afraid we are stuck.
Upgrading the firmware to the latest available might be a solution, unfortunately I do not know where to get the official VBIOS versions for the K80.
whistles
Looks like there are multiple versions for various different HP branded systems.
gbohannon might want to backup his VBIOS with appropriate utilities and later try one of the other different VBIOS installers to see if any of the versions allow selecting the correct clock speeds. There is also a Kepler BIOS editor floating around that should be able to load the VBIOS images and inspect what clocks are embedded into each VBIOS image before one even gets to flashing. You might have to perhaps to a bit of reverse engineering to extract the individual BIOS images from the posted HP binaries… start with opening each binary in a text editor to see how the extraction routine works and if it can be called without doing any other checks/changes.
Good luck!
Edit: Extracting the files is cake. Just use Notepad++ or what have you to get rid of the script up to the last line feed. Then it is compressed twice, I believe once with gzip, then another time with tar. The second tar reveals two VBIOS files, presumably one for each GPU, and the customary nvflash binary and another hpsetup_generic script in perl that goes through the motions of flashing after again checking everything is gravy.
Great suggestions. Thanks for the inputs. I noticed that the HP options are for Suse and RHEL systems (I only checked 2 of them) but maybe they would work on Ubuntu 16.04, too(?). I also saw something called GPU-Z that might be able to back up the VBIOS. It appears to be a Windows program; I think I can find a Windows box big enough for the GPU. Any experience with it? I’ll try looking for the Kepler BIOS editor. Thanks again for the comments.
More than likely any version will work fine, the differences might be very minor to correct particular issues found in a certified use scenario that may vary with O/S versions.
A few more links:
Load the VBIOS from HP into this tool (the *.rom files) and see what clocks are supported
GPU-z can indeed read and backup VBIOS from a GPU under Windows, but not flash.
The HP provided archives enclose nvflash, which is the binary that allows reading and or writing VBIOS to NVIDIA cards, usable under Linux. I believe the caveat is that the NVIDIA driver needs to be unloaded manually perhaps under linux, whereas the same utility under Windows is able to do it automatically. The hpsetup_generic perl script contains the syntax that is used for doing this under Linux.
For flashing under Windows the corresponding nvflash binary is here: BIOS Flashing Downloads | TechPowerUp
Please make sure you can extract the BIOS with nvflash and load it into kepler bios tweaker and it is detected correctly, meaning it’s not garbage to make sure the tool works correctly before attempting to flash a different version… that’s always good advice for any BIOS flashing… read first and make a backup and THEN write.
As far as experience with the nvflash tool, I’ve used it before, the last time being because a part of the VBIOS of a GTX Titan Black somehow got corrupted and I reflashed the original version again to get rid of the message that was output by nvidia-smi saying so. Since you’re not modifying any of the BIOS files, and only flashing you shouldn’t have any issues. Just make sure to read the documentation, I believe running the binary with no flags will print the usage to the terminal in both Linux and Windows with some example usage.
Another thing to look out for, depending on your system configuration if your motherboard has PLX chips or some weird topology is that the PLX chips will show in the nvflash list of cards, needless to say you would not want to pick those as a flash device, and only your K80 GPU1 and GPU2, however they are designated.
Thank you for the great advice. I’ll try that as soon as possible.
Flashing the BIOS worked for me on this refurb K80. I used GPU-z to grab the current BIOS. I saved a backup, then used the kepler tool with this guide. My K80 now auto-boosts as it should.
As I understand it, the auto-boost feature is only useful when one of the two GK210’s can hog the 300W supply, so maybe whatever server these came out of had that feature restricted for their workflow.
Obviously K80’s are long in the tooth, but there’s still tremendous value for DP calculations at this price point (0.2 $/FLOPS DP). So this could still be relevant until the P100 cards drop in price.
I have to say that K80 pricetag is an incredibly amazing value for 2x12GB GK210s, especially considering it is still supported by CUDA 11. Needs a big case for it to fit outside cooling though.
That fan solution worked for me (the long shroud anyways, I’m not sure I trust the shorter version); it keeps the card below 55C at full speed under full load. In my case, when I’m not using the card I can run at 40% without it being too noisy. A quieter Noctua fan would be nice to try, but I don’t think it would cut it. (The max CFM is like 18 vs the 40 from the Ebay version).
The other difficulty I had was linking the fan pwm to the card temperature. SpeedFan works pretty well in Win10; but in Ubuntu I couldn’t use lm-sensors + fancontrol (as done here), because I couldn’t locate a devpath for the K80.
This is probably a terrible solution, but I ended up just using pwmconfig to find the location of the pwm controls, then wrote a quick script to grab nvidia-smi temperatures and interpolate a desired pwm state. In my case, that meant piping “1” to “/sys/devices/platform/nct6775.656/hwmon/hwmon1/pwm4_enable” followed by the desired pwm value to “/sys/devices/platform/nct6775.656/hwmon/hwmon1/pwm4”
Takes some work, but yeah, amazing value.