K80 application clock limited to 562 Mhz

I recently purchased a used K80 and found that only one non-idle clock is available.
I’m using CUDA 8.0 on a new Linux computer, with PCI-e Gen 3 slots.

nvidia-smi -q -i 0 -d SUPPORTED_CLOCKS
Supported Clocks
Memory : 2505 MHz
Graphics : 562 MHz
Memory : 324 MHz
Graphics : 324 MHz

Is there any way to make the other clock speeds available? With only 562 MHz, the GPU is of limited usefulness. Or is this device damaged? Thank you for any reply.

https://devblogs.nvidia.com/parallelforall/increase-performance-gpu-boost-k80-autoboost/

I agree this looks weird. I would expect several frequencies to be listed, as in the example nvidia-smi output in the blog entry pointed to by txbob. Are you sure this GPU is a K80? Did you buy this used GPU from a reputable dealer? Does your GPU look like the results you get when you do a Google image search for “Tesla K80”?

Can you post the complete, unmodified output of nvidia-smi -q?

txbob: Thank you. I’ve seen that link, which adds to the mystery.

njufa: It looks like a K80, and I think the dealer is reputable, but he may have been duped when he bought it. Here is the complete output:

==============NVSMI LOG==============

Timestamp                           : Thu Apr 27 17:07:24 2017
Driver Version                      : 367.48

Attached GPUs                       : 2
GPU 0000:04:00.0
    Product Name                    : Tesla K80
    Product Brand                   : Tesla
    Display Mode                    : Disabled
    Display Active                  : Disabled
    Persistence Mode                : Enabled
    Accounting Mode                 : Disabled
    Accounting Mode Buffer Size     : 1920
    Driver Model
        Current                     : N/A
        Pending                     : N/A
    Serial Number                   : 0321215003976
    GPU UUID                        : GPU-e21c5415-4436-42b9-52db-d735b9c8f365
    Minor Number                    : 0
    VBIOS Version                   : 80.21.1F.00.07
    MultiGPU Board                  : Yes
    Board ID                        : 0x200
    GPU Part Number                 : 900-22080-6300-000
    Inforom Version
        Image Version               : 2080.0200.00.04
        OEM Object                  : 1.1
        ECC Object                  : 3.0
        Power Management Object     : N/A
    GPU Operation Mode
        Current                     : N/A
        Pending                     : N/A
    GPU Virtualization Mode
        Virtualization mode         : None
    PCI
        Bus                         : 0x04
        Device                      : 0x00
        Domain                      : 0x0000
        Device Id                   : 0x102D10DE
        Bus Id                      : 0000:04:00.0
        Sub System Id               : 0x106C10DE
        GPU Link Info
            PCIe Generation
                Max                 : 3
                Current             : 1
            Link Width
                Max                 : 16x
                Current             : 16x
        Bridge Chip
            Type                    : N/A
            Firmware                : N/A
        Replays since reset         : 0
        Tx Throughput               : N/A
        Rx Throughput               : N/A
    Fan Speed                       : N/A
    Performance State               : P8
    Clocks Throttle Reasons
        Idle                        : Active
        Applications Clocks Setting : Not Active
        SW Power Cap                : Not Active
        HW Slowdown                 : Not Active
        Sync Boost                  : Not Active
        Unknown                     : Not Active
    FB Memory Usage
        Total                       : 11439 MiB
        Used                        : 0 MiB
        Free                        : 11439 MiB
    BAR1 Memory Usage
        Total                       : 16384 MiB
        Used                        : 2 MiB
        Free                        : 16382 MiB
    Compute Mode                    : Exclusive_Process
    Utilization
        Gpu                         : 0 %
        Memory                      : 0 %
        Encoder                     : 0 %
        Decoder                     : 0 %
    Ecc Mode
        Current                     : Enabled
        Pending                     : Enabled
    ECC Errors
        Volatile
            Single Bit            
                Device Memory       : 0
                Register File       : 0
                L1 Cache            : 0
                L2 Cache            : 0
                Texture Memory      : 0
                Texture Shared      : N/A
                Total               : 0
            Double Bit            
                Device Memory       : 0
                Register File       : 0
                L1 Cache            : 0
                L2 Cache            : 0
                Texture Memory      : 0
                Texture Shared      : N/A
                Total               : 0
        Aggregate
            Single Bit            
                Device Memory       : 0
                Register File       : 0
                L1 Cache            : 0
                L2 Cache            : 0
                Texture Memory      : 0
                Texture Shared      : N/A
                Total               : 0
            Double Bit            
                Device Memory       : 0
                Register File       : 0
                L1 Cache            : 0
                L2 Cache            : 0
                Texture Memory      : 0
                Texture Shared      : N/A
                Total               : 0
    Retired Pages
        Single Bit ECC              : 0
        Double Bit ECC              : 0
        Pending                     : No
    Temperature
        GPU Current Temp            : 33 C
        GPU Shutdown Temp           : 93 C
        GPU Slowdown Temp           : 88 C
    Power Readings
        Power Management            : Supported
        Power Draw                  : 26.35 W
        Power Limit                 : 149.00 W
        Default Power Limit         : 149.00 W
        Enforced Power Limit        : 149.00 W
        Min Power Limit             : 100.00 W
        Max Power Limit             : 175.00 W
    Clocks
        Graphics                    : 324 MHz
        SM                          : 324 MHz
        Memory                      : 324 MHz
        Video                       : 405 MHz
    Applications Clocks
        Graphics                    : 562 MHz
        Memory                      : 2505 MHz
    Default Applications Clocks
        Graphics                    : 562 MHz
        Memory                      : 2505 MHz
    Max Clocks
        Graphics                    : 562 MHz
        SM                          : 562 MHz
        Memory                      : 2505 MHz
        Video                       : 540 MHz
    Clock Policy
        Auto Boost                  : On
        Auto Boost Default          : On
    Processes                       : None

GPU 0000:05:00.0
    Product Name                    : Tesla K80
    Product Brand                   : Tesla
    Display Mode                    : Disabled
    Display Active                  : Disabled
    Persistence Mode                : Enabled
    Accounting Mode                 : Disabled
    Accounting Mode Buffer Size     : 1920
    Driver Model
        Current                     : N/A
        Pending                     : N/A
    Serial Number                   : 0321215003976
    GPU UUID                        : GPU-5696e7b9-8e99-ad48-5bf9-b796d05e9568
    Minor Number                    : 1
    VBIOS Version                   : 80.21.1F.00.08
    MultiGPU Board                  : Yes
    Board ID                        : 0x200
    GPU Part Number                 : 900-22080-6300-000
    Inforom Version
        Image Version               : 2080.0200.00.04
        OEM Object                  : 1.1
        ECC Object                  : 3.0
        Power Management Object     : N/A
    GPU Operation Mode
        Current                     : N/A
        Pending                     : N/A
    GPU Virtualization Mode
        Virtualization mode         : None
    PCI
        Bus                         : 0x05
        Device                      : 0x00
        Domain                      : 0x0000
        Device Id                   : 0x102D10DE
        Bus Id                      : 0000:05:00.0
        Sub System Id               : 0x106C10DE
        GPU Link Info
            PCIe Generation
                Max                 : 3
                Current             : 1
            Link Width
                Max                 : 16x
                Current             : 16x
        Bridge Chip
            Type                    : N/A
            Firmware                : N/A
        Replays since reset         : 0
        Tx Throughput               : N/A
        Rx Throughput               : N/A
    Fan Speed                       : N/A
    Performance State               : P8
    Clocks Throttle Reasons
        Idle                        : Active
        Applications Clocks Setting : Not Active
        SW Power Cap                : Not Active
        HW Slowdown                 : Not Active
        Sync Boost                  : Not Active
        Unknown                     : Not Active
    FB Memory Usage
        Total                       : 11439 MiB
        Used                        : 0 MiB
        Free                        : 11439 MiB
    BAR1 Memory Usage
        Total                       : 16384 MiB
        Used                        : 2 MiB
        Free                        : 16382 MiB
    Compute Mode                    : Exclusive_Process
    Utilization
        Gpu                         : 0 %
        Memory                      : 0 %
        Encoder                     : 0 %
        Decoder                     : 0 %
    Ecc Mode
        Current                     : Enabled
        Pending                     : Enabled
    ECC Errors
        Volatile
            Single Bit            
                Device Memory       : 0
                Register File       : 0
                L1 Cache            : 0
                L2 Cache            : 0
                Texture Memory      : 0
                Texture Shared      : N/A
                Total               : 0
            Double Bit            
                Device Memory       : 0
                Register File       : 0
                L1 Cache            : 0
                L2 Cache            : 0
                Texture Memory      : 0
                Texture Shared      : N/A
                Total               : 0
        Aggregate
            Single Bit            
                Device Memory       : 0
                Register File       : 0
                L1 Cache            : 0
                L2 Cache            : 0
                Texture Memory      : 0
                Texture Shared      : N/A
                Total               : 0
            Double Bit            
                Device Memory       : 0
                Register File       : 0
                L1 Cache            : 0
                L2 Cache            : 0
                Texture Memory      : 0
                Texture Shared      : N/A
                Total               : 0
    Retired Pages
        Single Bit ECC              : 0
        Double Bit ECC              : 0
        Pending                     : No
    Temperature
        GPU Current Temp            : 26 C
        GPU Shutdown Temp           : 93 C
        GPU Slowdown Temp           : 88 C
    Power Readings
        Power Management            : Supported
        Power Draw                  : 28.42 W
        Power Limit                 : 149.00 W
        Default Power Limit         : 149.00 W
        Enforced Power Limit        : 149.00 W
        Min Power Limit             : 100.00 W
        Max Power Limit             : 175.00 W
    Clocks
        Graphics                    : 324 MHz
        SM                          : 324 MHz
        Memory                      : 324 MHz
        Video                       : 405 MHz
    Applications Clocks
        Graphics                    : 562 MHz
        Memory                      : 2505 MHz
    Default Applications Clocks
        Graphics                    : 562 MHz
        Memory                      : 2505 MHz
    Max Clocks
        Graphics                    : 562 MHz
        SM                          : 562 MHz
        Memory                      : 2505 MHz
        Video                       : 540 MHz
    Clock Policy
        Auto Boost                  : On
        Auto Boost Default          : On
    Processes                       : None

Also, here is:
nvidia-smi -q -d supported_clocks

==============NVSMI LOG==============

Timestamp                           : Thu Apr 27 17:31:46 2017
Driver Version                      : 367.48

Attached GPUs                       : 2
GPU 0000:04:00.0
    Supported Clocks
        Memory                      : 2505 MHz
            Graphics                : 562 MHz
        Memory                      : 324 MHz
            Graphics                : 324 MHz

GPU 0000:05:00.0
    Supported Clocks
        Memory                      : 2505 MHz
            Graphics                : 562 MHz
        Memory                      : 324 MHz
            Graphics                : 324 MHz

I don’t have an explanation as to what is going on, and I don’t have K80 to compare your output with.

Other than the maximum graphics/SM clocks reported, I do not see anything that would suggest that this might not be a K80. 562 MHz is the K80’s regular non-boosted base clock. But the K80 supports boosting up to 875 MHZ, so I would expect a higher clock rate to be listed under “Max Clocks”. The driver seems recent so I don’t a bug in the nvidia-smi reporting is likely.

The memory is reported as 11 GB instead of the 12 GB installed on the K80, but I think this is because ECC is enabled and nvidia-smi reports the useable memory, which is reduced with ECC turned on. Additional information to check whether this is a genuine K80 would be to the output of the deviceQuery sample app that ships with CUDA.

I see no obvious indications that the GPU is insufficiently cooled or operating with an insufficient power supply. Wouldn’t hurt to double check. This is a passively cooled card so requires plenty of airflow provided by a server enclosure. For the power supply, no splitters, adapters, etc should be in the power cables from the PSU.

I assume you have already checked the system log (dmesg) for any messages regarding this GPU? When you run a GPU accelerated application, what are the clocks reported by nvidia-smi while it is running?

Here’s what I see for a K80:

$ nvidia-smi -q | more

==============NVSMI LOG==============

Timestamp                           : Thu Apr 27 21:46:55 2017
Driver Version                      : 375.39

Attached GPUs                       : 8
GPU 0000:06:00.0
    Product Name                    : Tesla K80
    Product Brand                   : Tesla
    Display Mode                    : Disabled
    Display Active                  : Disabled
    Persistence Mode                : Enabled
    Accounting Mode                 : Enabled
    Accounting Mode Buffer Size     : 1920
    Driver Model
        Current                     : N/A
        Pending                     : N/A
    Serial Number                   : 0323814024923
    GPU UUID                        : GPU-03accf58-c2b2-c91c-3ca0-41e4f25fc123
    Minor Number                    : 0
    VBIOS Version                   : 80.21.1B.00.01
    MultiGPU Board                  : Yes
    Board ID                        : 0x400
    GPU Part Number                 : 900-22080-0000-000
    Inforom Version
        Image Version               : 2080.0200.00.04
        OEM Object                  : 1.1
        ECC Object                  : 3.0
        Power Management Object     : N/A
    GPU Operation Mode
        Current                     : N/A
        Pending                     : N/A
    GPU Virtualization Mode
        Virtualization mode         : None
    PCI
        Bus                         : 0x06
        Device                      : 0x00
        Domain                      : 0x0000
        Device Id                   : 0x102D10DE
        Bus Id                      : 0000:06:00.0
        Sub System Id               : 0x106C10DE
        GPU Link Info
            PCIe Generation
                Max                 : 3
                Current             : 1
            Link Width
                Max                 : 16x
                Current             : 16x
        Bridge Chip
            Type                    : N/A
            Firmware                : N/A
        Replays since reset         : 0
        Tx Throughput               : N/A
        Rx Throughput               : N/A
    Fan Speed                       : N/A
    Performance State               : P8
    Clocks Throttle Reasons
        Idle                        : Active
        Applications Clocks Setting : Not Active
        SW Power Cap                : Not Active
        HW Slowdown                 : Not Active
        Sync Boost                  : Not Active
        Unknown                     : Not Active
    FB Memory Usage
        Total                       : 11439 MiB
        Used                        : 0 MiB
        Free                        : 11439 MiB
    BAR1 Memory Usage
        Total                       : 16384 MiB
        Used                        : 2 MiB
        Free                        : 16382 MiB
    Compute Mode                    : Default
    Utilization
        Gpu                         : 0 %
        Memory                      : 0 %
        Encoder                     : 0 %
        Decoder                     : 0 %
    Ecc Mode
        Current                     : Enabled
        Pending                     : Enabled
    ECC Errors
        Volatile
            Single Bit
                Device Memory       : 0
                Register File       : 0
                L1 Cache            : 0
                L2 Cache            : 0
                Texture Memory      : 0
                Texture Shared      : N/A
                Total               : 0
            Double Bit
                Device Memory       : 0
                Register File       : 0
                L1 Cache            : 0
                L2 Cache            : 0
                Texture Memory      : 0
                Texture Shared      : N/A
                Total               : 0
        Aggregate
            Single Bit
                Device Memory       : 0
                Register File       : 0
                L1 Cache            : 0
                L2 Cache            : 0
                Texture Memory      : 0
                Texture Shared      : N/A
                Total               : 0
            Double Bit
                Device Memory       : 0
                Register File       : 0
                L1 Cache            : 0
                L2 Cache            : 0
                Texture Memory      : 0
                Texture Shared      : N/A
                Total               : 0
    Retired Pages
        Single Bit ECC              : 0
        Double Bit ECC              : 0
        Pending                     : No
    Temperature
        GPU Current Temp            : 23 C
        GPU Shutdown Temp           : 93 C
        GPU Slowdown Temp           : 88 C
    Power Readings
        Power Management            : Supported
        Power Draw                  : 26.55 W
        Power Limit                 : 149.00 W
        Default Power Limit         : 149.00 W
        Enforced Power Limit        : 149.00 W
        Min Power Limit             : 100.00 W
        Max Power Limit             : 175.00 W
    Clocks
        Graphics                    : 324 MHz
        SM                          : 324 MHz
        Memory                      : 324 MHz
        Video                       : 405 MHz
    Applications Clocks
        Graphics                    : 562 MHz
        Memory                      : 2505 MHz
    Default Applications Clocks
        Graphics                    : 562 MHz
        Memory                      : 2505 MHz
    Max Clocks
        Graphics                    : 875 MHz
        SM                          : 875 MHz
        Memory                      : 2505 MHz
        Video                       : 540 MHz
    Clock Policy
        Auto Boost                  : On
        Auto Boost Default          : On
    Processes                       : None

GPU 0000:07:00.0
    Product Name                    : Tesla K80
    Product Brand                   : Tesla
    Display Mode                    : Disabled
    Display Active                  : Disabled
    Persistence Mode                : Enabled
    Accounting Mode                 : Enabled
    Accounting Mode Buffer Size     : 1920
    Driver Model
        Current                     : N/A
        Pending                     : N/A
    Serial Number                   : 0323814024923
    GPU UUID                        : GPU-216ec34f-ad3d-6069-9ce6-b2b54981ca2d
    Minor Number                    : 1
    VBIOS Version                   : 80.21.1B.00.02
    MultiGPU Board                  : Yes
    Board ID                        : 0x400
    GPU Part Number                 : 900-22080-0000-000
    Inforom Version
        Image Version               : 2080.0200.00.04
        OEM Object                  : 1.1
        ECC Object                  : 3.0
        Power Management Object     : N/A
    GPU Operation Mode
        Current                     : N/A
        Pending                     : N/A
    GPU Virtualization Mode
        Virtualization mode         : None
    PCI
        Bus                         : 0x07
        Device                      : 0x00
        Domain                      : 0x0000
        Device Id                   : 0x102D10DE
        Bus Id                      : 0000:07:00.0
        Sub System Id               : 0x106C10DE
        GPU Link Info
            PCIe Generation
                Max                 : 3
                Current             : 1
            Link Width
                Max                 : 16x
                Current             : 16x
        Bridge Chip
            Type                    : N/A
            Firmware                : N/A
        Replays since reset         : 0
        Tx Throughput               : N/A
        Rx Throughput               : N/A
    Fan Speed                       : N/A
    Performance State               : P8
    Clocks Throttle Reasons
        Idle                        : Active
        Applications Clocks Setting : Not Active
        SW Power Cap                : Not Active
        HW Slowdown                 : Not Active
        Sync Boost                  : Not Active
        Unknown                     : Not Active
    FB Memory Usage
        Total                       : 11439 MiB
        Used                        : 0 MiB
        Free                        : 11439 MiB
    BAR1 Memory Usage
        Total                       : 16384 MiB
        Used                        : 2 MiB
        Free                        : 16382 MiB
    Compute Mode                    : Default
    Utilization
        Gpu                         : 0 %
        Memory                      : 0 %
        Encoder                     : 0 %
        Decoder                     : 0 %
    Ecc Mode
        Current                     : Enabled
        Pending                     : Enabled
    ECC Errors
        Volatile
            Single Bit
                Device Memory       : 0
                Register File       : 0
                L1 Cache            : 0
                L2 Cache            : 0
                Texture Memory      : 0
                Texture Shared      : N/A
                Total               : 0
            Double Bit
                Device Memory       : 0
                Register File       : 0
                L1 Cache            : 0
                L2 Cache            : 0
                Texture Memory      : 0
                Texture Shared      : N/A
                Total               : 0
        Aggregate
            Single Bit
                Device Memory       : 0
                Register File       : 0
                L1 Cache            : 0
                L2 Cache            : 0
                Texture Memory      : 0
                Texture Shared      : N/A
                Total               : 0
            Double Bit
                Device Memory       : 0
                Register File       : 0
                L1 Cache            : 0
                L2 Cache            : 0
                Texture Memory      : 0
                Texture Shared      : N/A
                Total               : 0
    Retired Pages
        Single Bit ECC              : 0
        Double Bit ECC              : 0
        Pending                     : No
    Temperature
        GPU Current Temp            : 24 C
        GPU Shutdown Temp           : 93 C
        GPU Slowdown Temp           : 88 C
    Power Readings
        Power Management            : Supported
        Power Draw                  : 28.45 W
        Power Limit                 : 149.00 W
        Default Power Limit         : 149.00 W
        Enforced Power Limit        : 149.00 W
        Min Power Limit             : 100.00 W
        Max Power Limit             : 175.00 W
    Clocks
        Graphics                    : 324 MHz
        SM                          : 324 MHz
        Memory                      : 324 MHz
        Video                       : 405 MHz
    Applications Clocks
        Graphics                    : 562 MHz
        Memory                      : 2505 MHz
    Default Applications Clocks
        Graphics                    : 562 MHz
        Memory                      : 2505 MHz
    Max Clocks
        Graphics                    : 875 MHz
        SM                          : 875 MHz
        Memory                      : 2505 MHz
        Video                       : 540 MHz
    Clock Policy
        Auto Boost                  : On
        Auto Boost Default          : On
    Processes                       : None

There are differences, but I haven’t researched it yet to see which if any are signficant.

Can you try latest R375 driver for this?

It is good to confirm from txbob’s log that the max clocks for the K80 are reported as 875 MHz, as I expected.

I compared the OP’s and txbob’s log and can’t spot any interesting differences other the max clock issue. It looks like the OP’s K80 is a newer card than txbob’s (looking at serial number and VBIOS versions). My assessment is that, in all likelihood, the OP’s GPU is in fact a real K80, not a fake with a modified VBIOS. It would still be a good idea to examine the output of deviceQuery to make sure.

So why is max clock == K80 base clock for the OP’s GPU? Driver issue? Refurbished card that was flashed with a modified VBIOS to prevent it from running at higher clocks where it might fail? A special OEM model from a dismantled supercomputer?

Here is the deviceQuery output. Meanwhile, I’ll start downloading the latest driver (8.0.61_375.26). (I don’t see a 375.39 for my OS, which is Ubuntu 16.04.). Thank you both for the help. Njuffa, those are interesting possibilities that you mentioned.

deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 2 CUDA Capable device(s)

Device 0: "Tesla K80"
  CUDA Driver Version / Runtime Version          8.0 / 8.0
  CUDA Capability Major/Minor version number:    3.7
  Total amount of global memory:                 11440 MBytes (11995578368 bytes)
  (13) Multiprocessors, (192) CUDA Cores/MP:     2496 CUDA Cores
  GPU Max Clock rate:                            562 MHz (0.56 GHz)
  Memory Clock rate:                             2505 Mhz
  Memory Bus Width:                              384-bit
  L2 Cache Size:                                 1572864 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
  Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 4 / 0
  Compute Mode:
     < Exclusive Process (many threads in one process is able to use ::cudaSetDevice() with this device) >

Device 1: "Tesla K80"
  CUDA Driver Version / Runtime Version          8.0 / 8.0
  CUDA Capability Major/Minor version number:    3.7
  Total amount of global memory:                 11440 MBytes (11995578368 bytes)
  (13) Multiprocessors, (192) CUDA Cores/MP:     2496 CUDA Cores
  GPU Max Clock rate:                            562 MHz (0.56 GHz)
  Memory Clock rate:                             2505 Mhz
  Memory Bus Width:                              384-bit
  L2 Cache Size:                                 1572864 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
  Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 5 / 0
  Compute Mode:
     < Exclusive Process (many threads in one process is able to use ::cudaSetDevice() with this device) >
> Peer access from Tesla K80 (GPU0) -> Tesla K80 (GPU1) : Yes
> Peer access from Tesla K80 (GPU1) -> Tesla K80 (GPU0) : Yes

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 8.0, CUDA Runtime Version = 8.0, NumDevs = 2, Device0 = Tesla K80, Device1 = Tesla K80
Result = PASS

The deviceQuery output confirms that this is a K80, all data is as expected.

I installed the newer driver, but the Max Clocks is still 562 MHz.
Here is the nvidia-smi -q using the newer driver. There are differences, but nothing that would seem to suggest a solution, at least to me. I had not enabled Persistence when I ran this, and it appears that the GPU was still doing initialization because it was in state P0. (It’s still in P0 several minutes later, even after compiling and running deviceQuery.) I have two of these, both with this problem; I’m returning one for a refund.

==============NVSMI LOG==============

Timestamp                           : Fri Apr 28 11:52:15 2017
Driver Version                      : 375.26

Attached GPUs                       : 2
GPU 0000:04:00.0
    Product Name                    : Tesla K80
    Product Brand                   : Tesla
    Display Mode                    : Disabled
    Display Active                  : Disabled
    Persistence Mode                : Disabled
    Accounting Mode                 : Disabled
    Accounting Mode Buffer Size     : 1920
    Driver Model
        Current                     : N/A
        Pending                     : N/A
    Serial Number                   : 0321215003976
    GPU UUID                        : GPU-e21c5415-4436-42b9-52db-d735b9c8f365
    Minor Number                    : 0
    VBIOS Version                   : 80.21.1F.00.07
    MultiGPU Board                  : Yes
    Board ID                        : 0x200
    GPU Part Number                 : 900-22080-6300-000
    Inforom Version
        Image Version               : 2080.0200.00.04
        OEM Object                  : 1.1
        ECC Object                  : 3.0
        Power Management Object     : N/A
    GPU Operation Mode
        Current                     : N/A
        Pending                     : N/A
    GPU Virtualization Mode
        Virtualization mode         : None
    PCI
        Bus                         : 0x04
        Device                      : 0x00
        Domain                      : 0x0000
        Device Id                   : 0x102D10DE
        Bus Id                      : 0000:04:00.0
        Sub System Id               : 0x106C10DE
        GPU Link Info
            PCIe Generation
                Max                 : 3
                Current             : 3
            Link Width
                Max                 : 16x
                Current             : 16x
        Bridge Chip
            Type                    : N/A
            Firmware                : N/A
        Replays since reset         : 0
        Tx Throughput               : N/A
        Rx Throughput               : N/A
    Fan Speed                       : N/A
    Performance State               : P0
    Clocks Throttle Reasons
        Idle                        : Not Active
        Applications Clocks Setting : Not Active
        SW Power Cap                : Not Active
        HW Slowdown                 : Not Active
        Sync Boost                  : Not Active
        Unknown                     : Not Active
    FB Memory Usage
        Total                       : 11439 MiB
        Used                        : 0 MiB
        Free                        : 11439 MiB
    BAR1 Memory Usage
        Total                       : 16384 MiB
        Used                        : 2 MiB
        Free                        : 16382 MiB
    Compute Mode                    : Default
    Utilization
        Gpu                         : 0 %
        Memory                      : 0 %
        Encoder                     : 0 %
        Decoder                     : 0 %
    Ecc Mode
        Current                     : Enabled
        Pending                     : Enabled
    ECC Errors
        Volatile
            Single Bit            
                Device Memory       : 0
                Register File       : 0
                L1 Cache            : 0
                L2 Cache            : 0
                Texture Memory      : 0
                Texture Shared      : N/A
                Total               : 0
            Double Bit            
                Device Memory       : 0
                Register File       : 0
                L1 Cache            : 0
                L2 Cache            : 0
                Texture Memory      : 0
                Texture Shared      : N/A
                Total               : 0
        Aggregate
            Single Bit            
                Device Memory       : 0
                Register File       : 0
                L1 Cache            : 0
                L2 Cache            : 0
                Texture Memory      : 0
                Texture Shared      : N/A
                Total               : 0
            Double Bit            
                Device Memory       : 0
                Register File       : 0
                L1 Cache            : 0
                L2 Cache            : 0
                Texture Memory      : 0
                Texture Shared      : N/A
                Total               : 0
    Retired Pages
        Single Bit ECC              : 0
        Double Bit ECC              : 0
        Pending                     : No
    Temperature
        GPU Current Temp            : 40 C
        GPU Shutdown Temp           : 93 C
        GPU Slowdown Temp           : 88 C
    Power Readings
        Power Management            : Supported
        Power Draw                  : 62.19 W
        Power Limit                 : 149.00 W
        Default Power Limit         : 149.00 W
        Enforced Power Limit        : 149.00 W
        Min Power Limit             : 100.00 W
        Max Power Limit             : 175.00 W
    Clocks
        Graphics                    : 562 MHz
        SM                          : 562 MHz
        Memory                      : 2505 MHz
        Video                       : 540 MHz
    Applications Clocks
        Graphics                    : 562 MHz
        Memory                      : 2505 MHz
    Default Applications Clocks
        Graphics                    : 562 MHz
        Memory                      : 2505 MHz
    Max Clocks
        Graphics                    : 562 MHz
        SM                          : 562 MHz
        Memory                      : 2505 MHz
        Video                       : 540 MHz
    Clock Policy
        Auto Boost                  : On
        Auto Boost Default          : On
    Processes                       : None

GPU 0000:05:00.0
    Product Name                    : Tesla K80
    Product Brand                   : Tesla
    Display Mode                    : Disabled
    Display Active                  : Disabled
    Persistence Mode                : Disabled
    Accounting Mode                 : Disabled
    Accounting Mode Buffer Size     : 1920
    Driver Model
        Current                     : N/A
        Pending                     : N/A
    Serial Number                   : 0321215003976
    GPU UUID                        : GPU-5696e7b9-8e99-ad48-5bf9-b796d05e9568
    Minor Number                    : 1
    VBIOS Version                   : 80.21.1F.00.08
    MultiGPU Board                  : Yes
    Board ID                        : 0x200
    GPU Part Number                 : 900-22080-6300-000
    Inforom Version
        Image Version               : 2080.0200.00.04
        OEM Object                  : 1.1
        ECC Object                  : 3.0
        Power Management Object     : N/A
    GPU Operation Mode
        Current                     : N/A
        Pending                     : N/A
    GPU Virtualization Mode
        Virtualization mode         : None
    PCI
        Bus                         : 0x05
        Device                      : 0x00
        Domain                      : 0x0000
        Device Id                   : 0x102D10DE
        Bus Id                      : 0000:05:00.0
        Sub System Id               : 0x106C10DE
        GPU Link Info
            PCIe Generation
                Max                 : 3
                Current             : 3
            Link Width
                Max                 : 16x
                Current             : 16x
        Bridge Chip
            Type                    : N/A
            Firmware                : N/A
        Replays since reset         : 0
        Tx Throughput               : N/A
        Rx Throughput               : N/A
    Fan Speed                       : N/A
    Performance State               : P0
    Clocks Throttle Reasons
        Idle                        : Not Active
        Applications Clocks Setting : Not Active
        SW Power Cap                : Not Active
        HW Slowdown                 : Not Active
        Sync Boost                  : Not Active
        Unknown                     : Not Active
    FB Memory Usage
        Total                       : 11439 MiB
        Used                        : 0 MiB
        Free                        : 11439 MiB
    BAR1 Memory Usage
        Total                       : 16384 MiB
        Used                        : 2 MiB
        Free                        : 16382 MiB
    Compute Mode                    : Default
    Utilization
        Gpu                         : 100 %
        Memory                      : 4 %
        Encoder                     : 0 %
        Decoder                     : 0 %
    Ecc Mode
        Current                     : Enabled
        Pending                     : Enabled
    ECC Errors
        Volatile
            Single Bit            
                Device Memory       : 0
                Register File       : 0
                L1 Cache            : 0
                L2 Cache            : 0
                Texture Memory      : 0
                Texture Shared      : N/A
                Total               : 0
            Double Bit            
                Device Memory       : 0
                Register File       : 0
                L1 Cache            : 0
                L2 Cache            : 0
                Texture Memory      : 0
                Texture Shared      : N/A
                Total               : 0
        Aggregate
            Single Bit            
                Device Memory       : 0
                Register File       : 0
                L1 Cache            : 0
                L2 Cache            : 0
                Texture Memory      : 0
                Texture Shared      : N/A
                Total               : 0
            Double Bit            
                Device Memory       : 0
                Register File       : 0
                L1 Cache            : 0
                L2 Cache            : 0
                Texture Memory      : 0
                Texture Shared      : N/A
                Total               : 0
    Retired Pages
        Single Bit ECC              : 0
        Double Bit ECC              : 0
        Pending                     : No
    Temperature
        GPU Current Temp            : 31 C
        GPU Shutdown Temp           : 93 C
        GPU Slowdown Temp           : 88 C
    Power Readings
        Power Management            : Supported
        Power Draw                  : 72.00 W
        Power Limit                 : 149.00 W
        Default Power Limit         : 149.00 W
        Enforced Power Limit        : 149.00 W
        Min Power Limit             : 100.00 W
        Max Power Limit             : 175.00 W
    Clocks
        Graphics                    : 562 MHz
        SM                          : 562 MHz
        Memory                      : 2505 MHz
        Video                       : 540 MHz
    Applications Clocks
        Graphics                    : 562 MHz
        Memory                      : 2505 MHz
    Default Applications Clocks
        Graphics                    : 562 MHz
        Memory                      : 2505 MHz
    Max Clocks
        Graphics                    : 562 MHz
        SM                          : 562 MHz
        Memory                      : 2505 MHz
        Video                       : 540 MHz
    Clock Policy
        Auto Boost                  : On
        Auto Boost Default          : On
    Processes                       : None

You asked about the clock rate while a task is running. The clock is 562 MHz then. Here is some output of “nvidia-smi dmon”. I’m using just one of the 2 GPU’s on the K80:

# gpu   pwr  temp    sm   mem   enc   dec  mclk  pclk
# Idx     W     C     %     %     %     %   MHz   MHz
    0    65    54    92     5     0     0  2505   562
    1    29    28     0     0     0     0   324   324
    0    91    54   100    54     0     0  2505   562
    1    29    28     0     0     0     0   324   324
    0    79    54   100    21     0     0  2505   562
    1    29    28     0     0     0     0   324   324
    0    77    54   100    14     0     0  2505   562
    1    29    28     0     0     0     0   324   324
    0    64    54    92     5     0     0  2505   562
    1    29    28     0     0     0     0   324   324
    0    65    54    92     5     0     0  2505   562
    1    29    28     0     0     0     0   324   324
    0    64    54    91     5     0     0  2505   562
    1    29    28     0     0     0     0   324   324
    0    65    54    92     5     0     0  2505   562
    1    29    28     0     0     0     0   324   324
    0    65    54    92     5     0     0  2505   562
    1    29    28     0     0     0     0   324   324
    0    64    54    92     5     0     0  2505   562
    1    29    28     0     0     0     0   324   324
    0    84    54   100    18     0     0  2505   562
    1    29    28     0     0     0     0   324   324
    0    86    54   100    29     0     0  2505   562
    1    29    28     0     0     0     0   324   324
    0    87    55   100    51     0     0  2505   562
    1    29    28     0     0     0     0   324   324
    0    64    54    92     5     0     0  2505   562
    1    29    28     0     0     0     0   324   324
    0    65    54    92     5     0     0  2505   562
    1    29    28     0     0     0     0   324   324
    0    65    54    81     4     0     0  2505   562
    1    29    28     0     0     0     0   324   324
    0    65    54    92     5     0     0  2505   562
    1    29    28     0     0     0     0   324   324
    0    65    54    92     5     0     0  2505   562
    1    29    28     0     0     0     0   324   324
    0    61    54     0     0     0     0  2505   562
    1    29    28     0     0     0     0   324   324
    0    93    54   100    53     0     0  2505   562
    1    29    28     0     0     0     0   324   324

Thanks for looking into it. I might try flashing the firmware if I get time, but not now.

OK, this confirms that there is no clock boosting and the clock is stuck at the “max clocks” limit printed by nvidia-smi. That’s a good indication that nvidia-smi is printing correct information.

At this point it seems clear that the max clock for this GPU has been clamped at the K80’s basic clock rate of 562 MHz, but it is not clear why. I have never encountered such an issue before, so I am afraid we are stuck.

Upgrading the firmware to the latest available might be a solution, unfortunately I do not know where to get the official VBIOS versions for the K80.

whistles

K80 VBIOS versions from HP

Looks like there are multiple versions for various different HP branded systems.

gbohannon might want to backup his VBIOS with appropriate utilities and later try one of the other different VBIOS installers to see if any of the versions allow selecting the correct clock speeds. There is also a Kepler BIOS editor floating around that should be able to load the VBIOS images and inspect what clocks are embedded into each VBIOS image before one even gets to flashing. You might have to perhaps to a bit of reverse engineering to extract the individual BIOS images from the posted HP binaries… start with opening each binary in a text editor to see how the extraction routine works and if it can be called without doing any other checks/changes.

Good luck!

Edit: Extracting the files is cake. Just use Notepad++ or what have you to get rid of the script up to the last line feed. Then it is compressed twice, I believe once with gzip, then another time with tar. The second tar reveals two VBIOS files, presumably one for each GPU, and the customary nvflash binary and another hpsetup_generic script in perl that goes through the motions of flashing after again checking everything is gravy.

Great suggestions. Thanks for the inputs. I noticed that the HP options are for Suse and RHEL systems (I only checked 2 of them) but maybe they would work on Ubuntu 16.04, too(?). I also saw something called GPU-Z that might be able to back up the VBIOS. It appears to be a Windows program; I think I can find a Windows box big enough for the GPU. Any experience with it? I’ll try looking for the Kepler BIOS editor. Thanks again for the comments.

More than likely any version will work fine, the differences might be very minor to correct particular issues found in a certified use scenario that may vary with O/S versions.

A few more links:

Load the VBIOS from HP into this tool (the *.rom files) and see what clocks are supported

GPU-z can indeed read and backup VBIOS from a GPU under Windows, but not flash.

The HP provided archives enclose nvflash, which is the binary that allows reading and or writing VBIOS to NVIDIA cards, usable under Linux. I believe the caveat is that the NVIDIA driver needs to be unloaded manually perhaps under linux, whereas the same utility under Windows is able to do it automatically. The hpsetup_generic perl script contains the syntax that is used for doing this under Linux.

For flashing under Windows the corresponding nvflash binary is here: https://www.techpowerup.com/download/bios-flashing/

Please make sure you can extract the BIOS with nvflash and load it into kepler bios tweaker and it is detected correctly, meaning it’s not garbage to make sure the tool works correctly before attempting to flash a different version… that’s always good advice for any BIOS flashing… read first and make a backup and THEN write.

As far as experience with the nvflash tool, I’ve used it before, the last time being because a part of the VBIOS of a GTX Titan Black somehow got corrupted and I reflashed the original version again to get rid of the message that was output by nvidia-smi saying so. Since you’re not modifying any of the BIOS files, and only flashing you shouldn’t have any issues. Just make sure to read the documentation, I believe running the binary with no flags will print the usage to the terminal in both Linux and Windows with some example usage.

Another thing to look out for, depending on your system configuration if your motherboard has PLX chips or some weird topology is that the PLX chips will show in the nvflash list of cards, needless to say you would not want to pick those as a flash device, and only your K80 GPU1 and GPU2, however they are designated.

Thank you for the great advice. I’ll try that as soon as possible.