Hi,
We have a 6 GPU (Ubuntu 16.01) machine with 2 Titans and 4 GTX 780s. Ever since the update to CUDA_8.0.61 we’ve been experiencing periodic crashes. The kernel module becomes unavailable and all GPU-related processes stall. Calling nvidia-smi hangs and cannot be killed. I’ve upgraded the drivers twice and now am running 381.22. I’ve again experienced a crash. On reboot I tried just reloading the kernel modules but the system couldn’t find them. I had to reinstall the driver to get everything running again. The syslog output on crash is this:
Jun 20 14:14:57 server_name kernel: [699655.484089] INFO: task kworker/37:1:37384 blocked for more than 120 seconds.
Jun 20 14:14:57 server_name kernel: [699655.484104] Tainted: P OE 4.4.0-79-generic #100-Ubuntu
Jun 20 14:14:57 server_name kernel: [699655.484115] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
Jun 20 14:14:57 server_name kernel: [699655.484127] kworker/37:1 D ffff885f9535fb78 0 37384 2 0x00000000
Jun 20 14:14:57 server_name kernel: [699655.484228] Workqueue: events os_execute_work_item [nvidia]
Jun 20 14:14:57 server_name kernel: [699655.484230] ffff885f9535fb78 00000000c92994dc ffff886006020e00 ffff885fac65aa00
Jun 20 14:14:57 server_name kernel: [699655.484232] ffff885f95360000 ffff883f81975ca8 ffff885fac65aa00 ffff885f9535fe10
Jun 20 14:14:57 server_name kernel: [699655.484234] ffff883eaa4e41c8 ffff885f9535fb90 ffffffff8183c955 7fffffffffffffff
Jun 20 14:14:57 server_name kernel: [699655.484236] Call Trace:
Jun 20 14:14:57 server_name kernel: [699655.484241] [] schedule+0x35/0x80
Jun 20 14:14:57 server_name kernel: [699655.484243] [] schedule_timeout+0x1b5/0x270
Jun 20 14:14:57 server_name kernel: [699655.484321] [] ? os_acquire_spinlock+0x12/0x20 [nvidia]
Jun 20 14:14:57 server_name kernel: [699655.484384] [] ? os_acquire_spinlock+0x12/0x20 [nvidia]
Jun 20 14:14:57 server_name kernel: [699655.484488] [] ? _nv019565rm+0xc/0x20 [nvidia]
Jun 20 14:14:57 server_name kernel: [699655.484490] [] __down+0x7f/0xd0
Jun 20 14:14:57 server_name kernel: [699655.484494] [] ? do_gettimeofday+0x29/0x90
Jun 20 14:14:57 server_name kernel: [699655.484496] [] down+0x41/0x50
Jun 20 14:14:57 server_name kernel: [699655.484559] [] os_acquire_semaphore+0x37/0x40 [nvidia]
Jun 20 14:14:57 server_name kernel: [699655.484622] [] os_acquire_mutex+0xe/0x10 [nvidia]
Jun 20 14:14:57 server_name kernel: [699655.484720] [] _nv020054rm+0x5c/0x120 [nvidia]
Jun 20 14:14:57 server_name kernel: [699655.484843] [] ? _nv021658rm+0x3ac/0x8a0 [nvidia]
Jun 20 14:14:57 server_name kernel: [699655.484936] [] ? _nv000807rm+0x22b/0xcd0 [nvidia]
Jun 20 14:14:57 server_name kernel: [699655.485028] [] ? rm_execute_work_item+0x49/0xc0 [nvidia]
Jun 20 14:14:57 server_name kernel: [699655.485092] [] ? os_execute_work_item+0x11/0x70 [nvidia]
Jun 20 14:14:57 server_name kernel: [699655.485155] [] ? os_execute_work_item+0x46/0x70 [nvidia]
Jun 20 14:14:57 server_name kernel: [699655.485158] [] ? process_one_work+0x165/0x480
Jun 20 14:14:57 server_name kernel: [699655.485160] [] ? worker_thread+0x4b/0x4c0
Jun 20 14:14:57 server_name kernel: [699655.485161] [] ? process_one_work+0x480/0x480
Jun 20 14:14:57 server_name kernel: [699655.485163] [] ? kthread+0xe5/0x100
Jun 20 14:14:57 server_name kernel: [699655.485165] [] ? kthread_create_on_node+0x1e0/0x1e0
Jun 20 14:14:57 server_name kernel: [699655.485167] [] ? ret_from_fork+0x3f/0x70
Jun 20 14:14:57 server_name kernel: [699655.485169] [] ? kthread_create_on_node+0x1e0/0x1e0
==============NVSMI LOG==============
Timestamp : Tue Jun 20 14:40:11 2017
Driver Version : 381.22
Attached GPUs : 6
GPU 0000:04:00.0
Product Name : GeForce GTX 780
Product Brand : GeForce
Display Mode : N/A
Display Active : N/A
Persistence Mode : Disabled
Accounting Mode : N/A
Accounting Mode Buffer Size : N/A
Driver Model
Current : N/A
Pending : N/A
Serial Number : N/A
GPU UUID : GPU-1dee2eb9-43de-be84-1759-c69480d674d8
Minor Number : 0
VBIOS Version : 80.80.21.00.53
MultiGPU Board : N/A
Board ID : N/A
GPU Part Number : N/A
Inforom Version
Image Version : N/A
OEM Object : N/A
ECC Object : N/A
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GPU Virtualization Mode
Virtualization mode : N/A
PCI
Bus : 0x04
Device : 0x00
Domain : 0x0000
Device Id : 0x100410DE
Bus Id : 0000:04:00.0
Sub System Id : 0x104B196E
GPU Link Info
PCIe Generation
Max : N/A
Current : N/A
Link Width
Max : N/A
Current : N/A
Bridge Chip
Type : N/A
Firmware : N/A
Replays since reset : 0
Tx Throughput : N/A
Rx Throughput : N/A
Fan Speed : 32 %
Performance State : P0
Clocks Throttle Reasons : N/A
FB Memory Usage
Total : 3019 MiB
Used : 0 MiB
Free : 3019 MiB
BAR1 Memory Usage
Total : N/A
Used : N/A
Free : N/A
Compute Mode : Default
Utilization
Gpu : N/A
Memory : N/A
Encoder : N/A
Decoder : N/A
Ecc Mode
Current : N/A
Pending : N/A
ECC Errors
Volatile
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Aggregate
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Retired Pages
Single Bit ECC : N/A
Double Bit ECC : N/A
Pending : N/A
Temperature
GPU Current Temp : 27 C
GPU Shutdown Temp : N/A
GPU Slowdown Temp : N/A
Power Readings
Power Management : N/A
Power Draw : N/A
Power Limit : N/A
Default Power Limit : N/A
Enforced Power Limit : N/A
Min Power Limit : N/A
Max Power Limit : N/A
Clocks
Graphics : N/A
SM : N/A
Memory : N/A
Video : N/A
Applications Clocks
Graphics : 1006 MHz
Memory : 3104 MHz
Default Applications Clocks
Graphics : 1006 MHz
Memory : 3104 MHz
Max Clocks
Graphics : N/A
SM : N/A
Memory : N/A
Video : N/A
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Processes : N/A
GPU 0000:05:00.0
Product Name : GeForce GTX TITAN Black
Product Brand : GeForce
Display Mode : Disabled
Display Active : Disabled
Persistence Mode : Disabled
Accounting Mode : Disabled
Accounting Mode Buffer Size : 1920
Driver Model
Current : N/A
Pending : N/A
Serial Number : 0322414012083
GPU UUID : GPU-4f8989f6-366e-9b98-c5d2-9471418cc036
Minor Number : 1
VBIOS Version : 80.80.4E.00.90
MultiGPU Board : No
Board ID : 0x500
GPU Part Number : N/A
Inforom Version
Image Version : 2083.0031.00.03
OEM Object : 1.1
ECC Object : N/A
Power Management Object : N/A
GPU Operation Mode
Current : Low Double Precision
Pending : Low Double Precision
GPU Virtualization Mode
Virtualization mode : None
PCI
Bus : 0x05
Device : 0x00
Domain : 0x0000
Device Id : 0x100C10DE
Bus Id : 0000:05:00.0
Sub System Id : 0x37903842
GPU Link Info
PCIe Generation
Max : 3
Current : 3
Link Width
Max : 16x
Current : 16x
Bridge Chip
Type : N/A
Firmware : N/A
Replays since reset : 0
Tx Throughput : N/A
Rx Throughput : N/A
Fan Speed : 26 %
Performance State : P0
Clocks Throttle Reasons
Idle : Not Active
Applications Clocks Setting : Active
SW Power Cap : Not Active
HW Slowdown : Not Active
Sync Boost : Not Active
Unknown : Not Active
FB Memory Usage
Total : 6082 MiB
Used : 0 MiB
Free : 6082 MiB
BAR1 Memory Usage
Total : 128 MiB
Used : 2 MiB
Free : 126 MiB
Compute Mode : Default
Utilization
Gpu : 0 %
Memory : 0 %
Encoder : 0 %
Decoder : 0 %
Ecc Mode
Current : N/A
Pending : N/A
ECC Errors
Volatile
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Aggregate
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Retired Pages
Single Bit ECC : N/A
Double Bit ECC : N/A
Pending : N/A
Temperature
GPU Current Temp : 42 C
GPU Shutdown Temp : 100 C
GPU Slowdown Temp : 95 C
Power Readings
Power Management : Supported
Power Draw : 84.36 W
Power Limit : 250.00 W
Default Power Limit : 250.00 W
Enforced Power Limit : 250.00 W
Min Power Limit : 150.00 W
Max Power Limit : 265.00 W
Clocks
Graphics : 888 MHz
SM : 888 MHz
Memory : 3500 MHz
Video : 540 MHz
Applications Clocks
Graphics : 888 MHz
Memory : 3500 MHz
Default Applications Clocks
Graphics : 888 MHz
Memory : 3500 MHz
Max Clocks
Graphics : 1202 MHz
SM : 1202 MHz
Memory : 3500 MHz
Video : 540 MHz
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Processes : None
GPU 0000:09:00.0
Product Name : GeForce GTX 780
Product Brand : GeForce
Display Mode : N/A
Display Active : N/A
Persistence Mode : Disabled
Accounting Mode : N/A
Accounting Mode Buffer Size : N/A
Driver Model
Current : N/A
Pending : N/A
Serial Number : N/A
GPU UUID : GPU-f89b5e8d-1e64-b5be-e9e8-f1b1157b824f
Minor Number : 2
VBIOS Version : 80.80.21.00.53
MultiGPU Board : N/A
Board ID : N/A
GPU Part Number : N/A
Inforom Version
Image Version : N/A
OEM Object : N/A
ECC Object : N/A
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GPU Virtualization Mode
Virtualization mode : N/A
PCI
Bus : 0x09
Device : 0x00
Domain : 0x0000
Device Id : 0x100410DE
Bus Id : 0000:09:00.0
Sub System Id : 0x104B196E
GPU Link Info
PCIe Generation
Max : N/A
Current : N/A
Link Width
Max : N/A
Current : N/A
Bridge Chip
Type : N/A
Firmware : N/A
Replays since reset : 0
Tx Throughput : N/A
Rx Throughput : N/A
Fan Speed : 32 %
Performance State : P0
Clocks Throttle Reasons : N/A
FB Memory Usage
Total : 3020 MiB
Used : 0 MiB
Free : 3020 MiB
BAR1 Memory Usage
Total : N/A
Used : N/A
Free : N/A
Compute Mode : Default
Utilization
Gpu : N/A
Memory : N/A
Encoder : N/A
Decoder : N/A
Ecc Mode
Current : N/A
Pending : N/A
ECC Errors
Volatile
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Aggregate
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Retired Pages
Single Bit ECC : N/A
Double Bit ECC : N/A
Pending : N/A
Temperature
GPU Current Temp : 26 C
GPU Shutdown Temp : N/A
GPU Slowdown Temp : N/A
Power Readings
Power Management : N/A
Power Draw : N/A
Power Limit : N/A
Default Power Limit : N/A
Enforced Power Limit : N/A
Min Power Limit : N/A
Max Power Limit : N/A
Clocks
Graphics : N/A
SM : N/A
Memory : N/A
Video : N/A
Applications Clocks
Graphics : 1006 MHz
Memory : 3104 MHz
Default Applications Clocks
Graphics : 1006 MHz
Memory : 3104 MHz
Max Clocks
Graphics : N/A
SM : N/A
Memory : N/A
Video : N/A
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Processes : N/A
GPU 0000:81:00.0
Product Name : GeForce GTX 780
Product Brand : GeForce
Display Mode : N/A
Display Active : N/A
Persistence Mode : Disabled
Accounting Mode : N/A
Accounting Mode Buffer Size : N/A
Driver Model
Current : N/A
Pending : N/A
Serial Number : N/A
GPU UUID : GPU-68e67fe1-a1aa-d9b0-a3c6-f0a135e08a7d
Minor Number : 3
VBIOS Version : 80.80.21.00.53
MultiGPU Board : N/A
Board ID : N/A
GPU Part Number : N/A
Inforom Version
Image Version : N/A
OEM Object : N/A
ECC Object : N/A
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GPU Virtualization Mode
Virtualization mode : N/A
PCI
Bus : 0x81
Device : 0x00
Domain : 0x0000
Device Id : 0x100410DE
Bus Id : 0000:81:00.0
Sub System Id : 0x104B196E
GPU Link Info
PCIe Generation
Max : N/A
Current : N/A
Link Width
Max : N/A
Current : N/A
Bridge Chip
Type : N/A
Firmware : N/A
Replays since reset : 0
Tx Throughput : N/A
Rx Throughput : N/A
Fan Speed : 32 %
Performance State : P0
Clocks Throttle Reasons : N/A
FB Memory Usage
Total : 3020 MiB
Used : 0 MiB
Free : 3020 MiB
BAR1 Memory Usage
Total : N/A
Used : N/A
Free : N/A
Compute Mode : Default
Utilization
Gpu : N/A
Memory : N/A
Encoder : N/A
Decoder : N/A
Ecc Mode
Current : N/A
Pending : N/A
ECC Errors
Volatile
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Aggregate
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Retired Pages
Single Bit ECC : N/A
Double Bit ECC : N/A
Pending : N/A
Temperature
GPU Current Temp : 27 C
GPU Shutdown Temp : N/A
GPU Slowdown Temp : N/A
Power Readings
Power Management : N/A
Power Draw : N/A
Power Limit : N/A
Default Power Limit : N/A
Enforced Power Limit : N/A
Min Power Limit : N/A
Max Power Limit : N/A
Clocks
Graphics : N/A
SM : N/A
Memory : N/A
Video : N/A
Applications Clocks
Graphics : 1006 MHz
Memory : 3104 MHz
Default Applications Clocks
Graphics : 1006 MHz
Memory : 3104 MHz
Max Clocks
Graphics : N/A
SM : N/A
Memory : N/A
Video : N/A
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Processes : N/A
GPU 0000:84:00.0
Product Name : GeForce GTX TITAN Black
Product Brand : GeForce
Display Mode : Disabled
Display Active : Disabled
Persistence Mode : Disabled
Accounting Mode : Disabled
Accounting Mode Buffer Size : 1920
Driver Model
Current : N/A
Pending : N/A
Serial Number : 0322514046003
GPU UUID : GPU-5cba0884-e167-ff09-42e6-563c5a2ad2e5
Minor Number : 4
VBIOS Version : 80.80.4E.00.90
MultiGPU Board : No
Board ID : 0x8400
GPU Part Number : N/A
Inforom Version
Image Version : 2083.0031.00.03
OEM Object : 1.1
ECC Object : N/A
Power Management Object : N/A
GPU Operation Mode
Current : Low Double Precision
Pending : Low Double Precision
GPU Virtualization Mode
Virtualization mode : None
PCI
Bus : 0x84
Device : 0x00
Domain : 0x0000
Device Id : 0x100C10DE
Bus Id : 0000:84:00.0
Sub System Id : 0x37903842
GPU Link Info
PCIe Generation
Max : 3
Current : 3
Link Width
Max : 16x
Current : 16x
Bridge Chip
Type : N/A
Firmware : N/A
Replays since reset : 0
Tx Throughput : N/A
Rx Throughput : N/A
Fan Speed : 26 %
Performance State : P0
Clocks Throttle Reasons
Idle : Not Active
Applications Clocks Setting : Active
SW Power Cap : Not Active
HW Slowdown : Not Active
Sync Boost : Not Active
Unknown : Not Active
FB Memory Usage
Total : 6082 MiB
Used : 0 MiB
Free : 6082 MiB
BAR1 Memory Usage
Total : 128 MiB
Used : 2 MiB
Free : 126 MiB
Compute Mode : Default
Utilization
Gpu : 0 %
Memory : 0 %
Encoder : 0 %
Decoder : 0 %
Ecc Mode
Current : N/A
Pending : N/A
ECC Errors
Volatile
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Aggregate
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Retired Pages
Single Bit ECC : N/A
Double Bit ECC : N/A
Pending : N/A
Temperature
GPU Current Temp : 33 C
GPU Shutdown Temp : 100 C
GPU Slowdown Temp : 95 C
Power Readings
Power Management : Supported
Power Draw : 82.09 W
Power Limit : 250.00 W
Default Power Limit : 250.00 W
Enforced Power Limit : 250.00 W
Min Power Limit : 150.00 W
Max Power Limit : 265.00 W
Clocks
Graphics : 888 MHz
SM : 888 MHz
Memory : 3500 MHz
Video : 540 MHz
Applications Clocks
Graphics : 888 MHz
Memory : 3500 MHz
Default Applications Clocks
Graphics : 888 MHz
Memory : 3500 MHz
Max Clocks
Graphics : 1202 MHz
SM : 1202 MHz
Memory : 3500 MHz
Video : 540 MHz
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Processes : None
GPU 0000:89:00.0
Product Name : GeForce GTX 780
Product Brand : GeForce
Display Mode : N/A
Display Active : N/A
Persistence Mode : Disabled
Accounting Mode : N/A
Accounting Mode Buffer Size : N/A
Driver Model
Current : N/A
Pending : N/A
Serial Number : N/A
GPU UUID : GPU-bd5a9444-8730-8b56-ac9e-04b0f7b70e12
Minor Number : 5
VBIOS Version : 80.80.21.00.53
MultiGPU Board : N/A
Board ID : N/A
GPU Part Number : N/A
Inforom Version
Image Version : N/A
OEM Object : N/A
ECC Object : N/A
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GPU Virtualization Mode
Virtualization mode : N/A
PCI
Bus : 0x89
Device : 0x00
Domain : 0x0000
Device Id : 0x100410DE
Bus Id : 0000:89:00.0
Sub System Id : 0x104B196E
GPU Link Info
PCIe Generation
Max : N/A
Current : N/A
Link Width
Max : N/A
Current : N/A
Bridge Chip
Type : N/A
Firmware : N/A
Replays since reset : 0
Tx Throughput : N/A
Rx Throughput : N/A
Fan Speed : 32 %
Performance State : P0
Clocks Throttle Reasons : N/A
FB Memory Usage
Total : 3020 MiB
Used : 0 MiB
Free : 3020 MiB
BAR1 Memory Usage
Total : N/A
Used : N/A
Free : N/A
Compute Mode : Default
Utilization
Gpu : N/A
Memory : N/A
Encoder : N/A
Decoder : N/A
Ecc Mode
Current : N/A
Pending : N/A
ECC Errors
Volatile
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Aggregate
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
Total : N/A
Retired Pages
Single Bit ECC : N/A
Double Bit ECC : N/A
Pending : N/A
Temperature
GPU Current Temp : 26 C
GPU Shutdown Temp : N/A
GPU Slowdown Temp : N/A
Power Readings
Power Management : N/A
Power Draw : N/A
Power Limit : N/A
Default Power Limit : N/A
Enforced Power Limit : N/A
Min Power Limit : N/A
Max Power Limit : N/A
Clocks
Graphics : N/A
SM : N/A
Memory : N/A
Video : N/A
Applications Clocks
Graphics : 1006 MHz
Memory : 3104 MHz
Default Applications Clocks
Graphics : 1006 MHz
Memory : 3104 MHz
Max Clocks
Graphics : N/A
SM : N/A
Memory : N/A
Video : N/A
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Processes : N/A