Hello,
I have a strange problem with my GPU setup.
I’m on Windows 11, Core i9-9920X, 128 GB RAM.
I have 3 GPUs: 5090, 6000 Ada and recently added RTX8000.
Everything is installed and configured and confirmed working.
Currently on 576.52 Studio driver.
When all 3 GPUs are turned on (in device manager) any kind of CUDA application fails usually reporting device 1 is busy (no matter on what order the GPUs are).
Or that CUDA context failed with unknown error (Blender)
When any 2 are on everything works fine no errors in multiple applications I’ve tested.
So to simplify everything I’ve got a simple test with pytorch:
python -c ‘import torch; torch.randn(1).to(0)’
With all 3 on:
When I run the test I get:
PS X:\!_Nvidia_problem> python -c 'import torch; torch.randn(1).to(0)' Traceback (most recent call last): File "<string>", line 1, in <module> import torch; torch.randn(1).to(0) ~~~~~~~~~~~~~~~~~^^^ RuntimeError: CUDA error: CUDA-capable device(s) is/are busy or unavailable CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
When I turn off any of those three GPUs test runs just fine.
Also tested with KoboldCPP
ggml_cuda_init: found 3 CUDA devices: Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes Device 1: NVIDIA RTX 6000 Ada Generation, compute capability 8.9, VMM: yes Device 2: Quadro RTX 8000, compute capability 7.5, VMM: yes CUDA error: CUDA-capable device(s) is/are busy or unavailable current device: 1, in function ggml_cuda_set_device at C:\a\koboldcpp\koboldcpp\ggml\src\ggml-cuda\ggml-cuda.cu:90 cudaSetDevice(device) C:\a\koboldcpp\koboldcpp\ggml\src\ggml-cuda\ggml-cuda.cu:77: CUDA error done
This is the most common one also in ComfyUI when generating.
Sometimes it just quits with no error (the simple test I mentioned before acts the same but it happens rarely).
With exact same settings (size of the model doesn’t matter acts same with smallest models) and 2 GPUS no matter which ones all works fine.
Blender refuses to initialize CUDA when all three are present, no matter which ones are turned on in preferences. But when one is turned off in device manager runs without problem.
I’ve also tried to run big LLM model on any 2 GPUs with third turned off, it runs no problem. Then I turn on the third one and start ComfyUI (only app I know how to designate on which GPU should run and designate the third one (–cuda-device n) where n is either 0,1 or 2 I can successfully run Flux alongside LLM on the other 2 GPUs. So all three can work at the same time, just can’t initialize for one app.
I’ve tried to switch 6000 Ada and 8000 to TCC but it changes nothing except the vRAM is not shared (VMM: no).
RTX8000 has no display ports so can run either in TCC or MCDM mode. MCDM seams to cooperate better with Windows as it recognizes it in task manager as NPU.
Here are nvidia-smi queries of each card:
==============NVSMI LOG==============
Timestamp : Thu May 29 11:53:00 2025
Driver Version : 576.52
CUDA Version : 12.9
Attached GPUs : 3
GPU 00000000:C1:00.0
Product Name : NVIDIA GeForce RTX 5090
Product Brand : GeForce
Product Architecture : Blackwell
Display Mode : Requested functionality has been deprecated
Display Attached : Yes
Display Active : Enabled
Persistence Mode : N/A
Addressing Mode : N/A
MIG Mode
Current : N/A
Pending : N/A
Accounting Mode : Disabled
Accounting Mode Buffer Size : 4000
Driver Model
Current : WDDM
Pending : WDDM
Serial Number : 0
GPU UUID : GPU-a4a0e04a-b19d-5b58-33f5-32fa6d7c1a0a
Minor Number : N/A
VBIOS Version : 98.02.2e.40.c0
MultiGPU Board : No
Board ID : 0xc100
Board Part Number : N/A
GPU Part Number : 2B85-300-A1
FRU Part Number : N/A
Platform Info
Chassis Serial Number :
Slot Number : 0
Tray Index : 0
Host ID : 1
Peer Type : Direct Connected
Module Id : 1
GPU Fabric GUID : 0x0000000000000000
Inforom Version
Image Version : G005.0000.98.01
OEM Object : 2.1
ECC Object : N/A
Power Management Object : N/A
Inforom BBX Object Flush
Latest Timestamp : N/A
Latest Duration : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GPU C2C Mode : Disabled
GPU Virtualization Mode
Virtualization Mode : None
Host VGPU Mode : N/A
vGPU Heterogeneous Mode : N/A
GPU Reset Status
Reset Required : Requested functionality has been deprecated
Drain and Reset Recommended : Requested functionality has been deprecated
GPU Recovery Action : None
GSP Firmware Version : N/A
IBMNPU
Relaxed Ordering Mode : N/A
PCI
Bus : 0xC1
Device : 0x00
Domain : 0x0000
Device Id : 0x2B8510DE
Bus Id : 00000000:C1:00.0
Sub System Id : 0x89EC1043
GPU Link Info
PCIe Generation
Max : 3
Current : 3
Device Current : 3
Device Max : 5
Host Max : 3
Link Width
Max : 16x
Current : 16x
Bridge Chip
Type : N/A
Firmware : N/A
Replays Since Reset : 0
Replay Number Rollovers : 0
Tx Throughput : 14689 KB/s
Rx Throughput : 16210 KB/s
Atomic Caps Outbound : N/A
Atomic Caps Inbound : N/A
Fan Speed : 30 %
Performance State : P0
Clocks Event Reasons
Idle : Not Active
Applications Clocks Setting : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
HW Thermal Slowdown : Not Active
HW Power Brake Slowdown : Not Active
Sync Boost : Not Active
SW Thermal Slowdown : Not Active
Display Clock Setting : Not Active
Clocks Event Reasons Counters
SW Power Capping : 219635 us
Sync Boost : 0 us
SW Thermal Slowdown : 0 us
HW Thermal Slowdown : 0 us
HW Power Braking : 0 us
Sparse Operation Mode : N/A
FB Memory Usage
Total : 32607 MiB
Reserved : 506 MiB
Used : 3882 MiB
Free : 28220 MiB
BAR1 Memory Usage
Total : 32768 MiB
Used : 32740 MiB
Free : 28 MiB
Conf Compute Protected Memory Usage
Total : N/A
Used : N/A
Free : N/A
Compute Mode : Default
Utilization
GPU : 5 %
Memory : 1 %
Encoder : 0 %
Decoder : 9 %
JPEG : 0 %
OFA : 0 %
Encoder Stats
Active Sessions : 0
Average FPS : 0
Average Latency : 0
FBC Stats
Active Sessions : 0
Average FPS : 0
Average Latency : 0
DRAM Encryption Mode
Current : Disabled
Pending : Disabled
ECC Mode
Current : N/A
Pending : N/A
ECC Errors
Volatile
SRAM Correctable : N/A
SRAM Uncorrectable Parity : N/A
SRAM Uncorrectable SEC-DED : N/A
DRAM Correctable : N/A
DRAM Uncorrectable : N/A
Aggregate
SRAM Correctable : N/A
SRAM Uncorrectable Parity : N/A
SRAM Uncorrectable SEC-DED : N/A
DRAM Correctable : N/A
DRAM Uncorrectable : N/A
SRAM Threshold Exceeded : N/A
Aggregate Uncorrectable SRAM Sources
SRAM L2 : N/A
SRAM SM : N/A
SRAM Microcontroller : N/A
SRAM PCIE : N/A
SRAM Other : N/A
Retired Pages
Single Bit ECC : N/A
Double Bit ECC : N/A
Pending Page Blacklist : N/A
Remapped Rows : N/A
Temperature
GPU Current Temp : 32 C
GPU T.Limit Temp : 57 C
GPU Shutdown T.Limit Temp : -5 C
GPU Slowdown T.Limit Temp : -2 C
GPU Max Operating T.Limit Temp : 0 C
GPU Target Temperature : N/A
Memory Current Temp : N/A
Memory Max Operating T.Limit Temp : N/A
GPU Power Readings
Average Power Draw : 61.96 W
Instantaneous Power Draw : 63.83 W
Current Power Limit : 450.00 W
Requested Power Limit : 450.00 W
Default Power Limit : 600.00 W
Min Power Limit : 400.00 W
Max Power Limit : 600.00 W
GPU Memory Power Readings
Average Power Draw : N/A
Instantaneous Power Draw : N/A
Module Power Readings
Average Power Draw : N/A
Instantaneous Power Draw : N/A
Current Power Limit : N/A
Requested Power Limit : N/A
Default Power Limit : N/A
Min Power Limit : N/A
Max Power Limit : N/A
Power Smoothing : N/A
Workload Power Profiles
Requested Profiles : N/A
Enforced Profiles : N/A
Clocks
Graphics : 907 MHz
SM : 907 MHz
Memory : 14001 MHz
Video : 1320 MHz
Applications Clocks
Graphics : N/A
Memory : N/A
Default Applications Clocks
Graphics : N/A
Memory : N/A
Deferred Clocks
Memory : N/A
Max Clocks
Graphics : 3090 MHz
SM : 3090 MHz
Memory : 14001 MHz
Video : 3090 MHz
Max Customer Boost Clocks
Graphics : N/A
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Voltage
Graphics : Requested functionality has been deprecated
Fabric
State : N/A
Status : N/A
CliqueId : N/A
ClusterUUID : N/A
Health
Bandwidth : N/A
Route Recovery in progress : N/A
Route Unhealthy : N/A
Access Timeout Recovery : N/A
Processes
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 2364
Type : C+G
Name :
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 3396
Type : C+G
Name : C:\Program Files\WindowsApps\MSTeams_25094.310.3616.953_x64__8wekyb3d8bbwe\ms-teams.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 4500
Type : C+G
Name : C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.22.11141.0_x64__8wekyb3d8bbwe\WindowsTerminal.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 5596
Type : C+G
Name : C:\Program Files\WindowsApps\MSTeams_25094.310.3616.953_x64__8wekyb3d8bbwe\ms-teams.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 8460
Type : C+G
Name : C:\Users\Thoron\AppData\Local\Programs\signal-desktop\Signal.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 9176
Type : C+G
Name : D:\Program Files\Session\Session.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 9620
Type : C+G
Name : D:\Program Files\QNAP\QVR Pro Client\QVR Pro Client.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 9964
Type : C+G
Name : D:\Users\Thoron\AppData\Local\Vivaldi\Application\vivaldi.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 13284
Type : C+G
Name : C:\Program Files\WindowsApps\AdobeNotificationClient_6.0.0.1_x86__enpm4xejd91yc\AdobeNotificationClient.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 13932
Type : C+G
Name : C:\Windows\explorer.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 13960
Type : C+G
Name : C:\Windows\System32\ShellHost.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 16628
Type : C+G
Name : C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 17416
Type : C+G
Name : C:\Windows\System32\ApplicationFrameHost.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 17440
Type : C+G
Name : C:\Program Files\WindowsApps\Microsoft.MicrosoftStickyNotes_6.1.4.0_x64__8wekyb3d8bbwe\Microsoft.Notes.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 17576
Type : C+G
Name : C:\Program Files\WindowsApps\MicrosoftWindows.Client.WebExperience_525.10401.30.0_x64__cw5n1h2txyewy\WidgetBoard.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 17616
Type : C+G
Name : C:\Windows\SystemApps\MicrosoftWindows.Client.CBS_cw5n1h2txyewy\SearchHost.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 19400
Type : C+G
Name : C:\Windows\SystemApps\ShellExperienceHost_cw5n1h2txyewy\ShellExperienceHost.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 19596
Type : C+G
Name : C:\Windows\explorer.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 20340
Type : C+G
Name : C:\Windows\SystemApps\Microsoft.LockApp_cw5n1h2txyewy\LockApp.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 21036
Type : C+G
Name : C:\Windows\SystemApps\MicrosoftWindows.Client.CBS_cw5n1h2txyewy\TextInputHost.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 21052
Type : C+G
Name : C:\Program Files (x86)\Microsoft\EdgeWebView\Application\136.0.3240.92\msedgewebview2.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 21508
Type : C+G
Name : C:\Program Files\Mozilla Firefox\firefox.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 21772
Type : C+G
Name : C:\Program Files\NZXT CAM\NZXT CAM.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 22356
Type : C+G
Name : C:\Program Files\LGHUB\system_tray\lghub_system_tray.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 24336
Type : C+G
Name : D:\Users\Thoron\AppData\Local\Vivaldi\Application\vivaldi.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 24528
Type : C+G
Name : C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 27460
Type : C+G
Name :
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 28012
Type : C+G
Name : D:\Program Files (x86)\VMware\VMware Workstation\x64\mksSandbox.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 28692
Type : C+G
Name : C:\Program Files\Mozilla Firefox\firefox.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 31884
Type : C+G
Name : C:\Program Files (x86)\Microsoft\EdgeWebView\Application\136.0.3240.92\msedgewebview2.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 34764
Type : C+G
Name :
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 39632
Type : C+G
Name : D:\Program Files\Session\Session.exe
Used GPU Memory : Not available in WDDM driver model
Capabilities
EGM : disabled
==============NVSMI LOG==============
Timestamp : Thu May 29 11:52:47 2025
Driver Version : 576.52
CUDA Version : 12.9
Attached GPUs : 3
GPU 00000000:A1:00.0
Product Name : NVIDIA RTX 6000 Ada Generation
Product Brand : NVIDIA RTX
Product Architecture : Ada Lovelace
Display Mode : Requested functionality has been deprecated
Display Attached : No
Display Active : Disabled
Persistence Mode : N/A
Addressing Mode : N/A
MIG Mode
Current : N/A
Pending : N/A
Accounting Mode : Disabled
Accounting Mode Buffer Size : 4000
Driver Model
Current : WDDM
Pending : WDDM
Serial Number : 1791225053806
GPU UUID : GPU-d41ff7ba-43d9-db12-e2a8-e08213b10599
Minor Number : N/A
VBIOS Version : 95.02.59.00.09
MultiGPU Board : No
Board ID : 0xa100
Board Part Number : 900-5G133-2250-000
GPU Part Number : 26B1-870-A1
FRU Part Number : N/A
Platform Info
Chassis Serial Number : N/A
Slot Number : N/A
Tray Index : N/A
Host ID : N/A
Peer Type : N/A
Module Id : 1
GPU Fabric GUID : N/A
Inforom Version
Image Version : G133.0510.00.02
OEM Object : 2.1
ECC Object : 6.16
Power Management Object : N/A
Inforom BBX Object Flush
Latest Timestamp : N/A
Latest Duration : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GPU C2C Mode : N/A
GPU Virtualization Mode
Virtualization Mode : None
Host VGPU Mode : N/A
vGPU Heterogeneous Mode : N/A
GPU Reset Status
Reset Required : Requested functionality has been deprecated
Drain and Reset Recommended : Requested functionality has been deprecated
GPU Recovery Action : None
GSP Firmware Version : N/A
IBMNPU
Relaxed Ordering Mode : N/A
PCI
Bus : 0xA1
Device : 0x00
Domain : 0x0000
Device Id : 0x26B110DE
Bus Id : 00000000:A1:00.0
Sub System Id : 0x16A110DE
GPU Link Info
PCIe Generation
Max : 3
Current : 1
Device Current : 1
Device Max : 4
Host Max : 3
Link Width
Max : 16x
Current : 16x
Bridge Chip
Type : N/A
Firmware : N/A
Replays Since Reset : 0
Replay Number Rollovers : 0
Tx Throughput : 50 KB/s
Rx Throughput : 550 KB/s
Atomic Caps Outbound : N/A
Atomic Caps Inbound : N/A
Fan Speed : 30 %
Performance State : P8
Clocks Event Reasons
Idle : Active
Applications Clocks Setting : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
HW Thermal Slowdown : Not Active
HW Power Brake Slowdown : Not Active
Sync Boost : Not Active
SW Thermal Slowdown : Not Active
Display Clock Setting : Not Active
Clocks Event Reasons Counters
SW Power Capping : 0 us
Sync Boost : 0 us
SW Thermal Slowdown : 0 us
HW Thermal Slowdown : 0 us
HW Power Braking : 0 us
Sparse Operation Mode : N/A
FB Memory Usage
Total : 49140 MiB
Reserved : 727 MiB
Used : 0 MiB
Free : 48413 MiB
BAR1 Memory Usage
Total : 65536 MiB
Used : 49156 MiB
Free : 16380 MiB
Conf Compute Protected Memory Usage
Total : N/A
Used : N/A
Free : N/A
Compute Mode : Default
Utilization
GPU : 0 %
Memory : 0 %
Encoder : 0 %
Decoder : 0 %
JPEG : 0 %
OFA : 0 %
Encoder Stats
Active Sessions : 0
Average FPS : 0
Average Latency : 0
FBC Stats
Active Sessions : 0
Average FPS : 0
Average Latency : 0
DRAM Encryption Mode
Current : N/A
Pending : N/A
ECC Mode
Current : Disabled
Pending : Disabled
ECC Errors
Volatile
SRAM Correctable : N/A
SRAM Uncorrectable Parity : N/A
SRAM Uncorrectable SEC-DED : N/A
DRAM Correctable : N/A
DRAM Uncorrectable : N/A
Aggregate
SRAM Correctable : N/A
SRAM Uncorrectable Parity : N/A
SRAM Uncorrectable SEC-DED : N/A
DRAM Correctable : N/A
DRAM Uncorrectable : N/A
SRAM Threshold Exceeded : N/A
Aggregate Uncorrectable SRAM Sources
SRAM L2 : N/A
SRAM SM : N/A
SRAM Microcontroller : N/A
SRAM PCIE : N/A
SRAM Other : N/A
Retired Pages
Single Bit ECC : N/A
Double Bit ECC : N/A
Pending Page Blacklist : N/A
Remapped Rows
Correctable Error : 0
Uncorrectable Error : 0
Pending : No
Remapping Failure Occurred : No
Bank Remap Availability Histogram
Max : 192 bank(s)
High : 0 bank(s)
Partial : 0 bank(s)
Low : 0 bank(s)
None : 0 bank(s)
Temperature
GPU Current Temp : 36 C
GPU T.Limit Temp : 55 C
GPU Shutdown T.Limit Temp : -7 C
GPU Slowdown T.Limit Temp : -2 C
GPU Max Operating T.Limit Temp : 0 C
GPU Target Temperature : 85 C
Memory Current Temp : N/A
Memory Max Operating T.Limit Temp : N/A
GPU Power Readings
Average Power Draw : 8.70 W
Instantaneous Power Draw : 8.79 W
Current Power Limit : 225.00 W
Requested Power Limit : 225.00 W
Default Power Limit : 300.00 W
Min Power Limit : 100.00 W
Max Power Limit : 300.00 W
GPU Memory Power Readings
Average Power Draw : N/A
Instantaneous Power Draw : N/A
Module Power Readings
Average Power Draw : N/A
Instantaneous Power Draw : N/A
Current Power Limit : N/A
Requested Power Limit : N/A
Default Power Limit : N/A
Min Power Limit : N/A
Max Power Limit : N/A
Power Smoothing : N/A
Workload Power Profiles
Requested Profiles : N/A
Enforced Profiles : N/A
Clocks
Graphics : 210 MHz
SM : 210 MHz
Memory : 405 MHz
Video : 1185 MHz
Applications Clocks
Graphics : 2505 MHz
Memory : 10001 MHz
Default Applications Clocks
Graphics : 2505 MHz
Memory : 10001 MHz
Deferred Clocks
Memory : N/A
Max Clocks
Graphics : 3105 MHz
SM : 3105 MHz
Memory : 10001 MHz
Video : 2415 MHz
Max Customer Boost Clocks
Graphics : N/A
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Voltage
Graphics : Requested functionality has been deprecated
Fabric
State : N/A
Status : N/A
CliqueId : N/A
ClusterUUID : N/A
Health
Bandwidth : N/A
Route Recovery in progress : N/A
Route Unhealthy : N/A
Access Timeout Recovery : N/A
Processes : None
Capabilities
EGM : disabled
==============NVSMI LOG==============
Timestamp : Thu May 29 11:53:14 2025
Driver Version : 576.52
CUDA Version : 12.9
Attached GPUs : 3
GPU 00000000:E1:00.0
Product Name : Quadro RTX 8000
Product Brand : NVIDIA
Product Architecture : Turing
Display Mode : Requested functionality has been deprecated
Display Attached : No
Display Active : Disabled
Persistence Mode : N/A
Addressing Mode : N/A
MIG Mode
Current : N/A
Pending : N/A
Accounting Mode : Disabled
Accounting Mode Buffer Size : 4000
Driver Model
Current : MCDM
Pending : MCDM
Serial Number : 1324420022419
GPU UUID : GPU-e6babc56-1f6d-da32-a65f-c51266d74f93
Minor Number : N/A
VBIOS Version : 90.02.4e.00.03
MultiGPU Board : No
Board ID : 0xe100
Board Part Number : 900-2G150-0150-030
GPU Part Number : 1E78-895-A1
FRU Part Number : N/A
Platform Info
Chassis Serial Number : N/A
Slot Number : N/A
Tray Index : N/A
Host ID : N/A
Peer Type : N/A
Module Id : 1
GPU Fabric GUID : N/A
Inforom Version
Image Version : G150.0231.00.02
OEM Object : 1.1
ECC Object : 5.0
Power Management Object : N/A
Inforom BBX Object Flush
Latest Timestamp : N/A
Latest Duration : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GPU C2C Mode : N/A
GPU Virtualization Mode
Virtualization Mode : None
Host VGPU Mode : N/A
vGPU Heterogeneous Mode : N/A
GPU Reset Status
Reset Required : Requested functionality has been deprecated
Drain and Reset Recommended : Requested functionality has been deprecated
GPU Recovery Action : None
GSP Firmware Version : N/A
IBMNPU
Relaxed Ordering Mode : N/A
PCI
Bus : 0xE1
Device : 0x00
Domain : 0x0000
Device Id : 0x1E7810DE
Bus Id : 00000000:E1:00.0
Sub System Id : 0x13D810DE
GPU Link Info
PCIe Generation
Max : 3
Current : 3
Device Current : 3
Device Max : 3
Host Max : 3
Link Width
Max : 16x
Current : 8x
Bridge Chip
Type : N/A
Firmware : N/A
Replays Since Reset : 0
Replay Number Rollovers : 0
Tx Throughput : 50 KB/s
Rx Throughput : 50 KB/s
Atomic Caps Outbound : N/A
Atomic Caps Inbound : N/A
Fan Speed : N/A
Performance State : P8
Clocks Event Reasons
Idle : Active
Applications Clocks Setting : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
HW Thermal Slowdown : Not Active
HW Power Brake Slowdown : Not Active
Sync Boost : Not Active
SW Thermal Slowdown : Not Active
Display Clock Setting : Not Active
Clocks Event Reasons Counters
SW Power Capping : 157900434 us
Sync Boost : 0 us
SW Thermal Slowdown : 0 us
HW Thermal Slowdown : 0 us
HW Power Braking : 0 us
Sparse Operation Mode : N/A
FB Memory Usage
Total : 49152 MiB
Reserved : 552 MiB
Used : 0 MiB
Free : 48601 MiB
BAR1 Memory Usage
Total : 32768 MiB
Used : 32739 MiB
Free : 29 MiB
Conf Compute Protected Memory Usage
Total : N/A
Used : N/A
Free : N/A
Compute Mode : Default
Utilization
GPU : 0 %
Memory : 0 %
Encoder : 0 %
Decoder : 0 %
JPEG : 0 %
OFA : 0 %
Encoder Stats
Active Sessions : 0
Average FPS : 0
Average Latency : 0
FBC Stats
Active Sessions : 0
Average FPS : 0
Average Latency : 0
DRAM Encryption Mode
Current : N/A
Pending : N/A
ECC Mode
Current : Disabled
Pending : Disabled
ECC Errors
Volatile
SRAM Correctable : N/A
SRAM Uncorrectable : N/A
DRAM Correctable : N/A
DRAM Uncorrectable : N/A
Aggregate
SRAM Correctable : N/A
SRAM Uncorrectable : N/A
DRAM Correctable : N/A
DRAM Uncorrectable : N/A
Retired Pages
Single Bit ECC : 0
Double Bit ECC : 0
Pending Page Blacklist : No
Remapped Rows : N/A
Temperature
GPU Current Temp : 29 C
GPU T.Limit Temp : N/A
GPU Shutdown Temp : 87 C
GPU Slowdown Temp : 84 C
GPU Max Operating Temp : 82 C
GPU Target Temperature : N/A
Memory Current Temp : N/A
Memory Max Operating Temp : N/A
GPU Power Readings
Average Power Draw : N/A
Instantaneous Power Draw : 18.37 W
Current Power Limit : 250.00 W
Requested Power Limit : 250.00 W
Default Power Limit : 250.00 W
Min Power Limit : 150.00 W
Max Power Limit : 250.00 W
GPU Memory Power Readings
Average Power Draw : N/A
Instantaneous Power Draw : N/A
Module Power Readings
Average Power Draw : N/A
Instantaneous Power Draw : N/A
Current Power Limit : N/A
Requested Power Limit : N/A
Default Power Limit : N/A
Min Power Limit : N/A
Max Power Limit : N/A
Power Smoothing : N/A
Workload Power Profiles
Requested Profiles : N/A
Enforced Profiles : N/A
Clocks
Graphics : 300 MHz
SM : 300 MHz
Memory : 405 MHz
Video : 540 MHz
Applications Clocks
Graphics : 1230 MHz
Memory : 6501 MHz
Default Applications Clocks
Graphics : 1230 MHz
Memory : 6501 MHz
Deferred Clocks
Memory : N/A
Max Clocks
Graphics : 1620 MHz
SM : 1620 MHz
Memory : 6501 MHz
Video : 1500 MHz
Max Customer Boost Clocks
Graphics : 1620 MHz
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Voltage
Graphics : Requested functionality has been deprecated
Fabric
State : N/A
Status : N/A
CliqueId : N/A
ClusterUUID : N/A
Health
Bandwidth : N/A
Route Recovery in progress : N/A
Route Unhealthy : N/A
Access Timeout Recovery : N/A
Processes : None
Capabilities
EGM : disabled
Also installed newest CUDA Toolkit.
I’ve been scratching my head for a while now and done so many tests. I’m sure its either something I forgot to setup or its just not possible for some reason.
Also I’m just a hobbyist and not all that experienced.
Haven’t tried running Ubuntu and setting everything up but I’d prefer it working on Windows just out of convenience.
If it was always the same GPU failing I’d say bad GPU but all three work fine in pairs so this is what makes it so frustrating :)
So any ideas welcome :)
And I’m sorry if I’ve posted it in the wrong section, its my first time :)
Also can attach just one picture cause I’m new so I’m sorry for bad formatting.