cudaGetDeviceCount returned 100 -> no CUDA-capable device is connected

Error

After installing CUDA 11.8 with NVIDIA 520.61.05 drivers, deviceQuery throws a 100 error:

./deviceQuery Starting...

CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 100
-> no CUDA-capable device is detected
Result = FAIL

System Information

lspci | grep -i nvidia returns:
04:00.0 VGA compatible controller: NVIDIA Corporation GA106 [GeForce RTX 3060 Lite Hash Rate] (rev a1)

uname -m && /etc/*release returns
x86_64
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION=“Ubuntu 22.04.1 LTS”
PRETTY_NAME=“Ubuntu 22.04.1 LTS”
NAME=“Ubuntu”
VERSION_ID=“22.04”
VERSION=“22.04.1 LTS (Jammy Jellyfish)”
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
"

UBUNTU_CODENAME=jammy

Lots of the issues that seem to duplicate this seem to be driver issues, but it doesn’t seem to be the case here:

==============NVSMI LOG==============

Timestamp : Thu Oct 27 19:17:09 2022
Driver Version : 520.61.05
CUDA Version : 11.8

Attached GPUs : 1
GPU 00000000:04:00.0
Product Name : NVIDIA GeForce RTX 3060
Product Brand : GeForce
Product Architecture : Ampere
Display Mode : Enabled
Display Active : Enabled
Persistence Mode : Enabled
MIG Mode
Current : N/A
Pending : N/A
Accounting Mode : Disabled
Accounting Mode Buffer Size : 4000
Driver Model
Current : N/A
Pending : N/A
Serial Number : N/A
GPU UUID : GPU-d1265ffd-6a59-16cc-0196-18d715282c4b
Minor Number : 0
VBIOS Version : 94.06.2F.00.9D
MultiGPU Board : No
Board ID : 0x400
GPU Part Number : N/A
Module ID : 0
Inforom Version
Image Version : G001.0000.03.03
OEM Object : 2.0
ECC Object : N/A
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GSP Firmware Version : N/A
GPU Virtualization Mode
Virtualization Mode : None
Host VGPU Mode : N/A
IBMNPU
Relaxed Ordering Mode : N/A
PCI
Bus : 0x04
Device : 0x00
Domain : 0x0000
Device Id : 0x250410DE
Bus Id : 00000000:04:00.0
Sub System Id : 0x39761462
GPU Link Info
PCIe Generation
Max : 3
Current : 1
Link Width
Max : 16x
Current : 4x
Bridge Chip
Type : N/A
Firmware : N/A
Replays Since Reset : 0
Replay Number Rollovers : 0
Tx Throughput : 1000 KB/s
Rx Throughput : 0 KB/s
Fan Speed : 0 %
Performance State : P8
Clocks Throttle Reasons
Idle : Active
Applications Clocks Setting : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
HW Thermal Slowdown : Not Active
HW Power Brake Slowdown : Not Active
Sync Boost : Not Active
SW Thermal Slowdown : Not Active
Display Clock Setting : Not Active
FB Memory Usage
Total : 12288 MiB
Reserved : 235 MiB
Used : 345 MiB
Free : 11707 MiB
BAR1 Memory Usage
Total : 256 MiB
Used : 5 MiB
Free : 251 MiB
Compute Mode : Default
Utilization
Gpu : 3 %
Memory : 26 %
Encoder : 0 %
Decoder : 0 %
Encoder Stats
Active Sessions : 0
Average FPS : 0
Average Latency : 0
FBC Stats
Active Sessions : 0
Average FPS : 0
Average Latency : 0
Ecc Mode
Current : N/A
Pending : N/A
ECC Errors
Volatile
SRAM Correctable : N/A
SRAM Uncorrectable : N/A
DRAM Correctable : N/A
DRAM Uncorrectable : N/A
Aggregate
SRAM Correctable : N/A
SRAM Uncorrectable : N/A
DRAM Correctable : N/A
DRAM Uncorrectable : N/A
Retired Pages
Single Bit ECC : N/A
Double Bit ECC : N/A
Pending Page Blacklist : N/A
Remapped Rows : N/A
Temperature
GPU Current Temp : 59 C
GPU Shutdown Temp : 98 C
GPU Slowdown Temp : 95 C
GPU Max Operating Temp : 93 C
GPU Target Temperature : 83 C
Memory Current Temp : N/A
Memory Max Operating Temp : N/A
Power Readings
Power Management : Supported
Power Draw : 26.50 W
Power Limit : 170.00 W
Default Power Limit : 170.00 W
Enforced Power Limit : 170.00 W
Min Power Limit : 100.00 W
Max Power Limit : 180.00 W
Clocks
Graphics : 210 MHz
SM : 210 MHz
Memory : 405 MHz
Video : 555 MHz
Applications Clocks
Graphics : N/A
Memory : N/A
Default Applications Clocks
Graphics : N/A
Memory : N/A
Max Clocks
Graphics : 2100 MHz
SM : 2100 MHz
Memory : 7501 MHz
Video : 1950 MHz
Max Customer Boost Clocks
Graphics : N/A
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Voltage
Graphics : 643.750 mV
Processes
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 2276
Type : G
Name : /usr/lib/xorg/Xorg
Used GPU Memory : 132 MiB
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 2563
Type : G
Name : /usr/bin/gnome-shell
Used GPU Memory : 52 MiB
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 3755
Type : G
Name : /opt/google/chrome/chrome --type=gpu-process --enable-crashpad --crashpad-handler-pid=3705 --enable-crash-reporter=, --change-stack-guard-on-fork=enable --gpu-preferences=WAAAAAAAAAAgAAAIAAAAAAAAAAAAAAAAAABgAAAAAAA4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAABAAAAAAAAAAgAAAAAAAAACAAAAAAAAAAIAAAAAAAAAA== --shared-files --field-trial-handle=0,i,11614745339610379924,13434341812416912508,131072
Used GPU Memory : 134 MiB
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 10459
Type : G
Name : /snap/discord/145/usr/share/discord/Discord --type=gpu-process --field-trial-handle=13422725312599898036,16082118744567239806,131072 --disable-features=CookiesWithoutSameSiteMustBeSecure,HardwareMediaKeyHandling,MediaSessionService,SameSiteByDefaultCookies,SpareRendererForSitePerProcess,WinRetrieveSuggestionsOnlyOnDemand --disable-seccomp-filter-sandbox --no-sandbox --enable-crash-reporter=988bcc73-1b83-4182-8e8f-ab74016726dc,no_channel --global-crash-keys=988bcc73-1b83-4182-8e8f-ab74016726dc,no_channel,_companyName=Discord Inc.,_productName=Discord,_version=0.0.21 --gpu-preferences=UAAAAAAAAAAgAAAQAAAAAAAAAAAAAAAAAABgAAAAAAAwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgAAAAAAAAACAAAAAAAAAAAAAAAAAAAAAIAAAAAAAAAAgAAAAAAAAACAAAAAAAAAA= --shared-files
Used GPU Memory : 23 MiB

Installation Process

I followed the installation CUDA installation instructions closely, and used the deb (local) installation.

The CUDA files can be seen at /usr/local/cuda-11.8 and are sym-linked as expected to /usr/local/cuda-11.8.

Compiling the cuda-samples

I had to export the CUDA_HOME environment variable to correctly compile the samples, but they otherwise work.

Pytorch

torch.__version__ returns 1.12.1+cu116
torch.cuda.is_available() returns False

Interestingly, I have dual-booted with Windows, and my GPU does work with Pytorch within Windows.

Does anyone have any ideas why my 3060 is unavailable as a CUDA device?
Thanks in advance

Do you have CUDA_VISIBLE_DEVICES env var set? Is this a laptop?

Hi,

Nope, CUDA_VISIBLE_DEVICES doesn’t yield anything.

It’s not a laptop, it’s a PC with an i5-12600K and RTX 3060

One observation, the latest publicly available driver at Nvidia, (including beta drivers), I could find for Linux, is 520.56.06.

Probably a red herring, but if all else fails, it may be worth a try rolling back to this.

Hi,

Thanks for the tip. I can try and go to an older driver, but I think this driver was released in October. It’s the the default driver with my install of Linux and it’s what I get if I do sudo apt install nvidia-driver-520 nvidia-dkms-520

The NVIDIA driver should be installed from a NVIDIA source. Please install using a NVIDIA provided CUDA toolkit installer. If you’re using the CUDA 11.8 toolkit installer, use the driver bundled with that installer.