Failure to Install CUDA 11.6 on WSL2.0 Ubuntu 20.04 | cudaGetDeviceCount returned 103

I can’t really find anything about error 103. I used the .run installer for the cuda-toolkit-11. Currently nvidia-docker does work properly, and I’m able to run the sample containers. However, locally, I’m not able to use the CUDA Toolkit. Here is some information.

./deviceQuery Starting…
CUDA Device Query (Runtime API) version (CUDART static linking)
cudaGetDeviceCount returned 103
→ integrity checks failed
Result = FAIL

$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.00       Driver Version: 510.06       CUDA Version: 11.6     | 
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro RTX 5000     On   | 00000000:01:00.0 Off |                  N/A |
| N/A    0C    P3    32W /  N/A |    214MiB / 16384MiB |     N/A      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                                  |                                                                                                               |  
GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_Mar__8_18:18:20_PST_2022
Cuda compilation tools, release 11.6, V11.6.124
Build cuda_11.6.r11.6/compiler.31057947_0

$ uname -r
5.10.102.1-microsoft-standard-WSL2

$ cat /proc/version
Linux version 5.10.102.1-microsoft-standard-WSL2 (oe-user@oe-host) (x86_64-msft-linux-gcc (GCC) 9.3.0, GNU ld (GNU Binutils) 2.34.0.20200220) #1 SMP Wed Mar 2 00:30:59 UTC 2022

$ ls -l /mnt/c/Windows/System32/lxss/lib/ total 66584
dr-xr-xr-x 1 pipemon pipemon 512 Dec 23 12:02 ./
dr-xr-xr-x 1 pipemon pipemon 512 Mar 15 07:00 …/
-r-xr-xr-x 1 pipemon pipemon 141464 Aug 24 2021 libcuda.so*
-r-xr-xr-x 1 pipemon pipemon 141464 Aug 24 2021 libcuda.so.1*
-r-xr-xr-x 1 pipemon pipemon 141464 Aug 24 2021 libcuda.so.1.1*
-r-xr-xr-x 2 pipemon pipemon 828840 Sep 16 2021 libd3d12.so*
-r-xr-xr-x 2 pipemon pipemon 4834848 Sep 16 2021 libd3d12core.so*
-r-xr-xr-x 2 pipemon pipemon 878768 Sep 16 2021 libdxcore.so*
-r-xr-xr-x 1 pipemon pipemon 5962912 Aug 24 2021 libnvcuvid.so.1*
-r-xr-xr-x 1 pipemon pipemon 424400 Aug 24 2021 libnvidia-encode.so.1*
-r-xr-xr-x 1 pipemon pipemon 197752 Aug 24 2021 libnvidia-ml.so.1*
-r-xr-xr-x 1 pipemon pipemon 354768 Aug 24 2021 libnvidia-opticalflow.so.1*
-r-xr-xr-x 1 pipemon pipemon 53568104 Aug 24 2021 libnvwgf2umx.so*
-r-xr-xr-x 1 pipemon pipemon 682552 Aug 24 2021 nvidia-smi*

$ echo $LD_LIBRARY_PATH
/usr/local/cuda-11.6/lib64:/usr/local/lib:/usr/local/cuda-11.6/lib64:/usr/local/lib:

$ echo $PATH
contains… /usr/local/cuda-11.6/bin: