Docker: Error response from daemon: OCI runtime create failed

Hi,

I’m getting the below error when running the CUDA container using below command:
docker run --gpus all nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -gpu -benchmark

docker: Error response from daemon: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: driver error: failed to process request: unknown.
ERRO[0000] error waiting for container: context canceled

As advised CUDA on WSL :: CUDA Toolkit Documentation, I’m raising the request to guide me how can this issue be resolved. Please let me know what information do you need to help me resolving this issue.

Providing some more info:

$ inxi -G
Graphics: Card: Failed to Detect Video Card!
Display Server: N/A driver: N/A tty size: 237x63 Advanced Data: N/A out of X

Regards

こんばんは。

私も同様のエラーが発生しました。
どうやらWindowsInsiderBuild(osbuild21354)固有の不具合のようです。

以下の記事に該当していませんか?

It’s a known issue on WIP 21354 that is semi-fixed on 21359.

Even then, you have to stop WSL from doing a driver version check by using a NVIDIA_DISABLE_REQUIRE=1 environment variable, as per ch14ota’s link below. E.g.:

docker run --gpus all --env NVIDIA_DISABLE_REQUIRE=1 nvcr.io/nvidia/k8s/cuda-sample:nbody nbody -gpu -benchmark

Good luck!

2 Likes

I also cannot get wsl to work with my GPU. Very annoying…

Build: 21359.1

(running export NVIDIA_DISABLE_REQUIRE=1)

root@Ubuntu20:/# cat /etc/os-release | grep VERSION
VERSION="20.04 LTS (Focal Fossa)"
VERSION_ID="20.04"
VERSION_CODENAME=focal
root@DESKTOP-F5AMOPO:/# cat /proc/version
Linux version 5.4.91-microsoft-standard-WSL2 (oe-user@oe-host) (gcc version 9.3.0 (GCC)) #1 SMP Mon Jan 25 18:39:31 UTC 2021

root@Ubuntu20:/# inxi -G
Graphics:  Device-1: Microsoft driver: dxgkrnl v: N/A
           Device-2: Microsoft driver: dxgkrnl v: N/A
           Display: server: No display server data found. Headless machine? tty: 120x30
           Message: Advanced graphics data unavailable in console for root.

root@DESKTOP-F5AMOPO:/# nvidia-smi.exe # <--- native nvidia-smi
Thu Apr 15 23:29:42 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.14       Driver Version: 470.14       CUDA Version: 11.3     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ... WDDM  | 00000000:2D:00.0  On |                  N/A |
|  0%   45C    P8    18W / 350W |    580MiB / 24576MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ... WDDM  | 00000000:2E:00.0  On |                  N/A |
|  0%   43C    P8    10W / 200W |    280MiB /  8192MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1708    C+G   Insufficient Permissions        N/A      |
|    0   N/A  N/A      7820    C+G   C:\Windows\explorer.exe         N/A      |
|    0   N/A  N/A      8836    C+G   ...5n1h2txyewy\SearchApp.exe    N/A      |
|    0   N/A  N/A      9504    C+G   ...IA GeForce Experience.exe    N/A      |
|    0   N/A  N/A     10716    C+G   ...lPanel\SystemSettings.exe    N/A      |
|    0   N/A  N/A     11964    C+G   ...2txyewy\TextInputHost.exe    N/A      |
|    0   N/A  N/A     12836    C+G   ...5n1h2txyewy\SearchApp.exe    N/A      |
|    0   N/A  N/A     13776    C+G   ...y\ShellExperienceHost.exe    N/A      |
|    0   N/A  N/A     15100    C+G   ...8bbwe\WindowsTerminal.exe    N/A      |
|    1   N/A  N/A      1708    C+G   Insufficient Permissions        N/A      |
+-----------------------------------------------------------------------------+

root@Ubuntu20:/# nvidia-smi # <--- the wsl one
Failed to initialize NVML: GPU access blocked by the operating system
Failed to properly shut down NVML: GPU access blocked by the operating system

root@Ubuntu20:/# /usr/local/cuda/samples/1_Utilities/deviceQuery/deviceQuery
/usr/local/cuda/samples/1_Utilities/deviceQuery/deviceQuery Starting...
 CUDA Device Query (Runtime API) version (CUDART static linking)
cudaGetDeviceCount returned 35
-> CUDA driver version is insufficient for CUDA runtime version
Result = FAIL

root@Ubuntu20:/# ls /usr/lib/wsl/lib/ -la
total 54408
dr-xr-xr-x 1 root root     4096 Apr 15 22:18 .
drwxr-xr-x 4 root root     4096 Apr 14 23:47 ..
-r-xr-xr-x 1 root root   137264 Mar 18 03:41 libcuda.so
-r-xr-xr-x 1 root root   137264 Mar 18 03:41 libcuda.so.1
-r-xr-xr-x 1 root root   137264 Mar 18 03:41 libcuda.so.1.1
-r-xr-xr-x 1 root root  6053064 Mar 18 03:41 libnvcuvid.so.1
-r-xr-xr-x 1 root root   424440 Mar 18 03:41 libnvidia-encode.so.1
-r-xr-xr-x 1 root root   197104 Mar 18 03:41 libnvidia-ml.so.1
-r-xr-xr-x 1 root root   354808 Mar 18 03:41 libnvidia-opticalflow.so.1
-r-xr-xr-x 1 root root 47582360 Mar 18 03:41 libnvwgf2umx.so
-r-xr-xr-x 1 root root   670104 Mar 18 03:41 nvidia-smi

Unfortunately this is still not fixed.
I have Windows version Build 21382.1000

None of the fixes described above work!

:-( :-( :-(

4 Likes