WSL2 CUDA Driver 465.42 not working with Nvidia's CUDA 11.1.1 Docker Containers

Hello all!

The title explains most of the issue, but to give more context, my windows insider build is Build 21301.rs_prerelease.210123-1645

My nvidia drivers are updated to the latest as well on driver version 465.42:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.27.04    Driver Version: 465.42       CUDA Version: 11.3     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 3090    Off  | 00000000:0A:00.0  On |                  N/A |
| 57%   58C    P0   114W / 350W |   3878MiB / 24576MiB |    ERR!      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

I also have the latest Docker Desktop with WSL2 Integration for the GPU!

When I launch this, it works:
docker run --rm --gpus all nvidia/cuda:11.0-cudnn8-devel-ubuntu18.04

BUT, when I launch this:
docker run --rm --gpus all nvidia/cuda:11.1.1-cudnn8-devel-ubuntu18.04

I get this:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: requirement error: unsatisfied condition: cuda>=11.1, please update your driver to a newer version, or use an earlier cuda container: unknown.

I made sure my nvcc --version also prints this:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Mon_Nov_30_19:08:53_PST_2020
Cuda compilation tools, release 11.2, V11.2.67
Build cuda_11.2.r11.2/compiler.29373293_0

Has anyone also been having the same issue? Here’s the reference link to the containers I’m using for Docker:
nvidia/cuda Tags - Docker Hub

4 Likes

Try adding --env NVIDIA_DISABLE_REQUIRE=1 to your docker run command, basically nvidia-container-cli reports incorrect CUDA toolkit version (11.0 instead of 11.x). I had the same issue recently running with CUDA 11.2.

3 Likes

This is a Bug and they are working on this issue.

https://github.com/NVIDIA/nvidia-docker/issues/1458