NVIDIA driver is not available on latest docker

[youkis]

I couldn’t use nvidia GPUs on docker build command. This issue only occurs on docker==23.0.1 but not on docker==20.10.10. I’d like to use latest docker so any idea or advice would be applicated.

$ docker --version
Docker version 23.0.1, build a5ee5b1
$ cat /etc/docker/daemon.json
{
    "default-runtime": "nvidia",
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}
$ docker run --rm --entrypoint "" nvcr.io/nvidia/pytorch:22.08-py3 nvidia-smi --query-gpu=name --format=csv
name
NVIDIA GeForce RTX 2080 Ti

Dockerfile

FROM nvcr.io/nvidia/pytorch:22.08-py3
RUN nvcc -V
RUN python -c "import torch; print(torch.cuda.is_available())"
RUN nvidia-smi

$ docker build -t test .
#4 [1/4] FROM nvcr.io/nvidia/pytorch:22.08-py3
#4 CACHED

#5 [2/4] RUN nvcc -V
#5 0.316 nvcc: NVIDIA (R) Cuda compiler driver
#5 0.316 Copyright (c) 2005-2022 NVIDIA Corporation
#5 0.316 Built on Wed_Jun__8_16:49:14_PDT_2022
#5 0.316 Cuda compilation tools, release 11.7, V11.7.99
#5 0.316 Build cuda_11.7.r11.7/compiler.31442593_0
#5 DONE 0.3s

#6 [3/4] RUN python -c "import torch; print(torch.cuda.is_available())"
#6 0.993 False
#6 DONE 1.1s

#7 [4/4] RUN nvidia-smi
#7 0.482 /bin/bash: nvidia-smi: command not found
#7 ERROR: process "/bin/sh -c nvidia-smi" did not complete successfully: exit code: 127
------
 > [4/4] RUN nvidia-smi:
#7 0.482 /bin/bash: nvidia-smi: command not found
------
Dockerfile:4
--------------------
   2 |     RUN nvcc -V
   3 |     RUN python -c "import torch; print(torch.cuda.is_available())"
   4 | >>> RUN nvidia-smi
   5 |
--------------------
ERROR: failed to solve: process "/bin/sh -c nvidia-smi" did not complete successfully: exit code: 127

[samjith888]

I’m having the same issue.

Its mentioned in the docs that the specific docker version supports it.

How did you downgraded/installed the docker version to 20.10.10?

[youkis]

@samjith888 If you are using ubuntu22.04, you can try this command.

sudo apt-cache policy docker-ce
sudo apt-get install docker-ce=5:20.10.13~3-0~ubuntu-jammy docker-ce-cli=5:20.10.13~3-0~ubuntu-jammy containerd.io docker-compose-plugin

[samjith888]

Thank you.

But I’m using Ubuntu 18.04

[pfeifer]

@youkis

sudo apt-get install docker-ce=5:20.10.13~3-0~ubuntu-jammy docker-ce-cli=5:20.10.13~3-0~ubuntu-jammy containerd.io docker-compose-plugin

Same issue here, I can’t build my Docker if the build command depends itself on the GPU.

The suggested fix of using a previous version of docker-ce and docker-ce-cli worked – but it’s sad to have to user the older docker, and likely sooner or later it will stop working for different reasons.

Having said that, maybe this is an issue with Docker ?

[youkis]

Same discussion at here.

[youkis]

Having same issue on 24.0.

[youkis]

@samjith888 DOCKER_BUILDKIT=0 docker build solved this issue.