Error setting up TAO Toolkit - 'nvidia-docker not found'

Accelerated Computing Intelligent Video Analytics TAO Toolkit
,
1

Hello,

I am setting up TAO Toolkit as per beginner guide. I am getting below docker errors and package missing errors.

image

image
image1791×712 54.7 KB

For the package related error, I installed the package individually as below, which succeedes. But, the error is shown when doing the complete install.

image

I am sharing the complete log file from console.

INFO: Check requirements
INFO: Checking Python installation
INFO: python3 found.
INFO: Python version: Python 3.12.9
INFO: pip3 found.
INFO: Pip version: pip 25.0 from /home/abhilash/miniconda3/lib/python3.12/site-packages/pip (python 3.12)
INFO: Docker found. Checking additional requirements for docker.
INFO: Checking nvidia-docker2 installation
ERROR: nvidia-docker not found.
INFO: NGC CLI found.
INFO: NGC CLI 3.63.0
INFO: Requirements check satisfied. Installing TAO Toolkit.
By installing the TAO Toolkit CLI, you accept the terms and conditions of this license: https://developer.nvidia.com/tao-toolkit-software-license-agreement
Would you like to continue? (y/n): y
INFO: EULA accepted.
INFO: Installing TAO Toolkit CLI
INFO: jupyter installation was found.
Selected Jupyter core packages...
IPython          : 9.1.0
ipykernel        : not installed
ipywidgets       : 8.1.6
jupyter_client   : not installed
jupyter_core     : 5.7.2
jupyter_server   : 2.15.0
jupyterlab       : not installed
nbclient         : not installed
nbconvert        : 7.16.6
nbformat         : 5.10.4
notebook         : 7.4.0
qtconsole        : not installed
traitlets        : 5.14.3
INFO: TAO Toolkit was found
Traceback (most recent call last):
  File "/home/abhilash/miniconda3/bin/tao", line 5, in <module>
    from nvidia_tao_cli.entrypoint.tao_launcher import main
  File "/home/abhilash/miniconda3/lib/python3.12/site-packages/nvidia_tao_cli/entrypoint/tao_launcher.py", line 23, in <module>
    from nvidia_tao_cli.components.instance_handler.builder import get_launcher
  File "/home/abhilash/miniconda3/lib/python3.12/site-packages/nvidia_tao_cli/components/instance_handler/builder.py", line 24, in <module>
    from nvidia_tao_cli.components.instance_handler.local_instance import LocalInstance
  File "/home/abhilash/miniconda3/lib/python3.12/site-packages/nvidia_tao_cli/components/instance_handler/local_instance.py", line 29, in <module>
    from nvidia_tao_cli.components.docker_handler.docker_handler import (
  File "/home/abhilash/miniconda3/lib/python3.12/site-packages/nvidia_tao_cli/components/docker_handler/docker_handler.py", line 29, in <module>
    import docker
  File "/home/abhilash/miniconda3/lib/python3.12/site-packages/docker/__init__.py", line 2, in <module>
    from .api import APIClient
  File "/home/abhilash/miniconda3/lib/python3.12/site-packages/docker/api/__init__.py", line 2, in <module>
    from .client import APIClient
  File "/home/abhilash/miniconda3/lib/python3.12/site-packages/docker/api/client.py", line 8, in <module>
    import websocket
  File "/home/abhilash/miniconda3/lib/python3.12/site-packages/websocket/__init__.py", line 23, in <module>
    from ._app import WebSocketApp
  File "/home/abhilash/miniconda3/lib/python3.12/site-packages/websocket/_app.py", line 36, in <module>
    from ._core import WebSocket, getdefaulttimeout
  File "/home/abhilash/miniconda3/lib/python3.12/site-packages/websocket/_core.py", line 34, in <module>
    from ._handshake import *
  File "/home/abhilash/miniconda3/lib/python3.12/site-packages/websocket/_handshake.py", line 30, in <module>
    from ._http import *
  File "/home/abhilash/miniconda3/lib/python3.12/site-packages/websocket/_http.py", line 33, in <module>
    from ._url import *
  File "/home/abhilash/miniconda3/lib/python3.12/site-packages/websocket/_url.py", line 27, in <module>
    from six.moves.urllib.parse import urlparse
ModuleNotFoundError: No module named 'six.moves'
INFO:

Any thoughts on how to fix this?

Any help is appreciated. Thanks in advance.

2

Please create conda environment with python 3.10, not python 3.12

Please install nvidia-docker.
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey |
sudo apt-key add -
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list |
sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update
sudo apt-get install -y nvidia-docker2
sudo pkill -SIGHUP dockerd

3

I tried the snippet

curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey |
sudo apt-key add -
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list |
sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update
sudo apt-get install -y nvidia-docker2
sudo pkill -SIGHUP dockerd

But that was giving warning ‘Unsupported distribution!’. Here is the completed log.

Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
OK
# Unsupported distribution!
# Check https://nvidia.github.io/nvidia-docker
Get:1 https://download.docker.com/linux/ubuntu noble InRelease [48.8 kB]
Hit:2 https://nvidia.github.io/libnvidia-container/stable/deb/amd64  InRelease
Get:3 https://dl.google.com/linux/chrome/deb stable InRelease [1825 B]
Hit:4 http://archive.ubuntu.com/ubuntu noble InRelease
Get:5 http://security.ubuntu.com/ubuntu noble-security InRelease [126 kB]
Get:6 http://archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
Get:7 https://dl.google.com/linux/chrome/deb stable/main amd64 Packages [1215 B]
Get:8 http://archive.ubuntu.com/ubuntu noble-backports InRelease [126 kB]
Get:9 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [992 kB]
Get:10 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages [748 kB]
Get:11 http://archive.ubuntu.com/ubuntu noble-updates/main Translation-en [219 kB]
Get:12 http://archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages [1054 kB]
Get:13 http://security.ubuntu.com/ubuntu noble-security/main Translation-en [143 kB]
Get:14 http://archive.ubuntu.com/ubuntu noble-updates/universe Translation-en [266 kB]
Get:15 http://archive.ubuntu.com/ubuntu noble-backports/universe amd64 Packages [27.1 kB]
Get:16 http://archive.ubuntu.com/ubuntu noble-backports/universe Translation-en [16.5 kB]
Get:17 http://security.ubuntu.com/ubuntu noble-security/universe amd64 Packages [830 kB]
Get:18 http://security.ubuntu.com/ubuntu noble-security/universe Translation-en [181 kB]
Get:19 http://security.ubuntu.com/ubuntu noble-security/restricted Translation-en [175 kB]
Fetched 5082 kB in 4s (1187 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  nvidia-docker2
0 upgraded, 1 newly installed, 0 to remove and 20 not upgraded.
Need to get 5128 B of archives.
After this operation, 21.5 kB of additional disk space will be used.
Get:1 https://nvidia.github.io/libnvidia-container/stable/deb/amd64  nvidia-docker2 2.14.0-1 [5128 B]
Fetched 5128 B in 0s (13.5 kB/s)
Selecting previously unselected package nvidia-docker2.
(Reading database ... 79698 files and directories currently installed.)
Preparing to unpack .../nvidia-docker2_2.14.0-1_all.deb ...
Unpacking nvidia-docker2 (2.14.0-1) ...
Setting up nvidia-docker2 (2.14.0-1) ...

I tried few things instead.

I was using Ubuntu 24 LTS. I downgraded that to Ubuntu 22 LTS and started with new setup.

4

In Ubuntu 22 LTS, I still got the docker error. Here is the completed log file.

(launcher) myaccount@mypcMSIStealth15:~/tao_tutorials$ bash setup/quickstart_launcher.sh --install
INFO: Check requirements
INFO: Checking Python installation
INFO: python3 found.
INFO: Python version: Python 3.10.16
INFO: pip3 found.
INFO: Pip version: pip 25.0 from /home/abhilash/miniconda3/envs/launcher/lib/python3.10/site-packages/pip (python 3.10)
INFO: Docker found. Checking additional requirements for docker.
INFO: Checking nvidia-docker2 installation
ERROR: nvidia-docker not found.
--2025-04-16 15:06:15--  https://ngc.nvidia.com/downloads/ngccli_linux.zip
Resolving ngc.nvidia.com (ngc.nvidia.com)... 13.32.251.58, 13.32.251.106, 13.32.251.89, ...
Connecting to ngc.nvidia.com (ngc.nvidia.com)|13.32.251.58|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 65346371 (62M) [application/zip]
Saving to: ‘ngccli_linux.zip’

ngccli_linux.zip            100%[===========================================>]  62.32M  6.57MB/s    in 9.4s

2025-04-16 15:06:24 (6.62 MB/s) - ‘ngccli_linux.zip’ saved [65346371/65346371]

Archive:  ngccli_linux.zip
replace ngc-cli/libgcc_s.so.1? [y]es, [n]o, [A]ll, [N]one, [r]ename: A
  inflating: ngc-cli/libgcc_s.so.1
  inflating: ngc-cli/grpc/_cython/cygrpc.cpython-311-x86_64-linux-gnu.so
  inflating: ngc-cli/grpc/_cython/_credentials/roots.pem
  inflating: ngc-cli/propcache/_helpers_c.cpython-311-x86_64-linux-gnu.so
  inflating: ngc-cli/liblzma.so.5
  # NOTE: REMOVED FEW LOGS INFLATING AND EXTRACTING LOGS TO KEEP WITHIN CHARACTER LIMIT
  inflating: ngc-cli/multidict/_multidict.cpython-311-x86_64-linux-gnu.so
  inflating: ngc-cli/aiohttp/_http_parser.cpython-311-x86_64-linux-gnu.so
  inflating: ngc-cli/aiohttp/_http_writer.cpython-311-x86_64-linux-gnu.so
  inflating: ngc-cli/aiohttp/_websocket/mask.cpython-311-x86_64-linux-gnu.so
  inflating: ngc-cli/aiohttp/_websocket/reader_c.cpython-311-x86_64-linux-gnu.so
 extracting: ngc-cli/prettytable-2.0.0.dist-info/INSTALLER
  inflating: ngc-cli/prettytable-2.0.0.dist-info/RECORD
  inflating: ngc-cli/prettytable-2.0.0.dist-info/METADATA
 extracting: ngc-cli/prettytable-2.0.0.dist-info/top_level.txt
 extracting: ngc-cli/prettytable-2.0.0.dist-info/WHEEL
  inflating: ngc-cli/prettytable-2.0.0.dist-info/COPYING
  inflating: ngc-cli/libncursesw.so.6
  inflating: ngc-cli.md5
setup/quickstart_launcher.sh: line 193: ngc: command not found
INFO:
INFO: Requirements check satisfied. Installing TAO Toolkit.
By installing the TAO Toolkit CLI, you accept the terms and conditions of this license: https://developer.nvidia.com/tao-toolkit-software-license-agreement
Would you like to continue? (y/n): y
INFO: EULA accepted.
INFO: Installing TAO Toolkit CLI
INFO: jupyter installation was found.
Selected Jupyter core packages...
IPython          : 8.35.0
ipykernel        : 6.29.5
ipywidgets       : not installed
jupyter_client   : 8.6.3
jupyter_core     : 5.7.2
jupyter_server   : not installed
jupyterlab       : not installed
nbclient         : not installed
nbconvert        : not installed
nbformat         : not installed
notebook         : not installed
qtconsole        : not installed
traitlets        : 5.14.3
INFO: TAO Toolkit was found
INFO: ~/.tao_mounts.json wasn't found. Falling back to obtain mount points and docker configs from ~/.tao_mounts.json.
Please note that this will be deprecated going forward.
Configuration of the TAO Toolkit Instance

task_group:
    model:
        dockers:
            nvidia/tao/tao-toolkit:
                5.5.0-pyt:
                    docker_registry: nvcr.io
                    tasks:
                        1. action_recognition
                        2. centerpose
                        3. visual_changenet
                        4. deformable_detr
                        5. dino
                        6. grounding_dino
                        7. mask_grounding_dino
                        8. mask2former
                        9. mal
                        10. ml_recog
                        11. ocdnet
                        12. ocrnet
                        13. optical_inspection
                        14. pointpillars
                        15. pose_classification
                        16. re_identification
                        17. classification_pyt
                        18. segformer
                        19. bevfusion
                5.0.0-tf1.15.5:
                    docker_registry: nvcr.io
                    tasks:
                        1. bpnet
                        2. classification_tf1
                        3. converter
                        4. detectnet_v2
                        5. dssd
                        6. efficientdet_tf1
                        7. faster_rcnn
                        8. fpenet
                        9. lprnet
                        10. mask_rcnn
                        11. multitask_classification
                        12. retinanet
                        13. ssd
                        14. unet
                        15. yolo_v3
                        16. yolo_v4
                        17. yolo_v4_tiny
                5.5.0-tf2:
                    docker_registry: nvcr.io
                    tasks:
                        1. classification_tf2
                        2. efficientdet_tf2
    dataset:
        dockers:
            nvidia/tao/tao-toolkit:
                5.5.0-data-services:
                    docker_registry: nvcr.io
                    tasks:
                        1. augmentation
                        2. auto_label
                        3. annotations
                        4. analytics
    deploy:
        dockers:
            nvidia/tao/tao-toolkit:
                5.5.0-deploy:
                    docker_registry: nvcr.io
                    tasks:
                        1. visual_changenet
                        2. centerpose
                        3. classification_pyt
                        4. classification_tf1
                        5. classification_tf2
                        6. deformable_detr
                        7. detectnet_v2
                        8. dino
                        9. dssd
                        10. efficientdet_tf1
                        11. efficientdet_tf2
                        12. faster_rcnn
                        13. grounding_dino
                        14. mask_grounding_dino
                        15. mask2former
                        16. lprnet
                        17. mask_rcnn
                        18. ml_recog
                        19. multitask_classification
                        20. ocdnet
                        21. ocrnet
                        22. optical_inspection
                        23. retinanet
                        24. segformer
                        25. ssd
                        26. trtexec
                        27. unet
                        28. yolo_v3
                        29. yolo_v4
                        30. yolo_v4_tiny
format_version: 3.0
toolkit_version: 5.5.0
published_date: 08/26/2024

I am using Ubuntu WSL in Windows 11. Could that be the reason?

5

I assume the error is with container toolkit installation.

After installing toolkit, I checked sample workload using sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi and it provides expected result.

image
image1091×417 19.3 KB

However, when I try to install the launcher, I am getting the same error:

image
image1079×449 46.1 KB

Any thoughts?

6

What is the error? I did not find any. I think you already install the tao launcher successfully.

7

@Morganh By error, I meant ERROR: nvdia-docker not found.

This appears every time I try to do quickstart_launcher.sh --install. Is that normal?

Are there any commands I could try to make sure docker installation is successful?

Awaiting your reply.

8

Please try again with below.
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey |
sudo apt-key add -
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list |
sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update
sudo apt-get install -y nvidia-docker2
sudo pkill -SIGHUP dockerd