TAO 5.3 docker error - Not supported URL scheme http+docker (requests 2.31.0)

Please provide the following information when requesting support.

• Hardware - L4
• Network Type - TAO classification_pyt / fan_small_12_p4_hybrid

I’m working through the classification_pyt notebook and am stuck with an error.

The notebook cell is:

print("Train Classification Model")
!tao model classification_pyt train \
                  -e $SPECS_DIR/train_cats_dogs.yaml \
                  -r $RESULTS_DIR/classification_experiment \
                  -g $NUM_GPUS \
                  model.init_cfg.checkpoint=/workspace/tao-experiments/pretrained_fan_hybrid_small/pretrained_fan_classification_imagenet_vfan_hybrid_small/fan_hybrid_small.pth \
                  train.train_config.runner.max_epochs=$EPOCHS

It’s failing with:

2024-07-09 21:29:23,283 [TAO Toolkit] [INFO] root 160: Registry: ['nvcr.io']
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 633, in send
    conn = self.get_connection_with_tls_context(
  File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 489, in get_connection_with_tls_context
    conn = self.poolmanager.connection_from_host(
  File "/usr/local/lib/python3.10/dist-packages/urllib3/poolmanager.py", line 246, in connection_from_host
    return self.connection_from_context(request_context)
  File "/usr/local/lib/python3.10/dist-packages/urllib3/poolmanager.py", line 258, in connection_from_context
    raise URLSchemeUnknown(scheme)
urllib3.exceptions.URLSchemeUnknown: Not supported URL scheme http+docker

This is reproducible in the shell with:

tao model classification_pyt run /bin/bash

I’m searched the forums and would add the following:

  • I’m running requests-2.31.0 (see here)
pip show requests
Name: requests
Version: 2.31.0
Summary: Python HTTP for Humans.
Home-page: https://requests.readthedocs.io
Author: Kenneth Reitz
Author-email: me@kennethreitz.org
License: Apache 2.0
Location: /home/****/miniconda3/envs/tao-launcher/lib/python3.7/site-packages
Requires: certifi, charset-normalizer, idna, urllib3
Required-by: wandb
  • Docker is being run as a user, not root

Output of tao info:

$ tao info
Configuration of the TAO Toolkit Instance
task_group: ['model', 'dataset', 'deploy']
format_version: 3.0
toolkit_version: 5.3.0
published_date: 03/14/2024

Full exception output:

2024-07-09 21:47:44,850 [TAO Toolkit] [INFO] root 160: Registry: ['nvcr.io']
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 633, in send
    conn = self.get_connection_with_tls_context(
  File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 489, in get_connection_with_tls_context
    conn = self.poolmanager.connection_from_host(
  File "/usr/local/lib/python3.10/dist-packages/urllib3/poolmanager.py", line 246, in connection_from_host
    return self.connection_from_context(request_context)
  File "/usr/local/lib/python3.10/dist-packages/urllib3/poolmanager.py", line 258, in connection_from_context
    raise URLSchemeUnknown(scheme)
urllib3.exceptions.URLSchemeUnknown: Not supported URL scheme http+docker

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/docker/api/client.py", line 205, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/usr/local/lib/python3.10/dist-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/usr/local/lib/python3.10/dist-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/docker/api/client.py", line 228, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 637, in send
    raise InvalidURL(e, request=request)
requests.exceptions.InvalidURL: Not supported URL scheme http+docker

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/tao", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/nvidia_tao_cli/entrypoint/tao_launcher.py", line 134, in main
    instance.launch_command(
  File "/usr/local/lib/python3.10/dist-packages/nvidia_tao_cli/components/instance_handler/local_instance.py", line 357, in launch_command
    docker_handler = self.handler_map[
  File "/usr/local/lib/python3.10/dist-packages/nvidia_tao_cli/components/instance_handler/local_instance.py", line 203, in handler_map
    handler_map[handler_key] = DockerHandler(
  File "/usr/local/lib/python3.10/dist-packages/nvidia_tao_cli/components/docker_handler/docker_handler.py", line 92, in __init__
    self._docker_client = docker.from_env()
  File "/usr/local/lib/python3.10/dist-packages/docker/client.py", line 84, in from_env
    return cls(
  File "/usr/local/lib/python3.10/dist-packages/docker/client.py", line 40, in __init__
    self.api = APIClient(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/docker/api/client.py", line 188, in __init__
    self._version = self._retrieve_server_version()
  File "/usr/local/lib/python3.10/dist-packages/docker/api/client.py", line 212, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: Not supported URL scheme http+docker

Did you login to the NGC docker registry ( nvcr.io ) ?
$ docker login nvcr.io

  1. Username: “$oauthtoken”
  2. Password: “YOUR_NGC_API_KEY”

Yes.

docker login nvcr.io -u "\$oauthtoken" -p $NGC_CLI_API_KEY
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /home/_____/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credential-stores

Login Succeeded
(tao-launcher) c___@s________:~/tao-getting-started_v5.3.0/notebooks/tao_launcher_starter_kit/$ tao model classification_pyt run /bin/bash
2024-07-09 21:47:44,850 [TAO Toolkit] [INFO] root 160: Registry: ['nvcr.io']
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 633, in send
    conn = self.get_connection_with_tls_context(
  File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 489, in get_connection_with_tls_context
    conn = self.poolmanager.connection_from_host(
  File "/usr/local/lib/python3.10/dist-packages/urllib3/poolmanager.py", line 246, in connection_from_host
    return self.connection_from_context(request_context)
  File "/usr/local/lib/python3.10/dist-packages/urllib3/poolmanager.py", line 258, in connection_from_context
    raise URLSchemeUnknown(scheme)
urllib3.exceptions.URLSchemeUnknown: Not supported URL scheme http+docker

I started with a fresh vm and ran through the install steps again. In short, it works. I think an issue may be that I skipped this step Installing the NVIDIA Container Toolkit — NVIDIA Container Toolkit 1.15.0 documentation

 tao model classification_pyt run /bin/bash
~/.tao_mounts.json wasn't found. Falling back to obtain mount points and docker configs from ~/.tao_mounts.json.
Please note that this will be deprecated going forward.
2024-07-10 21:21:11,988 [TAO Toolkit] [INFO] root 160: Registry: ['nvcr.io']
2024-07-10 21:21:12,062 [TAO Toolkit] [INFO] nvidia_tao_cli.components.instance_handler.local_instance 361: Running command in container: nvcr.io/nvidia/tao/tao-toolkit:5.3.0-pyt
2024-07-10 21:21:12,067 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 323: The required docker doesn't exist locally/the manifest has changed. Pulling a new docker.
2024-07-10 21:21:12,068 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 174: Pulling the required container. This may take several minutes if you're doing this for the first time. Please wait here.
...
Pulling from repository: nvcr.io/nvidia/tao/tao-toolkit
[Download 4f4fb700ef54] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--
[Download 5e8117c0bd28] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━  84% 0:00:01
[Download 9075b8c4201a] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺  98% 0:00:01