TAO Data services jupyter notebook permission error

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Trying to run the TAO Data services jupyter notebook
when I run this cell

!tao dataset annotations convert -e $SPECS_DIR/convert.yaml

I get this permmision error:

‘Error while fetching server API version: {0}’.format(e)
docker.errors.DockerException: Error while fetching server API version: (‘Connection aborted.’, PermissionError(13, ‘Permission denied’))

I am logged onto docker login nvcr.io

joev@joev-OMEN-by-HP-Obelisk-Desktop-875-1xxx:~$ docker login nvcr.io
Authenticating with existing credentials…
WARNING! Your password will be stored unencrypted in /home/joev/.docker/config.json.
Configure a credential helper to remove this warning. See
docker login | Docker Docs
Login Succeeded

Any ideas?

2

Could you share the full log?

3

Here you go.

2023-08-12 12:01:56,804 [TAO Toolkit] [INFO] root 160: Registry: [‘nvcr.io’]
Traceback (most recent call last):
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 721, in urlopen
chunked=chunked,
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 415, in _make_request
conn.request(method, url, **httplib_request_kw)
File “/snap/jupyter/6/lib/python3.7/http/client.py”, line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File “/snap/jupyter/6/lib/python3.7/http/client.py”, line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File “/snap/jupyter/6/lib/python3.7/http/client.py”, line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File “/snap/jupyter/6/lib/python3.7/http/client.py”, line 1016, in _send_output
self.send(msg)
File “/snap/jupyter/6/lib/python3.7/http/client.py”, line 956, in send
self.connect()
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/docker/transport/unixconn.py”, line 43, in connect
sock.connect(self.unix_socket)
PermissionError: [Errno 13] Permission denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/requests/adapters.py”, line 497, in send
chunked=chunked,
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 799, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/urllib3/util/retry.py”, line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/urllib3/packages/six.py”, line 769, in reraise
raise value.with_traceback(tb)
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 721, in urlopen
chunked=chunked,
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 415, in _make_request
conn.request(method, url, **httplib_request_kw)
File “/snap/jupyter/6/lib/python3.7/http/client.py”, line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)
File “/snap/jupyter/6/lib/python3.7/http/client.py”, line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File “/snap/jupyter/6/lib/python3.7/http/client.py”, line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File “/snap/jupyter/6/lib/python3.7/http/client.py”, line 1016, in _send_output
self.send(msg)
File “/snap/jupyter/6/lib/python3.7/http/client.py”, line 956, in send
self.connect()
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/docker/transport/unixconn.py”, line 43, in connect
sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: (‘Connection aborted.’, PermissionError(13, ‘Permission denied’))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/docker/api/client.py”, line 205, in _retrieve_server_version
return self.version(api_version=False)[“ApiVersion”]
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/docker/api/daemon.py”, line 181, in version
return self._result(self._get(url), json=True)
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/docker/utils/decorators.py”, line 46, in inner
return f(self, *args, **kwargs)
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/docker/api/client.py”, line 228, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/requests/sessions.py”, line 602, in get
return self.request(“GET”, url, **kwargs)
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/requests/sessions.py”, line 589, in request
resp = self.send(prep, **send_kwargs)
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/requests/sessions.py”, line 703, in send
r = adapter.send(request, **kwargs)
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/requests/adapters.py”, line 501, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: (‘Connection aborted.’, PermissionError(13, ‘Permission denied’))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/joev/snap/jupyter/common/bin/tao”, line 10, in
sys.exit(main())
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/nvidia_tao_cli/entrypoint/tao_launcher.py”, line 137, in main
args[2:]
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/nvidia_tao_cli/components/instance_handler/local_instance.py”, line 357, in launch_command
docker_handler = self.handler_map[
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/nvidia_tao_cli/components/instance_handler/local_instance.py”, line 208, in handler_map
docker_mount_file=os.getenv(“LAUNCHER_MOUNTS”, DOCKER_MOUNT_FILE)
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/nvidia_tao_cli/components/docker_handler/docker_handler.py”, line 74, in init
self._docker_client = docker.from_env()
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/docker/client.py”, line 85, in from_env
timeout=timeout, version=version, **kwargs_from_env(**kwargs)
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/docker/client.py”, line 40, in init
self.api = APIClient(*args, **kwargs)
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/docker/api/client.py”, line 188, in init
self._version = self._retrieve_server_version()
File “/home/joev/snap/jupyter/common/lib/python3.7/site-packages/docker/api/client.py”, line 213, in _retrieve_server_version
‘Error while fetching server API version: {0}’.format(e)
docker.errors.DockerException: Error while fetching server API version: (‘Connection aborted.’, PermissionError(13, ‘Permission denied’))

4

I am afraid you were triggering tao docker based on another docker.
In this case, please
add -v /var/run/docker.sock:/var/run/docker.sock

Reference: Run TLT inside docker - #8 by Morganh

5

I read the associated documents you recommended but I do not know where to add:

-v /var/run/docker.sock:/var/run/docker.sock

To run the jupyter notebook i just run this:

docker login nvcr.io

then this:

jupyter notebook --ip 0.0.0.0 --port 8888 --allow-root

and it usually runs fine

6

It is added in the command line of docker run.

Could you please run below to check if it works?
! tao info --verbose

7

Here is result of “!tao info --verbose”

Configuration of the TAO Toolkit Instance

task_group:
model:
dockers:
nvidia/tao/tao-toolkit:
5.0.0-tf2.11.0:
docker_registry: nvcr.io
tasks:
1. classification_tf2
2. efficientdet_tf2
5.0.0-tf1.15.5:
docker_registry: nvcr.io
tasks:
1. bpnet
2. classification_tf1
3. converter
4. detectnet_v2
5. dssd
6. efficientdet_tf1
7. faster_rcnn
8. fpenet
9. lprnet
10. mask_rcnn
11. multitask_classification
12. retinanet
13. ssd
14. unet
15. yolo_v3
16. yolo_v4
17. yolo_v4_tiny
5.0.0-pyt:
docker_registry: nvcr.io
tasks:
1. action_recognition
2. classification_pyt
3. deformable_detr
4. dino
5. mal
6. ml_recog
7. ocdnet
8. ocrnet
9. optical_inspection
10. pointpillars
11. pose_classification
12. re_identification
13. segformer
dataset:
dockers:
nvidia/tao/tao-toolkit:
5.0.0-data-services:
docker_registry: nvcr.io
tasks:
1. augmentation
2. auto_label
3. annotations
4. analytics
deploy:
dockers:
nvidia/tao/tao-toolkit:
5.0.0-deploy:
docker_registry: nvcr.io
tasks:
1. classification_pyt
2. classification_tf1
3. classification_tf2
4. deformable_detr
5. detectnet_v2
6. dino
7. dssd
8. efficientdet_tf1
9. efficientdet_tf2
10. faster_rcnn
11. lprnet
12. mask_rcnn
13. ml_recog
14. multitask_classification
15. ocdnet
16. ocrnet
17. optical_inspection
18. retinanet
19. segformer
20. ssd
21. trtexec
22. unet
23. yolo_v3
24. yolo_v4
25. yolo_v4_tiny
format_version: 3.0
toolkit_version: 5.0.0
published_date: 07/14/2023

8

I dont want to seem dificult but I dont run the “docker run” command

I run :

docker login nvcr.io

then after succesful login then I run this:

jupyter notebook --ip 0.0.0.0 --port 8888 --allow-root

then I find the jupyter notebook I want to run and click on it to start.
I have always ran my jupyter notebooks that way

9

I cannot reproduce the error with your commands.
Could you check nvidia-smi? Is it 525 driver?
If not , please install 525 and retry.

Uninstall:
sudo apt purge nvidia-driver-xxx
sudo apt autoremove
sudo apt autoclean

Install:
sudo apt install nvidia-driver-525

10

The issue seems to be this version of TAO works differently than the last versions. The way I have been running my jupyter notebooks dos not work with this version. Going to have to actually do some reading this time around.

11

It says that TAO can be installed with python wheels.
I looked all over the Docs and cant find any instruction to do it.
All I can find is this:

You can also run TAO directly on bare-metal without docker or K8s by using the python wheels. As a sample tutorial, TAO Toolkit provides sample tutorials to deploy directly on Google Colab without having to configure your infrastructure. The full instructions on how to work with google colab are provided here.

The link sends you to running TAO on google colab.

12

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

That is the way running from python wheels. You can ignore it since it is a different case from yours.

I still cannot reproduce the error as yours. Can you double check?
Below is my step.

  (base) morganh@xxx:~/demo_3.0$ docker login nvcr.io
  
  (base) morganh@xxx:~/demo_3.0$ . ~/venv_3.0/bin/activate
  
  (venv_3.0) (base) morganh@xxx:~/demo_3.0$ wget --content-disposition https://api.ngc.nvidia.com/v2/resources/nvidia/tao/tao-getting-started/versions/5.0.0/zip -O getting_started_v5.0.0.zip 
  
  (venv_3.0) (base) morganh@xxx:~/demo_3.0$ unzip -u getting_started_v5.0.0.zip -d ./getting_started_v5.0.0 && rm -rf getting_started_v5.0.0.zip && cd ./getting_started_v5.0.0
  
  (venv_3.0) (base) morganh@xxx:~/demo_3.0/getting_started_v5.0.0$ jupyter notebook --ip 0.0.0.0 --port 8888 --allow-root
13

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.