An error occurred while preparing the data set using TAO

[Diluk]

Hi,guys:

I got an error when using TAO to prepare the data set,as shown below:

Please provide the following information when requesting support.

• Hardware (T4/V100/Xavier/Nano/etc)
V100
• Network Type (Detectnet_v2/Faster_rcnn/Yolo_v4/LPRnet/Mask_rcnn/Classification/etc)
Yolo_v3
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here)
tao-toolkit-tf:v3.21.08-py3
• Training spec file(If have, please share here)

• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)

Any idea?Thanks.

[Morganh]

There should be invalid path for the file or directory.
Please note that the path after tao yolo_v3 dataset_convert should be the path inside the docker.

Or you can login tao docker and debug.
$ tao yolo_v3 run /bin/bash
# yolo_v3 dataset_convert xxx

[Diluk]

Hi, Morganh:
The path I set is as follows, I am very confused, which path is invalid

[Morganh]

Can you share your ~/.tao_mounts.json ?

[Diluk]

of course

[Morganh]

Can you check if below path is available?
$ tao yolo_v3 run ls /workspace/tao-experiments

The output should be the same as your local source
$ ls /workspace/LOCAL_PROJECT_DIR

[Diluk]

I got the following error:

[Diluk]

TAO is more difficult to use than TLT, and complicated mapping relationships and online training bring unnecessary trouble to users.

[Morganh]

TAO is just renaming of TLT. There is no change in ~/.tao_mounts.json.

Can you paste the full log? I cannot see which file is not found.

[Diluk]

full log is as follows:

2021-09-29 09:34:00,045 [INFO] root: Registry: [‘nvcr.io’]
Traceback (most recent call last):
File “/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py”, line 677, in urlopen
chunked=chunked,
File “/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py”, line 392, in _make_request
conn.request(method, url, **httplib_request_kw)
File “/usr/lib/python3.6/http/client.py”, line 1281, in request
self._send_request(method, url, body, headers, encode_chunked)
File “/usr/lib/python3.6/http/client.py”, line 1327, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File “/usr/lib/python3.6/http/client.py”, line 1276, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File “/usr/lib/python3.6/http/client.py”, line 1042, in _send_output
self.send(msg)
File “/usr/lib/python3.6/http/client.py”, line 980, in send
self.connect()
File “/usr/local/lib/python3.6/dist-packages/docker/transport/unixconn.py”, line 43, in connect
sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/lib/python3.6/dist-packages/requests/adapters.py”, line 449, in send
timeout=timeout
File “/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py”, line 727, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File “/usr/local/lib/python3.6/dist-packages/urllib3/util/retry.py”, line 403, in increment
raise six.reraise(type(error), error, _stacktrace)
File “/usr/local/lib/python3.6/dist-packages/urllib3/packages/six.py”, line 734, in reraise
raise value.with_traceback(tb)
File “/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py”, line 677, in urlopen
chunked=chunked,
File “/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py”, line 392, in _make_request
conn.request(method, url, **httplib_request_kw)
File “/usr/lib/python3.6/http/client.py”, line 1281, in request
self._send_request(method, url, body, headers, encode_chunked)
File “/usr/lib/python3.6/http/client.py”, line 1327, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File “/usr/lib/python3.6/http/client.py”, line 1276, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File “/usr/lib/python3.6/http/client.py”, line 1042, in _send_output
self.send(msg)
File “/usr/lib/python3.6/http/client.py”, line 980, in send
self.connect()
File “/usr/local/lib/python3.6/dist-packages/docker/transport/unixconn.py”, line 43, in connect
sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: (‘Connection aborted.’, FileNotFoundError(2, ‘No such file or directory’))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/lib/python3.6/dist-packages/docker/api/client.py”, line 205, in _retrieve_server_version
return self.version(api_version=False)[“ApiVersion”]
File “/usr/local/lib/python3.6/dist-packages/docker/api/daemon.py”, line 181, in version
return self._result(self._get(url), json=True)
File “/usr/local/lib/python3.6/dist-packages/docker/utils/decorators.py”, line 46, in inner
return f(self, *args, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/docker/api/client.py”, line 228, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File “/usr/local/lib/python3.6/dist-packages/requests/sessions.py”, line 543, in get
return self.request(‘GET’, url, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/requests/sessions.py”, line 530, in request
resp = self.send(prep, **send_kwargs)
File “/usr/local/lib/python3.6/dist-packages/requests/sessions.py”, line 643, in send
r = adapter.send(request, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/requests/adapters.py”, line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: (‘Connection aborted.’, FileNotFoundError(2, ‘No such file or directory’))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/bin/tao”, line 8, in
sys.exit(main())
File “/usr/local/lib/python3.6/dist-packages/tlt/entrypoint/entrypoint.py”, line 115, in main
args[1:]
File “/usr/local/lib/python3.6/dist-packages/tlt/components/instance_handler/local_instance.py”, line 259, in launch_command
docker_handler = self.handler_map[
File “/usr/local/lib/python3.6/dist-packages/tlt/components/instance_handler/local_instance.py”, line 114, in handler_map
docker_mount_file=os.getenv(“LAUNCHER_MOUNTS”, DOCKER_MOUNT_FILE)
File “/usr/local/lib/python3.6/dist-packages/tlt/components/docker_handler/docker_handler.py”, line 62, in init
self._docker_client = docker.from_env()
File “/usr/local/lib/python3.6/dist-packages/docker/client.py”, line 85, in from_env
timeout=timeout, version=version, **kwargs_from_env(**kwargs)
File “/usr/local/lib/python3.6/dist-packages/docker/client.py”, line 40, in init
self.api = APIClient(*args, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/docker/api/client.py”, line 188, in init
self._version = self._retrieve_server_version()
File “/usr/local/lib/python3.6/dist-packages/docker/api/client.py”, line 213, in _retrieve_server_version
‘Error while fetching server API version: {0}’.format(e)
docker.errors.DockerException: Error while fetching server API version: (‘Connection aborted.’, FileNotFoundError(2, ‘No such file or directory’))

[Morganh]

Are you triggering tao docker based on one docker?
In this case, please
add -v /var/run/docker.sock:/var/run/docker.sock

Refer to similar topics

Tlt augment not working

I am trying to run the tlt augment command inside the tlt container, but got the following error: Command used to run the container is : sudo docker run --gpus all -it -v /workspace/tlt-experiments/:/workspace/tlt-experiments -p 8888:8888 nvcr.io/nvidia/tlt-streamanalytics:v3.0-py3 /bin/bash root@1833e5376a0d:/workspace/tlt-experiments# tlt augment -d /workspace/tlt-experiments/data/training/ -a /workspace/tlt-experiments/specs/default_spec.txt -o /workspace/tlt-experiments/augment -v 2021-…

Is there some spacial things about bpnet? A question about "tlt bpnet dataset_convert " for bpnet

How about running $ sudo docker login nvcr.io in host pc ? If not able to run with sudo, please see NVIDIA TAO Documentation Once you have installed docker-ce, follow the post-installation steps to ensure that the docker can be run without sudo .

[Diluk]

It works, thank you again for your help.

[Diluk]

Hi,Morganh:
The following error occurred when using tao-converter to convert yolov3 weights:

I understand that the -d parameter is no longer necessary, any ideas?

[Morganh]

Could you please create a new topic since it is different from previous question?