Tao yolov3 training dataset conversion fails KeyError: 'status'

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Please provide the following information when requesting support.

• Hardware (T4)
• Network Type (Yolo_v3)
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)
Running the yolov3 training notebook of tao called “tao_launcher_starter_kit”

When i run below steps to convert the kitti raw dataset to tfrecord i got some errors after some time

# KITTI trainval
!tao model yolo_v3 dataset_convert -d $SPECS_DIR/tfrecords_kitti_trainval.txt \
                     -o $DATA_DOWNLOAD_DIR/yolo_v3/kitti_trainval/kitti_trainval \
                     -r $USER_EXPERIMENT_DIR/

The error is

Traceback (most recent call last):
  File "/home/rishi/miniconda3/envs/launcher/bin/tao", line 8, in <module>
    sys.exit(main())
  File "/home/rishi/miniconda3/envs/launcher/lib/python3.8/site-packages/nvidia_tao_cli/entrypoint/tao_launcher.py", line 134, in main
    instance.launch_command(
  File "/home/rishi/miniconda3/envs/launcher/lib/python3.8/site-packages/nvidia_tao_cli/components/instance_handler/local_instance.py", line 382, in launch_command
    docker_handler.run_container(command)
  File "/home/rishi/miniconda3/envs/launcher/lib/python3.8/site-packages/nvidia_tao_cli/components/docker_handler/docker_handler.py", line 325, in run_container
    self.pull()
  File "/home/rishi/miniconda3/envs/launcher/lib/python3.8/site-packages/nvidia_tao_cli/components/docker_handler/docker_handler.py", line 187, in pull
    docker_pull_progress(line, progress)
  File "/home/rishi/miniconda3/envs/launcher/lib/python3.8/site-packages/nvidia_tao_cli/components/docker_handler/docker_handler.py", line 58, in docker_pull_progress
    if line['status'] == 'Downloading':
KeyError: 'status'

Can you support me on this

3

The docker is downloading.
Please wait for it.
You can open a terminal to check the docker is pulled.
$ docker images

4

The notebook cell is stopped , saying that there is key error ‘status’

5

You can restart to notebook to check.
Or, as mentioned above, you can run in terminal instead of notebook to double check.
$ tao model yolo_v3 run /bin/bash

6

But that is a step to convert the dataset , it ended in the middle and i didn’t see the converted data as well

And i tried these steps multiple time as well , it is failing again and again there

7

From the log, the docker is downloading. You can run $docker images to check if there is a new tao docker. If yes, then you run as below to run dataset-convert.

$ tao model yolo_v3 run /bin/bash
Then inside the docker
#yolo_v3 dataset_convert xxx

8

When i run the command you gave tao model yolo_v3 run /bin/bash

I get below errors

Some downling steps ..

[Download 2c4c36af415f] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
[Download 303a4d4ebf9c] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 100% 0:00:01
[Download bde47f3ca690] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━  84% 0:00:01
[Extract  8e47ddf5daef] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
[Extract  556ab1e8d85a] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
[Extract  b704bd04fbf5] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
[Extract  3ff10bd8cf35] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
[Extract  7be515e856a0] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
[Extract  d115618a5cab] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
[Extract  bd04c8820090] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
[Extract  0384305027fa] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
[Extract  4f4fb700ef54] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
[Extract  df5b36ff9510] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
[Extract  7e69bcc98e9c] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
[Extract  ac9b98675d88] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
[Extract  6e8541639381] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
[Extract  1150aa1cb86b] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Traceback (most recent call last):
  File "/home/rishi/miniconda3/envs/launcher/bin/tao", line 8, in <module>
    sys.exit(main())
  File "/home/rishi/miniconda3/envs/launcher/lib/python3.8/site-packages/nvidia_tao_cli/entrypoint/tao_launcher.py", line 134, in main
    instance.launch_command(
  File "/home/rishi/miniconda3/envs/launcher/lib/python3.8/site-packages/nvidia_tao_cli/components/instance_handler/local_instance.py", line 382, in launch_command
    docker_handler.run_container(command)
  File "/home/rishi/miniconda3/envs/launcher/lib/python3.8/site-packages/nvidia_tao_cli/components/docker_handler/docker_handler.py", line 325, in run_container
    self.pull()
  File "/home/rishi/miniconda3/envs/launcher/lib/python3.8/site-packages/nvidia_tao_cli/components/docker_handler/docker_handler.py", line 187, in pull
    docker_pull_progress(line, progress)
  File "/home/rishi/miniconda3/envs/launcher/lib/python3.8/site-packages/nvidia_tao_cli/components/docker_handler/docker_handler.py", line 58, in docker_pull_progress
    if line['status'] == 'Downloading':
KeyError: 'status'

I can see the image with docker images
nvcr.io/nvidia/tensorrt 23.04-py3 7329292accfb 13 months ago 8.05GB

9

No, this is not tao images.
For yolo_v3 docker, it is nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf1.15.5

10

Ah yes, sorry , so the image is not successfully pulled , that’s where i get the error even when i run the command you gave in the terminal as well

11

Can you advice why i am getting the error when i even run in the terminal
tao model yolo_v3 run /bin/bash

As i have shown above

12

There should be many reasons for pulling error. For example, if there is not disk space.
If docker is pulled, instead of using “tao xxx”, you can use “docker run xxx” to check if it works.

13

docker run --runtime=nvidia -it --rm nvcr.io/nvidia/tao/tao-toolkit:4.0.0-tf1.15.5 /bin/bash

When i execute this commands it works

14

The latest one should be 5.0. nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf1.15.5

15

Thanks for the support , sorry it was due to a insufficient disk space
After removing unwanted images it worked

Thanks again

16

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.