Error in Generating TFrecords for yolov4

Morganh · April 27, 2022, 6:37am

Is it
docker run --runtime=nvidia -it -p 7001:8001 -v /var/run/docker.sock:/var/run/docker.sock nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3 /bin/bash

pallavi1.halarnkar · April 27, 2022, 6:43am

No i am using exec command as given below

docker exec -it taoyolov4 bash

I had created container named taoyolov4 using volume option, but while executing the container everytime i dont use volume option as in docker exec there is no -v option

Morganh · April 27, 2022, 6:49am

Officially we recommend users trigger jobs via tao launcher. See TAO Toolkit Launcher — TAO Toolkit 3.22.05 documentation
In this way, please set correct ~/.tao_mounts.json.
Then run something like below.
$ tao yolov4 train xxx
…

And you can also run below to check if the files are available. For example,
$ tao yolov4 run ls xxx

Besides tao launcher, if you trigger docker as below.
$docker run --runtime=nvidia -it -p 7001:8001 -v /var/run/docker.sock:/var/run/docker.sock -v yourlocal_folder:/workspace nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3
/bin/bash

In this way, you will find your files under /workspace . It is not related to ~/.tao_mounts.json.

pallavi1.halarnkar · April 27, 2022, 6:54am

i wanted a persistent container so i gave it a name called as taoyolov4, this run command will create a new temporary container whose status is not persistent , so i use the exec command to start my already created container named taoyolov4.

Tell me what to do for a persistent container so that i dont have to create a new container everytime.

Morganh · April 27, 2022, 6:56am

You can use the 2nd method.
Then there is a name when you run “$docker ps”.
You can use the container id to run “docker exec” every time.

pallavi1.halarnkar · April 27, 2022, 8:14am

Used the second method as suggested still getting file not found error

yolo_v4 dataset_convert -d $SPECS_DIR/yolo_v4_tfrecords_kitti_train.txt \
>                              -o $DATA_DOWNLOAD_DIR/training/tfrecords/train
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/dataset_convert.py", line 18, in <module>
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/dataset_convert.py", line 110, in main
FileNotFoundError: [Errno 2] No such file or directory: '/yolo_v4_tfrecords_kitti_train.txt'

Morganh · April 27, 2022, 8:17am

As mentioned above, -v yourlocal_folder:/workspace will map your local directory to the docker. Can you check if the file path is correct?

pallavi1.halarnkar · April 27, 2022, 8:23am

As suggested i have used this command

sudo docker run --runtime=nvidia -it -p 7001:8001 --name abc -v /var/run/docker.sock:/var/run/docker.sock -v tao_project:/workspace nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3 /bin/bash

a folder named tao_project is created on my host pc in /var/lib/docker/volumes path

Now i have updated this tao_project folder name in the jupyter notebook as follows .

# Setting up env variables for cleaner command line commands.
import os

print("Please replace the variable with your key.")
%env KEY=nvidia_tlt
%env USER_EXPERIMENT_DIR=/workspace/tao-experiments/yolo_v4
%env DATA_DOWNLOAD_DIR=/workspace/tao-experiments/data

# Set this path if you don't run the notebook from the samples directory.
# %env NOTEBOOK_ROOT=~/tao-samples/yolo_v4

# Please define this local project directory that needs to be mapped to the TAO docker session.
# The dataset expected to be present in $LOCAL_PROJECT_DIR/data, while the results for the steps
# in this notebook will be stored at $LOCAL_PROJECT_DIR/yolo_v4
%env LOCAL_PROJECT_DIR=tao_project
os.environ["LOCAL_DATA_DIR"] = os.path.join(os.getenv("LOCAL_PROJECT_DIR", os.getcwd()), "data")
os.environ["LOCAL_EXPERIMENT_DIR"] = os.path.join(os.getenv("LOCAL_PROJECT_DIR", os.getcwd()), "yolo_v4")

# The sample spec files are present in the same path as the downloaded samples.
os.environ["LOCAL_SPECS_DIR"] = os.path.join(
    os.getenv("NOTEBOOK_ROOT", os.getcwd()),
    "specs"
)
%env SPECS_DIR=/workspace/yolo_v4/specs

# Showing list of specification files.
!ls -rlt $LOCAL_SPECS_DIR

Pls check if i m going wrong in setting the Local_Proj_dir or other paths

Morganh · April 27, 2022, 8:27am

You can debug in terminal firstly instead of using jupyter notebook.

pallavi1.halarnkar · April 27, 2022, 8:31am

This output displayed is in the terminal itself not from jupyter notebook

root@3b5ffffef2b0:/workspace# yolo_v4 dataset_convert -d $SPECS_DIR/yolo_v4_tfrecords_kitti_train.txt \
>                              -o $DATA_DOWNLOAD_DIR/training/tfrecords/train
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/dataset_convert.py", line 18, in <module>
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/dataset_convert.py", line 110, in main
FileNotFoundError: [Errno 2] No such file or directory: '/yolo_v4_tfrecords_kitti_train.txt'
root@3b5ffffef2b0:/workspace# ^C
root@3b5ffffef2b0:/workspace# yolo_v4 dataset_convert -d $SPECS_DIR/yolo_v4_tfrecords_kitti_train.txt \
>                              -o $DATA_DOWNLOAD_DIR/training/tfrecords/train
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/dataset_convert.py", line 18, in <module>
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/dataset_convert.py", line 110, in main
FileNotFoundError: [Errno 2] No such file or directory: '/yolo_v4_tfrecords_kitti_train.txt'

Morganh · April 27, 2022, 8:35am

Where is your yolo_v4_tfrecords_kitti_train.txt ? If it locates at tao_project/yolo_v4_tfrecords_kitti_train.txt, then you can find it in the path /workspace/yolo_v4_tfrecords_kitti_train.txt

pallavi1.halarnkar · April 27, 2022, 8:40am

On host pc i can see yolo_v4_tfrecords_kitti_train.txt in the following path

/var/lib/docker/volumes/tao_project/_data/cv_samples_v1.3.0/yolo_v4/specs/yolo_v4_tfrecords_kitti_train.txt

when i browse through my docker i can see it in

:/workspace/cv_samples_v1.3.0/yolo_v4/specs/yolo_v4_tfrecords_kitti_train.txt

Morganh · April 27, 2022, 9:10am

Can you change your local files into another path instead of /var/lib/docker/ ?
Assume you put the spec files in /home/xxx/tao_project

Then
-v /home/xxx/tao_project:/workspace

Then, you can find all the files will be available in /workspace of the docker.

pallavi1.halarnkar · April 27, 2022, 11:43am

I am still getting the same error.

My local_project_dir is in /home/pallavi/tao_project and i have executed the docker using the command below

sudo docker run --runtime=nvidia -it -p 8888:9999 --name yolotao -v /var/run/docker.sock:/var/run/docker.sock -v /home/pallavi/tao_project:/workspace nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3 /bin/bash

Error on the command prompt while converting the dataset

root@59eed8d6e529:/workspace# yolo_v4 dataset_convert -d $SPECS_DIR/yolo_v4_tfrecords_kitti_train.txt \
>                              -o $DATA_DOWNLOAD_DIR/training/tfrecords/train
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/dataset_convert.py", line 18, in <module>
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/dataset_convert.py", line 110, in main
FileNotFoundError: [Errno 2] No such file or directory: '/yolo_v4_tfrecords_kitti_train.txt'

Morganh · April 27, 2022, 1:44pm

What is the “$SPECS_DIR” ? Is it correct?
You can change it to explicit path.

pallavi1.halarnkar · April 28, 2022, 6:13am

I think i have found the error. When i do ls to the host pc project directory it gives me permission denied error.

I am not able to access my host pc folder/file inside the docker container.

Can you now help me how to resolve this problem

Is it by setting UID and GID if yes where to set them

Docker run command OR
in mounts.json file

Pls guide

Morganh · April 29, 2022, 1:47am

No, if you trigger docker via -v yourlocal_folder:/workspace, you can access your hos pc folder/file inside the docker.

yingliu · May 17, 2022, 4:05am

I’m closing this topic due to there is no update from you for a period, assuming this issue was resolved.
If still need the support, please open a new topic. Thanks

system · May 31, 2022, 4:06am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.