Detectnet2 TAO Toolkit model training fail on formating dataset on kitti format

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Please provide the following information when requesting support.

• Hardware (T4)
• Network Type (Detectnet_v2)
• (launcher) glueck@glueck-WHITLEY:~/getting_started_v5.1.0/setup$ docker -v
Docker version 24.0.7, build afdd53b
detectnet_v2_tfrecords_kitti_trainval.txt (438 Bytes)

• Training spec file(If have, please share here)
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)

import os

Set your environment variables if not set already

os.environ[“LOCAL_DATA_DIR”] = “/home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/data”
os.environ[“SPECS_DIR”] = “/home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/specs”
os.environ[“USER_EXPERIMENT_DIR”] = “/workspace/tao-experiments/detectnet_v2”

Create a new directory for the output TFRecords dump

output_tfrecords_dir = os.path.join(os.environ[“LOCAL_DATA_DIR”], “tfrecords”, “kitti_trainval”)
os.makedirs(output_tfrecords_dir, exist_ok=True)
os.system(f’rm -rf {output_tfrecords_dir}/*')

Run the dataset conversion using TAO

print(“Converting Tfrecords for Penang dataset”)
!tao model detectnet_v2 dataset_convert
-d $SPECS_DIR/detectnet_v2_tfrecords_kitti_trainval.txt
-o $output_tfrecords_dir
-r $USER_EXPERIMENT_DIR/

Traceback (most recent call last):
File “/home/glueck/miniconda3/envs/launcher/bin/tao”, line 8, in
sys.exit(main())
File “/home/glueck/miniconda3/envs/launcher/lib/python3.7/site-packages/nvidia_tao_cli/entrypoint/tao_launcher.py”, line 137, in main
args[2:]
File “/home/glueck/miniconda3/envs/launcher/lib/python3.7/site-packages/nvidia_tao_cli/components/instance_handler/local_instance.py”, line 382, in launch_command
docker_handler.run_container(command)
File “/home/glueck/miniconda3/envs/launcher/lib/python3.7/site-packages/nvidia_tao_cli/components/docker_handler/docker_handler.py”, line 325, in run_container
self.pull()
File “/home/glueck/miniconda3/envs/launcher/lib/python3.7/site-packages/nvidia_tao_cli/components/docker_handler/docker_handler.py”, line 187, in pull
docker_pull_progress(line, progress)
File “/home/glueck/miniconda3/envs/launcher/lib/python3.7/site-packages/nvidia_tao_cli/components/docker_handler/docker_handler.py”, line 66, in docker_pull_progress
TASKS[idx] = progress.add_task(f"{idx}", total=line[‘progressDetail’][‘total’])
KeyError: ‘total’

3

How about
! tao info --verbose

4

Configuration of the TAO Toolkit Instance

task_group:
model:
dockers:
nvidia/tao/tao-toolkit:
5.0.0-tf2.11.0:
docker_registry: nvcr.io
tasks:
1. classification_tf2
2. efficientdet_tf2
5.0.0-tf1.15.5:
docker_registry: nvcr.io
tasks:
1. bpnet
2. classification_tf1
3. converter
4. detectnet_v2
5. dssd
6. efficientdet_tf1
7. faster_rcnn
8. fpenet
9. lprnet
10. mask_rcnn
11. multitask_classification
12. retinanet
13. ssd
14. unet
15. yolo_v3
16. yolo_v4
17. yolo_v4_tiny
5.2.0-pyt2.1.0:
docker_registry: nvcr.io
tasks:
1. action_recognition
2. centerpose
3. deformable_detr
4. dino
5. mal
6. ml_recog
7. ocdnet
8. ocrnet
9. optical_inspection
10. pointpillars
11. pose_classification
12. re_identification
13. visual_changenet
5.2.0-pyt1.14.0:
docker_registry: nvcr.io
tasks:
1. classification_pyt
2. segformer
dataset:
dockers:
nvidia/tao/tao-toolkit:
5.2.0-data-services:
docker_registry: nvcr.io
tasks:
1. augmentation
2. auto_label
3. annotations
4. analytics
deploy:
dockers:
nvidia/tao/tao-toolkit:
5.2.0-deploy:
docker_registry: nvcr.io
tasks:
1. visual_changenet
2. centerpose
3. classification_pyt
4. classification_tf1
5. classification_tf2
6. deformable_detr
7. detectnet_v2
8. dino
9. dssd
10. efficientdet_tf1
11. efficientdet_tf2
12. faster_rcnn
13. lprnet
14. mask_rcnn
15. ml_recog
16. multitask_classification
17. ocdnet
18. ocrnet
19. optical_inspection
20. retinanet
21. segformer
22. ssd
23. trtexec
24. unet
25. yolo_v3
26. yolo_v4
27. yolo_v4_tiny
format_version: 3.0
toolkit_version: 5.2.0
published_date: 12/06/2023

5

Could you run below in terminal instead of notebook?
$ docker images |grep tao

6

(launcher) glueck@glueck-WHITLEY:~$ docker images |grep tao

no output shown

7

OK, please docker pull the docker in the terminal.
$ docker pull nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf1.15.5

Then, close previous notebook. And trigger notebook again. To check if there is still issue.

8

Converting Tfrecords for Penang dataset
2024-01-11 11:54:30,056 [TAO Toolkit] [INFO] root 160: Registry: [‘nvcr.io’]
2024-01-11 11:54:30,126 [TAO Toolkit] [INFO] nvidia_tao_cli.components.instance_handler.local_instance 361: Running command in container: nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf1.15.5
2024-01-11 11:54:30,139 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 301: Printing tty value True
2024-01-11 03:54:30.862658: I tensorflow/stream_executor/platform/default/dso_loader.cc:50] Successfully opened dynamic library libcudart.so.12
2024-01-11 03:54:30,916 [TAO Toolkit] [WARNING] tensorflow 40: Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
2024-01-11 03:54:32,623 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use sklearn by default. This improves performance in some cases. To enable sklearn export the environment variable TF_ALLOW_IOLIBS=1.
2024-01-11 03:54:32,665 [TAO Toolkit] [WARNING] tensorflow 42: TensorFlow will not use Dask by default. This improves performance in some cases. To enable Dask export the environment variable TF_ALLOW_IOLIBS=1.
2024-01-11 03:54:32,670 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use Pandas by default. This improves performance in some cases. To enable Pandas export the environment variable TF_ALLOW_IOLIBS=1.
2024-01-11 03:54:34,541 [TAO Toolkit] [WARNING] matplotlib 500: Matplotlib created a temporary config/cache directory at /tmp/matplotlib-fqm4yfh7 because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
2024-01-11 03:54:34,861 [TAO Toolkit] [INFO] matplotlib.font_manager 1633: generated new fontManager
usage: detectnet_v2 dataset_convert [-h] [–num_processes NUM_PROCESSES]
[–gpus GPUS]
[–gpu_index GPU_INDEX [GPU_INDEX …]]
[–use_amp] [–log_file LOG_FILE] -d
DATASET_EXPORT_SPEC -o OUTPUT_FILENAME
[-f VALIDATION_FOLD] [-v] [-r RESULTS_DIR]
{train,prune,inference,export,evaluate,dataset_convert,calibration_tensorfile}
…
detectnet_v2 dataset_convert: error: argument -o/–output_filename: expected one argument
2024-01-11 11:54:36,109 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 363: Stopping container.

this is the details
kitti_config {
root_directory_path: “/home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/data/train”
image_dir_name: “images”
label_dir_name: “label”
image_extension: “.jpg”
partition_mode: “random”
num_partitions: 2
val_split: 14
num_shards: 10
}
image_directory_path: “/home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/data/train/images”

the code:
import os

Set your environment variables if not set already

os.environ[“LOCAL_DATA_DIR”] = “/home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/data”
os.environ[“SPECS_DIR”] = “/home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/specs”
os.environ[“USER_EXPERIMENT_DIR”] = “/workspace/tao-experiments/detectnet_v2”

Create a new directory for the output TFRecords dump

output_tfrecords_dir = os.path.join(os.environ[“LOCAL_DATA_DIR”], “tfrecords”, “kitti_trainval”)
os.makedirs(output_tfrecords_dir, exist_ok=True)
os.system(f’rm -rf {output_tfrecords_dir}/*')

Run the dataset conversion using TAO

print(“Converting Tfrecords for Penang dataset”)
!tao model detectnet_v2 dataset_convert
-d $SPECS_DIR/detectnet_v2_tfrecords_kitti_trainval.txt
-o $output_tfrecords_dir
-r $USER_EXPERIMENT_DIR/

9

Please make sure the path is available after -o.
Since you already restart the notebook, you can find the cell where to set the $USER_EXPERIMENT_DIR/ and rerun the cell.
Or simply you can set $USER_EXPERIMENT_DIR/ with explicit path instead of env variable.

10

import os

Set your environment variables

os.environ[“LOCAL_DATA_DIR”] = “/home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/data”
os.environ[“SPECS_DIR”] = “/home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/specs”
os.environ[“USER_EXPERIMENT_DIR”] = “/workspace/tao-experiments/detectnet_v2”# Create a new directory for the output TFRecords dump
output_tfrecords_dir = os.path.join(os.environ[“LOCAL_DATA_DIR”], “tfrecords”, “penang_trainval”)
os.makedirs(output_tfrecords_dir, exist_ok=True)
os.system(f’rm -rf {output_tfrecords_dir}/*')

Run the dataset conversion using TAO

print(“Converting Tfrecords for Penang dataset”)
!tao model detectnet_v2 dataset_convert
-d $SPECS_DIR/detectnet_v2_tfrecords_kitti_trainval.txt
-o $output_tfrecords_dir
-r $USER_EXPERIMENT_DIR/

this is the code where should i need to change

11

You can run the cell " Set your environment variables" again.

12

This is the error i got while running the “Set your environment variables” cell

Converting Tfrecords for Penang dataset
2024-01-11 14:30:10,071 [TAO Toolkit] [INFO] root 160: Registry: [‘nvcr.io’]
2024-01-11 14:30:10,145 [TAO Toolkit] [INFO] nvidia_tao_cli.components.instance_handler.local_instance 361: Running command in container: nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf1.15.5
2024-01-11 14:30:10,159 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 301: Printing tty value True
2024-01-11 06:30:10.836621: I tensorflow/stream_executor/platform/default/dso_loader.cc:50] Successfully opened dynamic library libcudart.so.12
2024-01-11 06:30:10,888 [TAO Toolkit] [WARNING] tensorflow 40: Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
2024-01-11 06:30:12,526 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use sklearn by default. This improves performance in some cases. To enable sklearn export the environment variable TF_ALLOW_IOLIBS=1.
2024-01-11 06:30:12,567 [TAO Toolkit] [WARNING] tensorflow 42: TensorFlow will not use Dask by default. This improves performance in some cases. To enable Dask export the environment variable TF_ALLOW_IOLIBS=1.
2024-01-11 06:30:12,571 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use Pandas by default. This improves performance in some cases. To enable Pandas export the environment variable TF_ALLOW_IOLIBS=1.
2024-01-11 06:30:14,420 [TAO Toolkit] [WARNING] matplotlib 500: Matplotlib created a temporary config/cache directory at /tmp/matplotlib-8z4wiit6 because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
2024-01-11 06:30:14,726 [TAO Toolkit] [INFO] matplotlib.font_manager 1633: generated new fontManager
usage: detectnet_v2 dataset_convert [-h] [–num_processes NUM_PROCESSES]
[–gpus GPUS]
[–gpu_index GPU_INDEX [GPU_INDEX …]]
[–use_amp] [–log_file LOG_FILE] -d
DATASET_EXPORT_SPEC -o OUTPUT_FILENAME
[-f VALIDATION_FOLD] [-v] [-r RESULTS_DIR]
{train,prune,inference,export,evaluate,dataset_convert,calibration_tensorfile}
…
detectnet_v2 dataset_convert: error: argument -o/–output_filename: expected one argument
2024-01-11 14:30:15,820 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 363: Stopping container.

13

Please share the result of
! tao model detectnet_v2 run ls $output_tfrecords_dir

and
! tao model detectnet_v2 run echo $output_tfrecords_dir

14

2024-01-11 14:35:24,027 [TAO Toolkit] [INFO] root 160: Registry: [‘nvcr.io’]
2024-01-11 14:35:24,105 [TAO Toolkit] [INFO] nvidia_tao_cli.components.instance_handler.local_instance 361: Running command in container: nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf1.15.5
2024-01-11 14:35:24,119 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 301: Printing tty value True
ls: cannot access ‘/home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/data/tfrecords/penang_trainval’: No such file or directory
2024-01-11 14:35:24,483 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 363: Stopping container.
detectnet_v2_tfrecords_kitti_trainval.txt (438 Bytes)

15

Can you share the tao_mounts.json file ?

16

2024-01-11 14:39:43,426 [TAO Toolkit] [INFO] root 160: Registry: [‘nvcr.io’]
2024-01-11 14:39:43,502 [TAO Toolkit] [INFO] nvidia_tao_cli.components.instance_handler.local_instance 361: Running command in container: nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf1.15.5
2024-01-11 14:39:43,516 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 301: Printing tty value True
/home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/data/tfrecords/penang_trainval
2024-01-11 14:39:43,860 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 363: Stopping container.

17

{
“Mounts”: [
{
“source”: “/home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection”,
“destination”: “/workspace/tao-experiments”
},
{
“source”: “/home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/specs”,
“destination”: “/workspace/tao-experiments/detectnet_v2/specs”
}
],
“DockerOptions”: {
“user”: “1000:1000”
}
}

import json
import os

Set this to the path where you want to save the mounts file

mounts_file_path = “/home/glueck/.tao_mounts.json”

Define the dictionary with the mapped drives

drive_map = {
“Mounts”: [
# Mapping the data directory
{
“source”: os.environ[“LOCAL_PROJECT_DIR”],
“destination”: “/workspace/tao-experiments”
},
# Mapping the specs directory
{
“source”: os.path.join(os.environ[“LOCAL_PROJECT_DIR”], “specs”),
“destination”: “/workspace/tao-experiments/detectnet_v2/specs”
},
# Add additional mappings if needed
],
“DockerOptions”: {
“user”: f"{os.getuid()}:{os.getgid()}"
# Add more Docker options if needed
}
}

Writing the mounts file

with open(mounts_file_path, “w”) as mfile:
json.dump(drive_map, mfile, indent=4)
print(f"Mounts file created at: {mounts_file_path}")

18

So, when you set -o $output_tfrecords_dir in the command line, the $output_tfrecords_dir is actually the local path instead of a path inside the docker. Because you did you set it in the tao_mounts.json.

Suggest you to map it as well.
Or change command line to
-o /workspace/tao-experiments/penang_trianval

Or refer to tao_tutorials/notebooks/tao_launcher_starter_kit/detectnet_v2/detectnet_v2.ipynb at main · NVIDIA/tao_tutorials · GitHub.

19

(base) glueck@glueck-WHITLEY:~$ ls /home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/data/train
images label
(base) glueck@glueck-WHITLEY:~$
(base) glueck@glueck-WHITLEY:~$ ls /home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/data/train/images
00001.jpg 00003.jpg 00005.jpg 00007.jpg 00009.jpg 00011.jpg 00013.jpg 00015.jpg 00017.jpg 00019.jpg 00021.jpg 00023.jpg 00025.jpg 00027.jpg 00029.jpg
00002.jpg 00004.jpg 00006.jpg 00008.jpg 00010.jpg 00012.jpg 00014.jpg 00016.jpg 00018.jpg 00020.jpg 00022.jpg 00024.jpg 00026.jpg 00028.jpg 00030.jpg
(base) glueck@glueck-WHITLEY:~$ ls /home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/specs/detectnet_v2_tfrecords_kitti_trainval.txt
/home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/specs/detectnet_v2_tfrecords_kitti_trainval.txt
(base) glueck@glueck-WHITLEY:~$ ls -lh /home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/specs/detectnet_v2_tfrecords_kitti_trainval.txt
-rw-rw-r-- 1 glueck glueck 438 Jan 10 17:27 /home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/specs/detectnet_v2_tfrecords_kitti_trainval.txt
(base) glueck@glueck-WHITLEY:~$
(base) glueck@glueck-WHITLEY:~$ ls /home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/data/tfrecords/kitti_trainval
(base) glueck@glueck-WHITLEY:~$ ls /home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/data/tfrecords/
kitti_trainval penang_trainval
(base) glueck@glueck-WHITLEY:~$ ls workspace/tao-experiments/penang_trianval
ls: cannot access ‘workspace/tao-experiments/penang_trianval’: No such file or directory
(base) glueck@glueck-WHITLEY:~$ ls /workspace/tao-experiments/penang_trianval
ls: cannot access ‘/workspace/tao-experiments/penang_trianval’: No such file or directory
(base) glueck@glueck-WHITLEY:~$ ls /workspace/tao-experiments/detectnet_v2
ls: cannot access ‘/workspace/tao-experiments/detectnet_v2’: No such file or directory

import os

Set your environment variables if not set already

os.environ[“LOCAL_DATA_DIR”] = “/home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/data”
os.environ[“SPECS_DIR”] = “/home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/specs”
os.environ[“USER_EXPERIMENT_DIR”] = “/workspace/tao-experiments/detectnet_v2”

Create a new directory for the output TFRecords dump

output_tfrecords_dir = os.path.join(os.environ[“LOCAL_DATA_DIR”], “tfrecords”, “kitti_trainval”)
os.makedirs(output_tfrecords_dir, exist_ok=True)
os.system(f’rm -rf {output_tfrecords_dir}/*')

Run the dataset conversion using TAO

print(“Converting Tfrecords for Penang dataset”)
!tao model detectnet_v2 dataset_convert
-d $SPECS_DIR/detectnet_v2_tfrecords_kitti_trainval.txt
-o $/workspace/tao-experiments/penang_trianval
-r $USER_EXPERIMENT_DIR/

Converting Tfrecords for Penang dataset
2024-01-11 15:14:43,260 [TAO Toolkit] [INFO] root 160: Registry: [‘nvcr.io’]
2024-01-11 15:14:43,342 [TAO Toolkit] [INFO] nvidia_tao_cli.components.instance_handler.local_instance 361: Running command in container: nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf1.15.5
2024-01-11 15:14:43,356 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 301: Printing tty value True
2024-01-11 07:14:44.041641: I tensorflow/stream_executor/platform/default/dso_loader.cc:50] Successfully opened dynamic library libcudart.so.12
2024-01-11 07:14:44,092 [TAO Toolkit] [WARNING] tensorflow 40: Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
2024-01-11 07:14:45,738 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use sklearn by default. This improves performance in some cases. To enable sklearn export the environment variable TF_ALLOW_IOLIBS=1.
2024-01-11 07:14:45,780 [TAO Toolkit] [WARNING] tensorflow 42: TensorFlow will not use Dask by default. This improves performance in some cases. To enable Dask export the environment variable TF_ALLOW_IOLIBS=1.
2024-01-11 07:14:45,784 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use Pandas by default. This improves performance in some cases. To enable Pandas export the environment variable TF_ALLOW_IOLIBS=1.
2024-01-11 07:14:47,669 [TAO Toolkit] [WARNING] matplotlib 500: Matplotlib created a temporary config/cache directory at /tmp/matplotlib-zr7r1892 because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
2024-01-11 07:14:47,977 [TAO Toolkit] [INFO] matplotlib.font_manager 1633: generated new fontManager
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
WARNING:tensorflow:TensorFlow will not use sklearn by default. This improves performance in some cases. To enable sklearn export the environment variable TF_ALLOW_IOLIBS=1.
2024-01-11 07:14:49,890 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use sklearn by default. This improves performance in some cases. To enable sklearn export the environment variable TF_ALLOW_IOLIBS=1.
WARNING:tensorflow:TensorFlow will not use Dask by default. This improves performance in some cases. To enable Dask export the environment variable TF_ALLOW_IOLIBS=1.
2024-01-11 07:14:49,930 [TAO Toolkit] [WARNING] tensorflow 42: TensorFlow will not use Dask by default. This improves performance in some cases. To enable Dask export the environment variable TF_ALLOW_IOLIBS=1.
WARNING:tensorflow:TensorFlow will not use Pandas by default. This improves performance in some cases. To enable Pandas export the environment variable TF_ALLOW_IOLIBS=1.
2024-01-11 07:14:49,935 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use Pandas by default. This improves performance in some cases. To enable Pandas export the environment variable TF_ALLOW_IOLIBS=1.
2024-01-11 07:14:50,536 [TAO Toolkit] [INFO] nvidia_tao_tf1.cv.common.logging.logging 197: Log file already exists at /workspace/tao-experiments/detectnet_v2/status.json
2024-01-11 07:14:50,536 [TAO Toolkit] [INFO] root 2102: Starting Object Detection Dataset Convert.
2024-01-11 07:14:50,537 [TAO Toolkit] [INFO] root 2102: [Errno 2] No such file or directory: ‘/home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/specs/detectnet_v2_tfrecords_kitti_trainval.txt’
Traceback (most recent call last):
File “/usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/detectnet_v2/scripts/dataset_convert.py”, line 168, in
raise e
File “/usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/detectnet_v2/scripts/dataset_convert.py”, line 137, in
main()
File “/usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/detectnet_v2/scripts/dataset_convert.py”, line 128, in main
with open(expand_path(args.dataset_export_spec), “r”) as f:
FileNotFoundError: [Errno 2] No such file or directory: ‘/home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/specs/detectnet_v2_tfrecords_kitti_trainval.txt’
Execution status: FAIL
2024-01-11 15:15:00,521 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 363: Stopping container.

20

To ease your work, please modify your tao_mounts.json file to set the same for “source” and “destination”.

Change

“Mounts”: [
# Mapping the data directory
{
“source”: os.environ[“LOCAL_PROJECT_DIR”],
“destination”: “/workspace/tao-experiments”
},
# Mapping the specs directory
{
“source”: os.path.join(os.environ[“LOCAL_PROJECT_DIR”], “specs”),
“destination”: “/workspace/tao-experiments/detectnet_v2/specs”
},
# Add additional mappings if needed
],

to

“Mounts”: [
# Mapping the data directory
{
“source”: “/home/glueck”,
“destination”: “/home/glueck”
}
],
21

import json
import os

Set this to the path where you want to save the mounts file

mounts_file_path = “/home/glueck/.tao_mounts.json”

Define the dictionary with the mapped drives

drive_map = {
“Mounts”: [
# Mapping the data directory
{
“source”: “/home/glueck”,
“destination”: “/home/glueck”
},
],
“DockerOptions”: {
“user”: f"{os.getuid()}:{os.getgid()}"
# Add more Docker options if needed
}
}

Writing the mounts file

with open(mounts_file_path, “w”) as mfile:
json.dump(drive_map, mfile, indent=4)
print(f"Mounts file created at: {mounts_file_path}")

like this is it

import os

Set your environment variables if not set already

os.environ[“LOCAL_DATA_DIR”] = “/home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/data”
os.environ[“SPECS_DIR”] = “/home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/specs”
os.environ[“USER_EXPERIMENT_DIR”] = “/home/glueck/getting_started_v5.1.0/notebooks/tao_launcher_starter_kit/PenangModelDetection/detectnet_v2”

Create a new directory for the output TFRecords dump

output_tfrecords_dir = os.path.join(os.environ[“LOCAL_DATA_DIR”], “tfrecords”, “kitti_trainval”)
os.makedirs(output_tfrecords_dir, exist_ok=True)
os.system(f’rm -rf {output_tfrecords_dir}/*')

Run the dataset conversion using TAO

print(“Converting Tfrecords for Penang dataset”)
!tao model detectnet_v2 dataset_convert
-d $SPECS_DIR/detectnet_v2_tfrecords_kitti_trainval.txt
-o $output_tfrecords_dir
-r $USER_EXPERIMENT_DIR/

here in this code i should change anything?