Please provide the following information when requesting support.
• Hardware (2080ti)
• Network Type (Detectnet_v2/Yolo_v4/Classification)
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here)
• Training spec file(If have, please share here)
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)
I am running the yolo_v4_tiny example.
It seems that the yolo_v4_tiny dataset_convert has been successfully run, but I can`t find the /home/jeff/tao-experiments/data/training/tfrecords file in my host.
This is how my file structed.
For running
!tao yolo_v4_tiny dataset_convert -d $SPECS_DIR/yolo_v4_tiny_tfrecords_kitti_train.txt -o $DATA_DOWNLOAD_DIR/training/tfrecords/train
the log is below:
2022-03-30 11:32:20,448 [INFO] root: Registry: [‘nvcr.io’]
2022-03-30 11:32:20,638 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3
2022-03-30 11:32:20,701 [WARNING] tlt.components.docker_handler.docker_handler:
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the “user”:“UID:GID” in the
DockerOptions portion of the “/home/jeff/.tao_mounts.json” file. You can obtain your
users UID and GID by using the “id -u” and “id -g” commands on the
terminal.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
2022-03-30 03:32:27,414 [INFO] iva.detectnet_v2.dataio.build_converter: Instantiating a kitti converter
2022-03-30 03:32:27,414 [INFO] root: Instantiating a kitti converter
2022-03-30 03:32:27,414 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Creating output directory /home/jeff/tao-experiments/data/training/tfrecords
2022-03-30 03:32:27,414 [INFO] root: Generating partitions
2022-03-30 03:32:27,433 [INFO] iva.detectnet_v2.dataio.kitti_converter_lib: Num images in
Train: 5212 Val: 848
2022-03-30 03:32:27,433 [INFO] root: Num images in
Train: 5212 Val: 848
2022-03-30 03:32:27,433 [INFO] iva.detectnet_v2.dataio.kitti_converter_lib: Validation data in partition 0. Hence, while choosing the validationset during training choose validation_fold 0.
2022-03-30 03:32:27,433 [INFO] root: Validation data in partition 0. Hence, while choosing the validationset during training choose validation_fold 0.
2022-03-30 03:32:27,438 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 0
2022-03-30 03:32:27,438 [INFO] root: Writing partition 0, shard 0
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py:161: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.2022-03-30 03:32:27,438 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py:161: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:297: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.
2022-03-30 03:32:27,526 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 1
2022-03-30 03:32:27,526 [INFO] root: Writing partition 0, shard 1
2022-03-30 03:32:27,608 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 2
2022-03-30 03:32:27,608 [INFO] root: Writing partition 0, shard 2
2022-03-30 03:32:27,691 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 3
2022-03-30 03:32:27,691 [INFO] root: Writing partition 0, shard 3
2022-03-30 03:32:27,772 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 4
2022-03-30 03:32:27,772 [INFO] root: Writing partition 0, shard 4
2022-03-30 03:32:27,855 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 5
2022-03-30 03:32:27,855 [INFO] root: Writing partition 0, shard 5
2022-03-30 03:32:27,936 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 6
2022-03-30 03:32:27,936 [INFO] root: Writing partition 0, shard 6
2022-03-30 03:32:28,019 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 7
2022-03-30 03:32:28,019 [INFO] root: Writing partition 0, shard 7
2022-03-30 03:32:28,100 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 8
2022-03-30 03:32:28,100 [INFO] root: Writing partition 0, shard 8
2022-03-30 03:32:28,183 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 9
2022-03-30 03:32:28,183 [INFO] root: Writing partition 0, shard 9
2022-03-30 03:32:28,272 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib:
Wrote the following numbers of objects:
b’pedestrian’: 455
b’car’: 3285
b’dontcare’: 1338
b’van’: 349
b’misc’: 98
b’truck’: 121
b’cyclist’: 193
b’tram’: 52
b’person_sitting’: 232022-03-30 03:32:28,272 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 0
2022-03-30 03:32:28,272 [INFO] root: Writing partition 1, shard 0
2022-03-30 03:32:28,775 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 1
2022-03-30 03:32:28,775 [INFO] root: Writing partition 1, shard 1
2022-03-30 03:32:29,283 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 2
2022-03-30 03:32:29,284 [INFO] root: Writing partition 1, shard 2
2022-03-30 03:32:29,784 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 3
2022-03-30 03:32:29,784 [INFO] root: Writing partition 1, shard 3
2022-03-30 03:32:30,281 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 4
2022-03-30 03:32:30,281 [INFO] root: Writing partition 1, shard 4
2022-03-30 03:32:30,782 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 5
2022-03-30 03:32:30,782 [INFO] root: Writing partition 1, shard 5
2022-03-30 03:32:31,283 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 6
2022-03-30 03:32:31,283 [INFO] root: Writing partition 1, shard 6
2022-03-30 03:32:31,787 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 7
2022-03-30 03:32:31,787 [INFO] root: Writing partition 1, shard 7
2022-03-30 03:32:32,293 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 8
2022-03-30 03:32:32,293 [INFO] root: Writing partition 1, shard 8
2022-03-30 03:32:32,806 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 9
2022-03-30 03:32:32,806 [INFO] root: Writing partition 1, shard 9
2022-03-30 03:32:33,315 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib:
Wrote the following numbers of objects:
b’car’: 19969
b’misc’: 697
b’dontcare’: 7818
b’cyclist’: 1169
b’van’: 2028
b’pedestrian’: 3169
b’truck’: 772
b’tram’: 346
b’person_sitting’: 1492022-03-30 03:32:33,315 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Cumulative object statistics
2022-03-30 03:32:33,315 [INFO] root: Cumulative object statistics
2022-03-30 03:32:33,316 [INFO] root: {
“pedestrian”: 3624,
“car”: 23254,
“dontcare”: 9156,
“van”: 2377,
“misc”: 795,
“truck”: 893,
“cyclist”: 1362,
“tram”: 398,
“person_sitting”: 172
}
2022-03-30 03:32:33,316 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib:
Wrote the following numbers of objects:
b’pedestrian’: 3624
b’car’: 23254
b’dontcare’: 9156
b’van’: 2377
b’misc’: 795
b’truck’: 893
b’cyclist’: 1362
b’tram’: 398
b’person_sitting’: 1722022-03-30 03:32:33,316 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Class map.
Label in GT: Label in tfrecords file
b’Pedestrian’: b’pedestrian’
b’Car’: b’car’
b’DontCare’: b’dontcare’
b’Van’: b’van’
b’Misc’: b’misc’
b’Truck’: b’truck’
b’Cyclist’: b’cyclist’
b’Tram’: b’tram’
b’Person_sitting’: b’person_sitting’
2022-03-30 03:32:33,316 [INFO] root: Class map.
Label in GT: Label in tfrecords file
b’Pedestrian’: b’pedestrian’
b’Car’: b’car’
b’DontCare’: b’dontcare’
b’Van’: b’van’
b’Misc’: b’misc’
b’Truck’: b’truck’
b’Cyclist’: b’cyclist’
b’Tram’: b’tram’
b’Person_sitting’: b’person_sitting’
For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.2022-03-30 03:32:33,316 [INFO] root: For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.
2022-03-30 03:32:33,316 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Tfrecords generation complete.
2022-03-30 03:32:33,316 [INFO] root: TFRecords generation complete.
2022-03-30 11:32:34,188 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.