Tfrecords file can not be created when running Generate tfrecords in yolo_v4_tiny example

Please provide the following information when requesting support.

• Hardware (2080ti)
• Network Type (Detectnet_v2/Yolo_v4/Classification)
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here)
• Training spec file(If have, please share here)
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)

I am running the yolo_v4_tiny example.
It seems that the yolo_v4_tiny dataset_convert has been successfully run, but I can`t find the /home/jeff/tao-experiments/data/training/tfrecords file in my host.

This is how my file structed.
企业微信截图_20220330114320

For running
!tao yolo_v4_tiny dataset_convert -d $SPECS_DIR/yolo_v4_tiny_tfrecords_kitti_train.txt -o $DATA_DOWNLOAD_DIR/training/tfrecords/train
the log is below:

2022-03-30 11:32:20,448 [INFO] root: Registry: [‘nvcr.io’]
2022-03-30 11:32:20,638 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3
2022-03-30 11:32:20,701 [WARNING] tlt.components.docker_handler.docker_handler:
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the “user”:“UID:GID” in the
DockerOptions portion of the “/home/jeff/.tao_mounts.json” file. You can obtain your
users UID and GID by using the “id -u” and “id -g” commands on the
terminal.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
2022-03-30 03:32:27,414 [INFO] iva.detectnet_v2.dataio.build_converter: Instantiating a kitti converter
2022-03-30 03:32:27,414 [INFO] root: Instantiating a kitti converter
2022-03-30 03:32:27,414 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Creating output directory /home/jeff/tao-experiments/data/training/tfrecords
2022-03-30 03:32:27,414 [INFO] root: Generating partitions
2022-03-30 03:32:27,433 [INFO] iva.detectnet_v2.dataio.kitti_converter_lib: Num images in
Train: 5212 Val: 848
2022-03-30 03:32:27,433 [INFO] root: Num images in
Train: 5212 Val: 848
2022-03-30 03:32:27,433 [INFO] iva.detectnet_v2.dataio.kitti_converter_lib: Validation data in partition 0. Hence, while choosing the validationset during training choose validation_fold 0.
2022-03-30 03:32:27,433 [INFO] root: Validation data in partition 0. Hence, while choosing the validationset during training choose validation_fold 0.
2022-03-30 03:32:27,438 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 0
2022-03-30 03:32:27,438 [INFO] root: Writing partition 0, shard 0
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py:161: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.

2022-03-30 03:32:27,438 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py:161: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.

/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:297: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.
2022-03-30 03:32:27,526 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 1
2022-03-30 03:32:27,526 [INFO] root: Writing partition 0, shard 1
2022-03-30 03:32:27,608 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 2
2022-03-30 03:32:27,608 [INFO] root: Writing partition 0, shard 2
2022-03-30 03:32:27,691 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 3
2022-03-30 03:32:27,691 [INFO] root: Writing partition 0, shard 3
2022-03-30 03:32:27,772 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 4
2022-03-30 03:32:27,772 [INFO] root: Writing partition 0, shard 4
2022-03-30 03:32:27,855 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 5
2022-03-30 03:32:27,855 [INFO] root: Writing partition 0, shard 5
2022-03-30 03:32:27,936 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 6
2022-03-30 03:32:27,936 [INFO] root: Writing partition 0, shard 6
2022-03-30 03:32:28,019 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 7
2022-03-30 03:32:28,019 [INFO] root: Writing partition 0, shard 7
2022-03-30 03:32:28,100 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 8
2022-03-30 03:32:28,100 [INFO] root: Writing partition 0, shard 8
2022-03-30 03:32:28,183 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 9
2022-03-30 03:32:28,183 [INFO] root: Writing partition 0, shard 9
2022-03-30 03:32:28,272 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib:
Wrote the following numbers of objects:
b’pedestrian’: 455
b’car’: 3285
b’dontcare’: 1338
b’van’: 349
b’misc’: 98
b’truck’: 121
b’cyclist’: 193
b’tram’: 52
b’person_sitting’: 23

2022-03-30 03:32:28,272 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 0
2022-03-30 03:32:28,272 [INFO] root: Writing partition 1, shard 0
2022-03-30 03:32:28,775 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 1
2022-03-30 03:32:28,775 [INFO] root: Writing partition 1, shard 1
2022-03-30 03:32:29,283 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 2
2022-03-30 03:32:29,284 [INFO] root: Writing partition 1, shard 2
2022-03-30 03:32:29,784 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 3
2022-03-30 03:32:29,784 [INFO] root: Writing partition 1, shard 3
2022-03-30 03:32:30,281 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 4
2022-03-30 03:32:30,281 [INFO] root: Writing partition 1, shard 4
2022-03-30 03:32:30,782 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 5
2022-03-30 03:32:30,782 [INFO] root: Writing partition 1, shard 5
2022-03-30 03:32:31,283 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 6
2022-03-30 03:32:31,283 [INFO] root: Writing partition 1, shard 6
2022-03-30 03:32:31,787 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 7
2022-03-30 03:32:31,787 [INFO] root: Writing partition 1, shard 7
2022-03-30 03:32:32,293 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 8
2022-03-30 03:32:32,293 [INFO] root: Writing partition 1, shard 8
2022-03-30 03:32:32,806 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 9
2022-03-30 03:32:32,806 [INFO] root: Writing partition 1, shard 9
2022-03-30 03:32:33,315 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib:
Wrote the following numbers of objects:
b’car’: 19969
b’misc’: 697
b’dontcare’: 7818
b’cyclist’: 1169
b’van’: 2028
b’pedestrian’: 3169
b’truck’: 772
b’tram’: 346
b’person_sitting’: 149

2022-03-30 03:32:33,315 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Cumulative object statistics
2022-03-30 03:32:33,315 [INFO] root: Cumulative object statistics
2022-03-30 03:32:33,316 [INFO] root: {
“pedestrian”: 3624,
“car”: 23254,
“dontcare”: 9156,
“van”: 2377,
“misc”: 795,
“truck”: 893,
“cyclist”: 1362,
“tram”: 398,
“person_sitting”: 172
}
2022-03-30 03:32:33,316 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib:
Wrote the following numbers of objects:
b’pedestrian’: 3624
b’car’: 23254
b’dontcare’: 9156
b’van’: 2377
b’misc’: 795
b’truck’: 893
b’cyclist’: 1362
b’tram’: 398
b’person_sitting’: 172

2022-03-30 03:32:33,316 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Class map.
Label in GT: Label in tfrecords file
b’Pedestrian’: b’pedestrian’
b’Car’: b’car’
b’DontCare’: b’dontcare’
b’Van’: b’van’
b’Misc’: b’misc’
b’Truck’: b’truck’
b’Cyclist’: b’cyclist’
b’Tram’: b’tram’
b’Person_sitting’: b’person_sitting’
2022-03-30 03:32:33,316 [INFO] root: Class map.
Label in GT: Label in tfrecords file
b’Pedestrian’: b’pedestrian’
b’Car’: b’car’
b’DontCare’: b’dontcare’
b’Van’: b’van’
b’Misc’: b’misc’
b’Truck’: b’truck’
b’Cyclist’: b’cyclist’
b’Tram’: b’tram’
b’Person_sitting’: b’person_sitting’
For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.

2022-03-30 03:32:33,316 [INFO] root: For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.

2022-03-30 03:32:33,316 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Tfrecords generation complete.
2022-03-30 03:32:33,316 [INFO] root: TFRecords generation complete.
2022-03-30 11:32:34,188 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

According to your command, please note that the output tfrecords files locate at $DATA_DOWNLOAD_DIR/training/tfrecords/train

And also this is a path inside the docker. It depends on the setting in your ~/.tao_mounts.json .

the env values are set as follow:

%env USER_EXPERIMENT_DIR=/home/jeff/tao-experiments/yolo_v4_tiny
%env DATA_DOWNLOAD_DIR=/home/jeff/tao-experiments/data

this is how the .tao_mounts.json looks like:
2

I also check the volume mapped in the container:

and everything seems nice,I can event mkdir a folder there.
But, the tfrecods folder still miss after running that command.

You already login the tao docker.
So, please try to run below command again.
# dataset_convert -d yolo_v4_tiny_tfrecords_kitti_train.txt -o training/tfrecords/train


I use the dataset_converter_lib.pyc in the path
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio
and this is the result.

How about
# dataset_convert -d yolo_v4_tiny_tfrecords_kitti_train.txt -o training/tfrecords/train


An error is reported

How about
# yolov4_tiny dataset_convert xxx


the error is still the same

You are using wrong spec file.
Refer to jupyter notebook or DetectNet_v2 — TAO Toolkit 3.22.05 documentation

The model I want to train is the Yolo_v4_tiny

The spec file I am using is located in the yolo_v4_tiny/specs/yolo_v4_tiny_tfrecords_kitti_train.txt

and this is the content inside this file:

kitti_config {
  root_directory_path: "/workspace/tao-experiments/data/training"
  image_dir_name: "image_2"
  label_dir_name: "label_2"
  image_extension: ".png"
  partition_mode: "random"
  num_partitions: 2
  val_split: 14
  num_shards: 10
}
image_directory_path: "/workspace/tao-experiments/data/training"

also, according to the yolo_v4_tiny.ipynb , it also uses this one in the dataset_converter

so, now I should try to use the dataset converter command in the DetectNet_v2 ??

No, it is the same. You can still use “yolov4_tiny dataset_convert”.

See

Can you double check if you are using correct spec?

sorry, I neglect…

The yolo_v4_tiny_tfrecords_train_kitti.txt should be yolo_v4_tiny_tfrecords_kitti_train.txt.

Thank you for your patience, after the correction, the dataset has been successfully converted.

but I am still confused why outside the container, the converter can not generate the folder.

Maybe it is related to
“user”: “1000:1000”,

You can refer to TAO Toolkit Launcher — TAO Toolkit 3.22.05 documentation.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.