usually I set the height and the width of the images, so the NN first input layer will be build according to these arguments, and I didnt see them in the config files (‘yolo_v4_tiny_train/retrain…’) files. I that why i though the No training configuration found
message was displayed. so I asked you where are these parameters are need to be set.
I have another problem at the “retrain pruned model”:
I changed in both configuration files , in the augmentation_config section the output_width and output_height variables to 640 and 480, since these are the raw image size
augmentation_config {
hue: 0.1
saturation: 1.5
exposure:1.5
vertical_flip:0
horizontal_flip: 0.5
jitter: 0.3
output_width: 640
output_height: 480
output_channel: 3
randomize_input_shape_period: 10
mosaic_prob: 0.5
mosaic_min_ratio:0.2
}
and after I ran all the steps till the “retrain pruned model” every thing worked fine, but in this step I received the following error:
Epoch 1/80
Traceback (most recent call last):
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py", line 110, in <module>
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/utils.py", line 528, in return_func
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/utils.py", line 516, in return_func
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py", line 106, in main
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py", line 63, in run_experiment
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/models/yolov4_model.py", line 644, in train
File "/usr/local/lib/python3.6/dist-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1418, in fit_generator
initial_epoch=initial_epoch)
File "/usr/local/lib/python3.6/dist-packages/keras/engine/training_generator.py", line 217, in fit_generator
class_weight=class_weight)
File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1211, in train_on_batch
class_weight=class_weight)
File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 751, in _standardize_user_data
exception_prefix='input')
File "/usr/local/lib/python3.6/dist-packages/keras/engine/training_utils.py", line 138, in standardize_input_data
str(data_shape))
ValueError: Error when checking input: expected Input to have shape (3, 480, 640) but got array with shape (3, 352, 672)
terminate called without an active exception
[e8f2afdbefe4:00048] *** Process received signal ***
[e8f2afdbefe4:00048] Signal: Aborted (6)
[e8f2afdbefe4:00048] Signal code: (-6)
what could be the problem and how can I fix it?
thanks again for your help!
no I wonder if there are initial parameters that I should set before the training, and i missed it. for example the image size or number of channels?
No, you did not miss something.
For
“ValueError: Error when checking input: expected Input to have shape (3, 480, 640) but got array with shape (3, 352, 672)”
Please double check your spec of below. There are 3 kinds of model. Need to set correctly. For example, if you run retraining against pruned model, need to set correct pruned_model_path only.
You’re something special! Thanks!
As for my previous question, I actually use IR images, i.e. it has one channel, where can I change the configuration so that the NN will reduce the amount of calculation and support one channel instead of three by default?
For one channel, please set below.
output_channel: 1
in the augmentation scope?
so i double checked the config ‘yolo_v4_tiny_retrain’
and it config to point on the right pruned.tlt file
pruned_model_path: "/workspace/tao-experiments/yolo_v4_tiny/experiment_dir_pruned/yolov4_cspdarknet_tiny_pruned.tlt"
so again I received the error:
ValueError: Error when checking input: expected Input to have shape (3, 480, 640) but got array with shape (3, 352, 672)
OK, I have tried that but while executing
!tao yolo_v4_tiny dataset_convert -d $SPECS_DIR/yolo_v4_tiny_tfrecords_chimera_train.txt \
-o $DATA_DOWNLOAD_DIR/training/tfrecords/train
with the file format of :
chimera_config {
root_directory_path: "/workspace/tao-experiments/data/chimera_ir_training"
image_dir_name: "images"
label_dir_name: "labels"
image_extension: ".png"
partition_mode: "random"
num_partitions: 2
val_split: 14
num_shards: 10
}
image_directory_path: "/workspace/tao-experiments/data/chimera_ir_training"
received the following error:
google.protobuf.text_format.ParseError: 1:1 : Message type “DatasetExportConfig” has no field named “chimera_config”. 2022-01-12 16:18:35,822 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.
changing it back to “kitti_config”
kitti_config {
root_directory_path: "/workspace/tao-experiments/data/chimera_ir_training"
image_dir_name: "images"
label_dir_name: "labels"
image_extension: ".png"
partition_mode: "random"
num_partitions: 2
val_split: 14
num_shards: 10
}
image_directory_path: "/workspace/tao-experiments/data/chimera_ir_training"
produce the following error:
Traceback (most recent call last):
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/dataset_convert.py", line 18, in <module>
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/dataset_convert.py", line 114, in main
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py", line 73, in convert
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py", line 121, in _write_partitions
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py", line 165, in _write_shard
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/kitti_converter_lib.py", line 192, in _create_example_proto
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/kitti_converter_lib.py", line 321, in _add_targets
AttributeError: 'int' object has no attribute 'lower'
BTW could not find what these two parameter mean, can you please provide a link for the file format documentation?
changing the output channel to 1 cause to the following error while training:
ValueError: Layer weight shape (3, 3, 1, 32) not compatible with provided weight shape (3, 3, 3, 32)
For 1 channel, please train from scratch. The pretrained model in ngc is not compatible with 1 channel.
OK thanks,
I tried to the train stage with yolo_v4_tiny_retrain.txt file, after I generated the tfrecords files.
the output raise the follow error :
Traceback (most recent call last):
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py", line 110, in <module>
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/utils.py", line 528, in return_func
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/utils.py", line 516, in return_func
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py", line 106, in main
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py", line 58, in run_experiment
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/models/utils.py", line 77, in build_training_pipeline
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/dataio/tf_data_pipe.py", line 316, in __init__
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/data_loader/yolo_v3_data_loader.py", line 629, in get_dataset_tensors
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/blocks/trainers/multi_task_trainer/data_loader_interface.py", line 77, in __call__
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/blocks/data_loaders/multi_source_loader/data_loader.py", line 598, in call
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 2081, in apply
return DatasetV1Adapter(super(DatasetV1, self).apply(transformation_func))
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 1422, in apply
dataset = transformation_func(self)
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/data_loader/yolo_v3_data_loader.py", line 392, in <lambda>
File "/opt/nvidia/third_party/keras/tensorflow_backend.py", line 356, in new_map
self, _map_func_set_random_wrapper, num_parallel_calls=num_parallel_calls
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 2000, in map
MapDataset(self, map_func, preserve_cardinality=False))
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 3531, in __init__
use_legacy_function=use_legacy_function)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 2810, in __init__
self._function = wrapper_fn._get_concrete_function_internal()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 1853, in _get_concrete_function_internal
*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 1847, in _get_concrete_function_internal_garbage_collected
graph_function, _, _ = self._maybe_define_function(args, kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2147, in _maybe_define_function
graph_function = self._create_graph_function(args, kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2038, in _create_graph_function
capture_by_value=self._capture_by_value),
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/func_graph.py", line 915, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 2804, in wrapper_fn
ret = _wrapper_helper(*args)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 2749, in _wrapper_helper
ret = autograph.tf_convert(func, ag_ctx)(*nested_args)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/impl/api.py", line 237, in wrapper
raise e.ag_error_metadata.to_exception(e)
StopIteration: in converted code:
/opt/nvidia/third_party/keras/tensorflow_backend.py:353 _map_func_set_random_wrapper *
return map_func(*args, **kwargs)
/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/data_loader/yolo_v3_data_loader.py:156 __call__
/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/data_loader/yolo_v3_data_loader.py:124 _get_parse_example
/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataloader/utilities.py:217 extract_tfrecords_features
StopIteration:
2022-01-13 10:04:33,306 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.
what you advice to do?
thanks
Can you share the log when you generate tfrecords file?
More, can you run “ls -sh yourtfrecord_folder”?
!tao yolo_v4_tiny dataset_convert -d $SPECS_DIR/yolo_v4_tiny_tfrecords_chimera_train.txt
-o $DATA_DOWNLOAD_DIR/chimera_ir_training/tfrecords/train
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:297: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.
2022-01-13 09:34:41,404 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 1
2022-01-13 09:34:41,404 [INFO] root: Writing partition 0, shard 1
2022-01-13 09:34:41,406 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 2
2022-01-13 09:34:41,406 [INFO] root: Writing partition 0, shard 2
2022-01-13 09:34:41,407 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 3
2022-01-13 09:34:41,407 [INFO] root: Writing partition 0, shard 3
2022-01-13 09:34:41,408 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 4
2022-01-13 09:34:41,409 [INFO] root: Writing partition 0, shard 4
2022-01-13 09:34:41,410 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 5
2022-01-13 09:34:41,410 [INFO] root: Writing partition 0, shard 5
2022-01-13 09:34:41,411 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 6
2022-01-13 09:34:41,412 [INFO] root: Writing partition 0, shard 6
2022-01-13 09:34:41,413 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 7
2022-01-13 09:34:41,413 [INFO] root: Writing partition 0, shard 7
2022-01-13 09:34:41,414 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 8
2022-01-13 09:34:41,415 [INFO] root: Writing partition 0, shard 8
2022-01-13 09:34:41,416 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 9
2022-01-13 09:34:41,416 [INFO] root: Writing partition 0, shard 9
2022-01-13 09:34:41,420 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib:
Wrote the following numbers of objects:
b'car': 64
b'truck': 11
b'tank': 4
2022-01-13 09:34:41,421 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 0
2022-01-13 09:34:41,421 [INFO] root: Writing partition 1, shard 0
2022-01-13 09:34:41,429 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 1
2022-01-13 09:34:41,429 [INFO] root: Writing partition 1, shard 1
2022-01-13 09:34:41,439 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 2
2022-01-13 09:34:41,439 [INFO] root: Writing partition 1, shard 2
2022-01-13 09:34:41,448 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 3
2022-01-13 09:34:41,448 [INFO] root: Writing partition 1, shard 3
2022-01-13 09:34:41,457 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 4
2022-01-13 09:34:41,457 [INFO] root: Writing partition 1, shard 4
2022-01-13 09:34:41,466 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 5
2022-01-13 09:34:41,466 [INFO] root: Writing partition 1, shard 5
2022-01-13 09:34:41,475 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 6
2022-01-13 09:34:41,475 [INFO] root: Writing partition 1, shard 6
2022-01-13 09:34:41,485 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 7
2022-01-13 09:34:41,485 [INFO] root: Writing partition 1, shard 7
2022-01-13 09:34:41,495 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 8
2022-01-13 09:34:41,495 [INFO] root: Writing partition 1, shard 8
2022-01-13 09:34:41,503 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 9
2022-01-13 09:34:41,503 [INFO] root: Writing partition 1, shard 9
2022-01-13 09:34:41,521 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib:
Wrote the following numbers of objects:
b'car': 529
b'tank': 16
b'truck': 86
b'person': 30
2022-01-13 09:34:41,521 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Cumulative object statistics
2022-01-13 09:34:41,521 [INFO] root: Cumulative object statistics
2022-01-13 09:34:41,522 [INFO] root: {
"car": 593,
"truck": 97,
"tank": 20,
"person": 30
}
2022-01-13 09:34:41,522 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib:
Wrote the following numbers of objects:
b'car': 593
b'truck': 97
b'tank': 20
b'person': 30
2022-01-13 09:34:41,522 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Class map.
Label in GT: Label in tfrecords file
b'car': b'car'
b'truck': b'truck'
b'tank': b'tank'
b'person': b'person'
2022-01-13 09:34:41,522 [INFO] root: Class map.
Label in GT: Label in tfrecords file
b'car': b'car'
b'truck': b'truck'
b'tank': b'tank'
b'person': b'person'
For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.
2022-01-13 09:34:41,522 [INFO] root: For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.
2022-01-13 09:34:41,522 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Tfrecords generation complete.
2022-01-13 09:34:41,522 [INFO] root: TFRecords generation complete.
2022-01-13 09:34:42,491 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.
!tao yolo_v4_tiny dataset_convert -d $SPECS_DIR/yolo_v4_tiny_tfrecords_chimera_val.txt \
-o $DATA_DOWNLOAD_DIR/chimera_ir_val/tfrecords/val
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
2022-01-13 09:48:01,947 [INFO] iva.detectnet_v2.dataio.build_converter: Instantiating a kitti converter
2022-01-13 09:48:01,947 [INFO] root: Instantiating a kitti converter
2022-01-13 09:48:01,947 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Creating output directory /workspace/tao-experiments/data/chimera_ir_val/tfrecords
2022-01-13 09:48:01,947 [INFO] root: Generating partitions
2022-01-13 09:48:01,948 [INFO] iva.detectnet_v2.dataio.kitti_converter_lib: Num images in
Train: 8 Val: 1
2022-01-13 09:48:01,948 [INFO] root: Num images in
Train: 8 Val: 1
2022-01-13 09:48:01,948 [INFO] iva.detectnet_v2.dataio.kitti_converter_lib: Validation data in partition 0. Hence, while choosing the validationset during training choose validation_fold 0.
2022-01-13 09:48:01,948 [INFO] root: Validation data in partition 0. Hence, while choosing the validationset during training choose validation_fold 0.
2022-01-13 09:48:01,948 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 0
2022-01-13 09:48:01,948 [INFO] root: Writing partition 0, shard 0
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py:161: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.
2022-01-13 09:48:01,948 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py:161: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.
2022-01-13 09:48:01,949 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 1
2022-01-13 09:48:01,949 [INFO] root: Writing partition 0, shard 1
2022-01-13 09:48:01,949 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 2
2022-01-13 09:48:01,949 [INFO] root: Writing partition 0, shard 2
2022-01-13 09:48:01,949 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 3
2022-01-13 09:48:01,949 [INFO] root: Writing partition 0, shard 3
2022-01-13 09:48:01,949 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 4
2022-01-13 09:48:01,949 [INFO] root: Writing partition 0, shard 4
2022-01-13 09:48:01,950 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 5
2022-01-13 09:48:01,950 [INFO] root: Writing partition 0, shard 5
2022-01-13 09:48:01,950 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 6
2022-01-13 09:48:01,950 [INFO] root: Writing partition 0, shard 6
2022-01-13 09:48:01,950 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 7
2022-01-13 09:48:01,950 [INFO] root: Writing partition 0, shard 7
2022-01-13 09:48:01,950 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 8
2022-01-13 09:48:01,950 [INFO] root: Writing partition 0, shard 8
2022-01-13 09:48:01,950 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 9
2022-01-13 09:48:01,951 [INFO] root: Writing partition 0, shard 9
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:297: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.
2022-01-13 09:48:01,959 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib:
Wrote the following numbers of objects:
b'car': 4
2022-01-13 09:48:01,959 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 0
2022-01-13 09:48:01,959 [INFO] root: Writing partition 1, shard 0
2022-01-13 09:48:01,959 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 1
2022-01-13 09:48:01,959 [INFO] root: Writing partition 1, shard 1
2022-01-13 09:48:01,959 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 2
2022-01-13 09:48:01,960 [INFO] root: Writing partition 1, shard 2
2022-01-13 09:48:01,960 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 3
2022-01-13 09:48:01,960 [INFO] root: Writing partition 1, shard 3
2022-01-13 09:48:01,960 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 4
2022-01-13 09:48:01,960 [INFO] root: Writing partition 1, shard 4
2022-01-13 09:48:01,960 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 5
2022-01-13 09:48:01,960 [INFO] root: Writing partition 1, shard 5
2022-01-13 09:48:01,961 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 6
2022-01-13 09:48:01,961 [INFO] root: Writing partition 1, shard 6
2022-01-13 09:48:01,961 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 7
2022-01-13 09:48:01,961 [INFO] root: Writing partition 1, shard 7
2022-01-13 09:48:01,961 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 8
2022-01-13 09:48:01,961 [INFO] root: Writing partition 1, shard 8
2022-01-13 09:48:01,961 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 9
2022-01-13 09:48:01,961 [INFO] root: Writing partition 1, shard 9
2022-01-13 09:48:01,971 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib:
Wrote the following numbers of objects:
b'truck': 13
b'car': 34
b'person': 2
b'tank': 1
2022-01-13 09:48:01,971 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Cumulative object statistics
2022-01-13 09:48:01,971 [INFO] root: Cumulative object statistics
2022-01-13 09:48:01,972 [INFO] root: {
"car": 38,
"truck": 13,
"person": 2,
"tank": 1
}
2022-01-13 09:48:01,972 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib:
Wrote the following numbers of objects:
b'car': 38
b'truck': 13
b'person': 2
b'tank': 1
2022-01-13 09:48:01,972 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Class map.
Label in GT: Label in tfrecords file
b'car': b'car'
b'truck': b'truck'
b'person': b'person'
b'tank': b'tank'
2022-01-13 09:48:01,972 [INFO] root: Class map.
Label in GT: Label in tfrecords file
b'car': b'car'
b'truck': b'truck'
b'person': b'person'
b'tank': b'tank'
For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.
2022-01-13 09:48:01,972 [INFO] root: For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.
ls -sh chimera_ir_training/tfrecords
total 132K
4.0K train-fold-000-of-002-shard-00000-of-00010 4.0K train-fold-000-of-002-shard-00005-of-00010 8.0K train-fold-001-of-002-shard-00000-of-00010 8.0K train-fold-001-of-002-shard-00005-of-00010
4.0K train-fold-000-of-002-shard-00001-of-00010 4.0K train-fold-000-of-002-shard-00006-of-00010 8.0K train-fold-001-of-002-shard-00001-of-00010 12K train-fold-001-of-002-shard-00006-of-00010
4.0K train-fold-000-of-002-shard-00002-of-00010 4.0K train-fold-000-of-002-shard-00007-of-00010 8.0K train-fold-001-of-002-shard-00002-of-00010 8.0K train-fold-001-of-002-shard-00007-of-00010
4.0K train-fold-000-of-002-shard-00003-of-00010 4.0K train-fold-000-of-002-shard-00008-of-00010 8.0K train-fold-001-of-002-shard-00003-of-00010 8.0K train-fold-001-of-002-shard-00008-of-00010
4.0K train-fold-000-of-002-shard-00004-of-00010 4.0K train-fold-000-of-002-shard-00009-of-00010 8.0K train-fold-001-of-002-shard-00004-of-00010 16K train-fold-001-of-002-shard-00009-of-00010
ls -sh chimera_ir_val/tfrecords
total 12K
0 val-fold-000-of-002-shard-00000-of-00010 0 val-fold-000-of-002-shard-00005-of-00010 0 val-fold-001-of-002-shard-00000-of-00010 0 val-fold-001-of-002-shard-00005-of-00010
0 val-fold-000-of-002-shard-00001-of-00010 0 val-fold-000-of-002-shard-00006-of-00010 0 val-fold-001-of-002-shard-00001-of-00010 0 val-fold-001-of-002-shard-00006-of-00010
0 val-fold-000-of-002-shard-00002-of-00010 0 val-fold-000-of-002-shard-00007-of-00010 0 val-fold-001-of-002-shard-00002-of-00010 0 val-fold-001-of-002-shard-00007-of-00010
0 val-fold-000-of-002-shard-00003-of-00010 0 val-fold-000-of-002-shard-00008-of-00010 0 val-fold-001-of-002-shard-00003-of-00010 0 val-fold-001-of-002-shard-00008-of-00010
0 val-fold-000-of-002-shard-00004-of-00010 4.0K val-fold-000-of-002-shard-00009-of-00010 0 val-fold-001-of-002-shard-00004-of-00010 8.0K val-fold-001-of-002-shard-00009-of-00010