Tao pre-trained yolo4tiny - AssertionError: Must have more boxes than clusters

usually I set the height and the width of the images, so the NN first input layer will be build according to these arguments, and I didnt see them in the config files (‘yolo_v4_tiny_train/retrain…’) files. I that why i though the No training configuration found message was displayed. so I asked you where are these parameters are need to be set.

Do you mean you did not see them in below?

I have another problem at the “retrain pruned model”:
I changed in both configuration files , in the augmentation_config section the output_width and output_height variables to 640 and 480, since these are the raw image size

augmentation_config {
  hue: 0.1
  saturation: 1.5
  exposure:1.5
  vertical_flip:0
  horizontal_flip: 0.5
  jitter: 0.3
  output_width: 640
  output_height: 480
  output_channel: 3
  randomize_input_shape_period: 10
  mosaic_prob: 0.5
  mosaic_min_ratio:0.2
}

and after I ran all the steps till the “retrain pruned model” every thing worked fine, but in this step I received the following error:

Epoch 1/80
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py", line 110, in <module>
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/utils.py", line 528, in return_func
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/utils.py", line 516, in return_func
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py", line 106, in main
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py", line 63, in run_experiment
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/models/yolov4_model.py", line 644, in train
  File "/usr/local/lib/python3.6/dist-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1418, in fit_generator
    initial_epoch=initial_epoch)
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/training_generator.py", line 217, in fit_generator
    class_weight=class_weight)
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1211, in train_on_batch
    class_weight=class_weight)
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 751, in _standardize_user_data
    exception_prefix='input')
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/training_utils.py", line 138, in standardize_input_data
    str(data_shape))
ValueError: Error when checking input: expected Input to have shape (3, 480, 640) but got array with shape (3, 352, 672)
terminate called without an active exception
[e8f2afdbefe4:00048] *** Process received signal ***
[e8f2afdbefe4:00048] Signal: Aborted (6)
[e8f2afdbefe4:00048] Signal code:  (-6)

what could be the problem and how can I fix it?
thanks again for your help!

no I wonder if there are initial parameters that I should set before the training, and i missed it. for example the image size or number of channels?

No, you did not miss something.

For

“ValueError: Error when checking input: expected Input to have shape (3, 480, 640) but got array with shape (3, 352, 672)”

Please double check your spec of below. There are 3 kinds of model. Need to set correctly. For example, if you run retraining against pruned model, need to set correct pruned_model_path only.

You’re something special! Thanks!

As for my previous question, I actually use IR images, i.e. it has one channel, where can I change the configuration so that the NN will reduce the amount of calculation and support one channel instead of three by default?

For one channel, please set below.
output_channel: 1

in the augmentation scope?

Correct. YOLOv4-tiny — TAO Toolkit 3.22.05 documentation

so i double checked the config ‘yolo_v4_tiny_retrain’
and it config to point on the right pruned.tlt file

pruned_model_path: "/workspace/tao-experiments/yolo_v4_tiny/experiment_dir_pruned/yolov4_cspdarknet_tiny_pruned.tlt"

so again I received the error:
ValueError: Error when checking input: expected Input to have shape (3, 480, 640) but got array with shape (3, 352, 672)

Please check if tfrecords way can help you.
Refer to Training custom model using Yolo_v4_tiny - #14

OK, I have tried that but while executing

!tao yolo_v4_tiny dataset_convert -d $SPECS_DIR/yolo_v4_tiny_tfrecords_chimera_train.txt \
                             -o $DATA_DOWNLOAD_DIR/training/tfrecords/train

with the file format of :

chimera_config {
  root_directory_path: "/workspace/tao-experiments/data/chimera_ir_training"
  image_dir_name: "images"
  label_dir_name: "labels"
  image_extension: ".png"
  partition_mode: "random"
  num_partitions: 2
  val_split: 14
  num_shards: 10
}
image_directory_path: "/workspace/tao-experiments/data/chimera_ir_training"

received the following error:

google.protobuf.text_format.ParseError: 1:1 : Message type “DatasetExportConfig” has no field named “chimera_config”. 2022-01-12 16:18:35,822 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

changing it back to “kitti_config”

kitti_config {
  root_directory_path: "/workspace/tao-experiments/data/chimera_ir_training"
  image_dir_name: "images"
  label_dir_name: "labels"
  image_extension: ".png"
  partition_mode: "random"
  num_partitions: 2
  val_split: 14
  num_shards: 10
}
image_directory_path: "/workspace/tao-experiments/data/chimera_ir_training"

produce the following error:

Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/dataset_convert.py", line 18, in <module>
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/dataset_convert.py", line 114, in main
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py", line 73, in convert
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py", line 121, in _write_partitions
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py", line 165, in _write_shard
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/kitti_converter_lib.py", line 192, in _create_example_proto
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/kitti_converter_lib.py", line 321, in _add_targets
AttributeError: 'int' object has no attribute 'lower'

BTW could not find what these two parameter mean, can you please provide a link for the file format documentation?

See DetectNet_v2 - NVIDIA Docs

Is the class name a number instead of character?

changing the output channel to 1 cause to the following error while training:
ValueError: Layer weight shape (3, 3, 1, 32) not compatible with provided weight shape (3, 3, 3, 32)

For 1 channel, please train from scratch. The pretrained model in ngc is not compatible with 1 channel.

OK thanks,
I tried to the train stage with yolo_v4_tiny_retrain.txt file, after I generated the tfrecords files.
the output raise the follow error :

Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py", line 110, in <module>
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/utils.py", line 528, in return_func
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/utils.py", line 516, in return_func
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py", line 106, in main
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py", line 58, in run_experiment
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/models/utils.py", line 77, in build_training_pipeline
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/dataio/tf_data_pipe.py", line 316, in __init__
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/data_loader/yolo_v3_data_loader.py", line 629, in get_dataset_tensors
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/blocks/trainers/multi_task_trainer/data_loader_interface.py", line 77, in __call__
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build_wheel.runfiles/ai_infra/moduluspy/modulus/blocks/data_loaders/multi_source_loader/data_loader.py", line 598, in call
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 2081, in apply
    return DatasetV1Adapter(super(DatasetV1, self).apply(transformation_func))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 1422, in apply
    dataset = transformation_func(self)
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/data_loader/yolo_v3_data_loader.py", line 392, in <lambda>
  File "/opt/nvidia/third_party/keras/tensorflow_backend.py", line 356, in new_map
    self, _map_func_set_random_wrapper, num_parallel_calls=num_parallel_calls
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 2000, in map
    MapDataset(self, map_func, preserve_cardinality=False))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 3531, in __init__
    use_legacy_function=use_legacy_function)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 2810, in __init__
    self._function = wrapper_fn._get_concrete_function_internal()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 1853, in _get_concrete_function_internal
    *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 1847, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2147, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/function.py", line 2038, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/func_graph.py", line 915, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 2804, in wrapper_fn
    ret = _wrapper_helper(*args)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py", line 2749, in _wrapper_helper
    ret = autograph.tf_convert(func, ag_ctx)(*nested_args)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/impl/api.py", line 237, in wrapper
    raise e.ag_error_metadata.to_exception(e)
StopIteration: in converted code:

    /opt/nvidia/third_party/keras/tensorflow_backend.py:353 _map_func_set_random_wrapper  *
        return map_func(*args, **kwargs)
    /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/data_loader/yolo_v3_data_loader.py:156 __call__
        
    /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/data_loader/yolo_v3_data_loader.py:124 _get_parse_example
        
    /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataloader/utilities.py:217 extract_tfrecords_features
        

    StopIteration: 

2022-01-13 10:04:33,306 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

what you advice to do?
thanks

Can you share the log when you generate tfrecords file?
More, can you run “ls -sh yourtfrecord_folder”?

!tao yolo_v4_tiny dataset_convert -d $SPECS_DIR/yolo_v4_tiny_tfrecords_chimera_train.txt
-o $DATA_DOWNLOAD_DIR/chimera_ir_training/tfrecords/train

/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:297: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.
2022-01-13 09:34:41,404 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 1
2022-01-13 09:34:41,404 [INFO] root: Writing partition 0, shard 1
2022-01-13 09:34:41,406 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 2
2022-01-13 09:34:41,406 [INFO] root: Writing partition 0, shard 2
2022-01-13 09:34:41,407 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 3
2022-01-13 09:34:41,407 [INFO] root: Writing partition 0, shard 3
2022-01-13 09:34:41,408 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 4
2022-01-13 09:34:41,409 [INFO] root: Writing partition 0, shard 4
2022-01-13 09:34:41,410 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 5
2022-01-13 09:34:41,410 [INFO] root: Writing partition 0, shard 5
2022-01-13 09:34:41,411 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 6
2022-01-13 09:34:41,412 [INFO] root: Writing partition 0, shard 6
2022-01-13 09:34:41,413 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 7
2022-01-13 09:34:41,413 [INFO] root: Writing partition 0, shard 7
2022-01-13 09:34:41,414 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 8
2022-01-13 09:34:41,415 [INFO] root: Writing partition 0, shard 8
2022-01-13 09:34:41,416 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 9
2022-01-13 09:34:41,416 [INFO] root: Writing partition 0, shard 9
2022-01-13 09:34:41,420 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: 
Wrote the following numbers of objects:
b'car': 64
b'truck': 11
b'tank': 4

2022-01-13 09:34:41,421 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 0
2022-01-13 09:34:41,421 [INFO] root: Writing partition 1, shard 0
2022-01-13 09:34:41,429 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 1
2022-01-13 09:34:41,429 [INFO] root: Writing partition 1, shard 1
2022-01-13 09:34:41,439 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 2
2022-01-13 09:34:41,439 [INFO] root: Writing partition 1, shard 2
2022-01-13 09:34:41,448 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 3
2022-01-13 09:34:41,448 [INFO] root: Writing partition 1, shard 3
2022-01-13 09:34:41,457 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 4
2022-01-13 09:34:41,457 [INFO] root: Writing partition 1, shard 4
2022-01-13 09:34:41,466 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 5
2022-01-13 09:34:41,466 [INFO] root: Writing partition 1, shard 5
2022-01-13 09:34:41,475 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 6
2022-01-13 09:34:41,475 [INFO] root: Writing partition 1, shard 6
2022-01-13 09:34:41,485 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 7
2022-01-13 09:34:41,485 [INFO] root: Writing partition 1, shard 7
2022-01-13 09:34:41,495 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 8
2022-01-13 09:34:41,495 [INFO] root: Writing partition 1, shard 8
2022-01-13 09:34:41,503 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 9
2022-01-13 09:34:41,503 [INFO] root: Writing partition 1, shard 9
2022-01-13 09:34:41,521 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: 
Wrote the following numbers of objects:
b'car': 529
b'tank': 16
b'truck': 86
b'person': 30

2022-01-13 09:34:41,521 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Cumulative object statistics
2022-01-13 09:34:41,521 [INFO] root: Cumulative object statistics
2022-01-13 09:34:41,522 [INFO] root: {
    "car": 593,
    "truck": 97,
    "tank": 20,
    "person": 30
}
2022-01-13 09:34:41,522 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: 
Wrote the following numbers of objects:
b'car': 593
b'truck': 97
b'tank': 20
b'person': 30

2022-01-13 09:34:41,522 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Class map. 
Label in GT: Label in tfrecords file 
b'car': b'car'
b'truck': b'truck'
b'tank': b'tank'
b'person': b'person'
2022-01-13 09:34:41,522 [INFO] root: Class map. 
Label in GT: Label in tfrecords file 
b'car': b'car'
b'truck': b'truck'
b'tank': b'tank'
b'person': b'person'
For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.

2022-01-13 09:34:41,522 [INFO] root: For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.

2022-01-13 09:34:41,522 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Tfrecords generation complete.
2022-01-13 09:34:41,522 [INFO] root: TFRecords generation complete.
2022-01-13 09:34:42,491 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.
!tao yolo_v4_tiny dataset_convert -d $SPECS_DIR/yolo_v4_tiny_tfrecords_chimera_val.txt \
                             -o $DATA_DOWNLOAD_DIR/chimera_ir_val/tfrecords/val
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
2022-01-13 09:48:01,947 [INFO] iva.detectnet_v2.dataio.build_converter: Instantiating a kitti converter
2022-01-13 09:48:01,947 [INFO] root: Instantiating a kitti converter
2022-01-13 09:48:01,947 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Creating output directory /workspace/tao-experiments/data/chimera_ir_val/tfrecords
2022-01-13 09:48:01,947 [INFO] root: Generating partitions
2022-01-13 09:48:01,948 [INFO] iva.detectnet_v2.dataio.kitti_converter_lib: Num images in
Train: 8	Val: 1
2022-01-13 09:48:01,948 [INFO] root: Num images in
Train: 8	Val: 1
2022-01-13 09:48:01,948 [INFO] iva.detectnet_v2.dataio.kitti_converter_lib: Validation data in partition 0. Hence, while choosing the validationset during training choose validation_fold 0.
2022-01-13 09:48:01,948 [INFO] root: Validation data in partition 0. Hence, while choosing the validationset during training choose validation_fold 0.
2022-01-13 09:48:01,948 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 0
2022-01-13 09:48:01,948 [INFO] root: Writing partition 0, shard 0
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py:161: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.

2022-01-13 09:48:01,948 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py:161: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.

2022-01-13 09:48:01,949 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 1
2022-01-13 09:48:01,949 [INFO] root: Writing partition 0, shard 1
2022-01-13 09:48:01,949 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 2
2022-01-13 09:48:01,949 [INFO] root: Writing partition 0, shard 2
2022-01-13 09:48:01,949 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 3
2022-01-13 09:48:01,949 [INFO] root: Writing partition 0, shard 3
2022-01-13 09:48:01,949 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 4
2022-01-13 09:48:01,949 [INFO] root: Writing partition 0, shard 4
2022-01-13 09:48:01,950 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 5
2022-01-13 09:48:01,950 [INFO] root: Writing partition 0, shard 5
2022-01-13 09:48:01,950 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 6
2022-01-13 09:48:01,950 [INFO] root: Writing partition 0, shard 6
2022-01-13 09:48:01,950 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 7
2022-01-13 09:48:01,950 [INFO] root: Writing partition 0, shard 7
2022-01-13 09:48:01,950 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 8
2022-01-13 09:48:01,950 [INFO] root: Writing partition 0, shard 8
2022-01-13 09:48:01,950 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 0, shard 9
2022-01-13 09:48:01,951 [INFO] root: Writing partition 0, shard 9
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:297: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.
2022-01-13 09:48:01,959 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: 
Wrote the following numbers of objects:
b'car': 4

2022-01-13 09:48:01,959 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 0
2022-01-13 09:48:01,959 [INFO] root: Writing partition 1, shard 0
2022-01-13 09:48:01,959 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 1
2022-01-13 09:48:01,959 [INFO] root: Writing partition 1, shard 1
2022-01-13 09:48:01,959 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 2
2022-01-13 09:48:01,960 [INFO] root: Writing partition 1, shard 2
2022-01-13 09:48:01,960 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 3
2022-01-13 09:48:01,960 [INFO] root: Writing partition 1, shard 3
2022-01-13 09:48:01,960 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 4
2022-01-13 09:48:01,960 [INFO] root: Writing partition 1, shard 4
2022-01-13 09:48:01,960 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 5
2022-01-13 09:48:01,960 [INFO] root: Writing partition 1, shard 5
2022-01-13 09:48:01,961 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 6
2022-01-13 09:48:01,961 [INFO] root: Writing partition 1, shard 6
2022-01-13 09:48:01,961 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 7
2022-01-13 09:48:01,961 [INFO] root: Writing partition 1, shard 7
2022-01-13 09:48:01,961 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 8
2022-01-13 09:48:01,961 [INFO] root: Writing partition 1, shard 8
2022-01-13 09:48:01,961 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Writing partition 1, shard 9
2022-01-13 09:48:01,961 [INFO] root: Writing partition 1, shard 9
2022-01-13 09:48:01,971 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: 
Wrote the following numbers of objects:
b'truck': 13
b'car': 34
b'person': 2
b'tank': 1

2022-01-13 09:48:01,971 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Cumulative object statistics
2022-01-13 09:48:01,971 [INFO] root: Cumulative object statistics
2022-01-13 09:48:01,972 [INFO] root: {
    "car": 38,
    "truck": 13,
    "person": 2,
    "tank": 1
}
2022-01-13 09:48:01,972 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: 
Wrote the following numbers of objects:
b'car': 38
b'truck': 13
b'person': 2
b'tank': 1

2022-01-13 09:48:01,972 [INFO] iva.detectnet_v2.dataio.dataset_converter_lib: Class map. 
Label in GT: Label in tfrecords file 
b'car': b'car'
b'truck': b'truck'
b'person': b'person'
b'tank': b'tank'
2022-01-13 09:48:01,972 [INFO] root: Class map. 
Label in GT: Label in tfrecords file 
b'car': b'car'
b'truck': b'truck'
b'person': b'person'
b'tank': b'tank'
For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.

2022-01-13 09:48:01,972 [INFO] root: For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.

ls -sh chimera_ir_training/tfrecords

total 132K
4.0K train-fold-000-of-002-shard-00000-of-00010  4.0K train-fold-000-of-002-shard-00005-of-00010  8.0K train-fold-001-of-002-shard-00000-of-00010  8.0K train-fold-001-of-002-shard-00005-of-00010
4.0K train-fold-000-of-002-shard-00001-of-00010  4.0K train-fold-000-of-002-shard-00006-of-00010  8.0K train-fold-001-of-002-shard-00001-of-00010   12K train-fold-001-of-002-shard-00006-of-00010
4.0K train-fold-000-of-002-shard-00002-of-00010  4.0K train-fold-000-of-002-shard-00007-of-00010  8.0K train-fold-001-of-002-shard-00002-of-00010  8.0K train-fold-001-of-002-shard-00007-of-00010
4.0K train-fold-000-of-002-shard-00003-of-00010  4.0K train-fold-000-of-002-shard-00008-of-00010  8.0K train-fold-001-of-002-shard-00003-of-00010  8.0K train-fold-001-of-002-shard-00008-of-00010
4.0K train-fold-000-of-002-shard-00004-of-00010  4.0K train-fold-000-of-002-shard-00009-of-00010  8.0K train-fold-001-of-002-shard-00004-of-00010   16K train-fold-001-of-002-shard-00009-of-00010

ls -sh chimera_ir_val/tfrecords

total 12K
   0 val-fold-000-of-002-shard-00000-of-00010     0 val-fold-000-of-002-shard-00005-of-00010     0 val-fold-001-of-002-shard-00000-of-00010     0 val-fold-001-of-002-shard-00005-of-00010
   0 val-fold-000-of-002-shard-00001-of-00010     0 val-fold-000-of-002-shard-00006-of-00010     0 val-fold-001-of-002-shard-00001-of-00010     0 val-fold-001-of-002-shard-00006-of-00010
   0 val-fold-000-of-002-shard-00002-of-00010     0 val-fold-000-of-002-shard-00007-of-00010     0 val-fold-001-of-002-shard-00002-of-00010     0 val-fold-001-of-002-shard-00007-of-00010
   0 val-fold-000-of-002-shard-00003-of-00010     0 val-fold-000-of-002-shard-00008-of-00010     0 val-fold-001-of-002-shard-00003-of-00010     0 val-fold-001-of-002-shard-00008-of-00010
   0 val-fold-000-of-002-shard-00004-of-00010  4.0K val-fold-000-of-002-shard-00009-of-00010     0 val-fold-001-of-002-shard-00004-of-00010  8.0K val-fold-001-of-002-shard-00009-of-00010